Where Measurement and Reporting Strategies Go Wrong

measuring and reporting

by Peggy Parskey, CTR Assistant Director

Imagine you are a manager in a Learning and Development function in your company. Each month, you receive several reports (or links to a dashboard) with a plethora of data. The reports focus on the function overall, but with a bit of digging you can find the data relevant to you. While the reports and dashboard provide a lot of information, you lament that they don’t help you manage your operation. Beyond the inadequacy of the reporting, you believe the organization doesn’t measure the right things or provide enough insights about improvement opportunities. You suggest to your CLO that the organization needs a measurement and reporting strategy.  The CLO agrees and delegates the job to your measurement person if you have one, or lacking that, to you. (Since you asked, you own it.)

Having never created a measurement and reporting strategy, you conduct a Google search.  Before November 2020, your search would have surfaced ads for product or service companies at the top, followed by a few white papers from these same companies or blogs with best practices and advice. Some content focused on measurement while others only addressed reporting.  Few, if any, provided a checklist of elements to include in your strategy. Lacking any meaningful guidance, you cobble something together from the disparate pieces of information found on the web.

This theoretical manager exists in organizations across the globe. Over the past 5-10 years, I have reviewed dozens of client-generated strategies. Most have some of the components of a strategy but nearly all lack the critical elements to advance measurement capability in the organization.  In November 2020, David Vance and I published Measurement Demystified with the express purpose to provide practical guidance on how to create a robust measurement and reporting strategy.

In this post, I’ll give you an overview of what comprises a well-designed strategy. But before sharing that with you, I’d like to review what a measurement and reporting strategy is not. Each bullet below represents something I’ve seen over the years as clients have shared their strategies.

  • Focused on reporting rather than measurement and reporting: You need both: what and how you measure as well as what and how you report. Reports are an output. Without the appropriate inputs (that is, the measures), you will be challenged to meet user needs.
  • Kirkpatrick’s four levels or Phillip’s five levels: Both frameworks provide important guidance and processes to select and report effectiveness measures. However, neither framework addresses the efficiency measures you should choose. Moreover, while both address outcome measures, they don’t provide detailed guidance on when and how to use them. They are an essential element of your strategy, but they are not the strategy.
  • One size fits all approach to measurement: To avoid implementing an unsustainable complex approach to measurement and reporting, many organizations have opted to go in the opposite direction and adopt an overly simplified approach. They identify a single suite of measures, methods, and reports across all programs regardless of their purpose. While this approach may meet the needs of some users, inevitably, it will frustrate others who lack the data to manage their operation.
  • A focus on one type of report: Dashboards are extraordinarily popular as a reporting tool and enable “speed of thought reporting”. The data is up-to-date, users can filter the data, and they can drill down to look for root cause of specific program issues. But dashboards can only take you so far. A robust strategy adapts the reporting to the user needs and their reasons for measuring.
  • A tool to gather the data. If you are a large organization, you will need a tool to enable you to aggregate and disaggregate large volumes of data. The tool is not the strategy.

Now that you have insight into what a measurement and reporting strategy is not, let’s turn to what it is.

  1. Begin your strategy with a clear articulation of why you are measuring. Different purposes to measure influence what you measure and what, how and when you report.
  2. Next, identify your users and their needs. Who will consume the data? What do they need? What decisions might they make? What actions might result from the information you provide? A strategy needs to be grounded in their requirements.
  3. Specify the measures overall that you will use. However, don’t stop there. Identify the specific measures for key learning programs or department initiatives. Include a balanced suite of measures including efficiency and effectiveness measures for all programs and outcome measures for strategic, business-aligned programs.
  4. Define your data collection approach. Where will you get the data? Where should you consider sampling? How will you ensure that you get sufficient responses to make meaningful inferences from survey data? Where can you automate to reduce effort, increase data reliability, and improve speed time to insight?
  5. Specify the types of reports you intend to use. When will you employ scorecards vs dashboards? When should you use program evaluation reports or management reports?
  6. Plan how and when you will share reports. What are the decision-making cadences and how can you align reporting to them?
  7. Finally, define the resources you need to execute and sustain the strategy. Be clear about what funding, capability, and tools your organization will require to build sustainable measurement.

Creating a measurement and reporting strategy will take time and effort.  You will need to meet with the CLO as well as senior L&D leaders and perhaps key business goal owners. The payoff for this effort will be significant and will enhance the value the L&D function delivers to the organization.  Don’t hesitate to reach out to us at the Center for Talent Reporting. We are here to help you in your journey.

 
 

Speak Your Mind