The Importance of a Balanced Set of Measures

I recently spoke with someone who shared that their organization had stopped collecting efficiency measures (which are quantity metrics like the number of participants). He said they had matured to focusing only on effectiveness and outcome measures like the application rate and impact—a big mistake. Every organization should collect a balanced set of measures for holistic reporting and analysis but also to avoid unintended consequences.

First, a quick review of the three types of measures. As noted above, efficiency measures are quantity metrics, also referred to as activity measures or Level 0. Examples are the number of participants, courses, hours, costs, completions rates, and various utilization rates. Effectiveness measures are about quality. They answer questions about how good the program is. For these, we have the Kirkpatrick/Phillips five-level framework where Level 1 is participant reaction, Level 2 is learning, Level 3 is application on the job, and Level 5 is return on investment (ROI). Level 4, results or impact, constitutes the ultimate outcome of the learning and is in a category by itself called outcomes.

A balanced set of measures is required for holistic reporting and analysis. Without a balanced set, it is impossible to see the whole picture. For example, an organization that no longer collects efficiency measures will not know how many participants have taken a course, the percentage who completed it, the cost of the course, or whether it was completed on time. At an aggregate level across all programs and courses, this organization will not know which courses are being used, utilizations rates for instructors and classrooms, percentage of employees reached by learning, or total costs for learning. Frankly, it is unimaginable that a CLO would not want to know these basic and foundational measures.

Likewise, if an organization decided to collect only efficiency measures, it would have no insight into the quality of its programs. Without level 1, how do you know if the content and instructors are good? Without a level 1 question on relevance, how do you know you have the right target audience? Level 1 is your early warning if something is wrong and needs to be addressed. Level 2 measures the learning. Without it, how do you know if participants learned the required knowledge or gained the required skills? Without it, how do you demonstrate compliance with laws, regulations, and best practices?

Level 3 is application which can be measured as intent to apply at the end of the course or actual application two-three months later.  The level 3 intent-to-apply question can easily be added to the level 1 questions in your post-event survey. Without it, how do you know if the content is likely to be applied or has been applied? If content is not applied back on the job, it is considered scrap. The scrap rate is simply 1- the application rate, and unfortunately, the scrap rate for our profession is embarrassingly high at 40%-80%.

Level 4 is results or impact and answers the question, “Did this program make any difference?” Ideally, we would like the isolated impact of learning, and Phillips provides five methods to get it. With this, we can make a statement like, “Learning contributed 3% higher sales”. If the isolated impact is not possible, the Kirkpatricks provide a methodology for making a compelling case that learning contributed to the business result. Either way, level 4 is the only way to show that learning made a difference.

Last, Level 5 is ROI. This answers the question of whether the program is worth what it cost. It is possible the program was attended by the right people who liked it, learned it, and applied it and that the program had an impact, but it cost more than it was worth. If this program is to be repeated in the future, a way must be found to lower the cost or increase the impact.

Just as with the efficiency measures, it is hard to imagine running a learning department without effectiveness measures. You simply would not know if what you are doing has any value.  And lower-level effectiveness measures are required to analyze higher-level measures. For example, if Level 4 impact is small or nonexistent, clues can be found by looking at lLvels 1-3. If participants told you that it was not relevant, you probably have the wrong audience, or the content was not designed appropriately. If they cannot pass the knowledge tests for Level 2, perhaps the content is unclear. If they cannot apply it, there may be issues with their supervisor not supporting them or providing the necessary time and resources.

So, you need a balanced set of measures to manage a single program or an entire department. Programs should always have several efficiency and effectiveness measures. For most programs, these include the number of participants, completion rate, completion date, cost, and Levels 1-3. These are the minimum. If the program supports a high-level organizational goal like increasing revenue or reducing injuries, then an outcome measure should be added as well.

Last, a balanced set is required to avoid unintended consequences. If a CLO measured staff only on efficiency measures like the number of courses produced or meeting deadlines, staff might try to comply by rushing programs out the door. In this case, effectiveness and outcome would suffer. If, instead, the CLO were to measure staff only by effectiveness and outcome, staff might try to comply by spending a long time designing each course to ensure that is of the highest quality. Consequently, very few courses would be produced, deadlines would be missed, and costs would skyrocket.

In conclusion, a balanced set of measures is key to success in designing, delivering, measuring, and managing learning at both the program and department levels.

Learn more about the three types of measures and Talent Development Reporting Principles (TDRP) at our November 2 virtual conference.