How Difficult is Good L&D Measurement?

In the past, I have always said that good measurement of L&D is not that difficult. However, the more we have worked with individuals and organizations and the more workshops we teach, I am beginning to reconsider for three reasons.

1. Metrics are Not as Simple as They Appear

First, the metrics themselves. Some metrics are easy to understand and easy to measure. For example, the number of courses offered or used is easy to determine if you have Learning Management System (LMS) and not too difficult even if you use an Excel worksheet. However, many metrics are not as simple as they seem. For example, the number of participants where the user needs to know the difference between unique (no duplications allowed) and total participants (the same participant may be counted more than once). Another example is program cost, which sounds straightforward. It requires the user to understand how to calculate and use the fully-burdened labor and related rate to cost out the hours staff spend on learning development, delivery, and management.

Two more commonly used metrics fall into the category of easy-to-understand but complicated to measure. Level 2 learning, for example, sounds simple. Just calculate the average test scores, and you are done. In practice, however, many participants must keep taking the test until they receive a passing score. In this case, the final score doesn’t tell you much about how well the course or the test is designed. If you want to know whether a problem needs to be addressed, you need to report the score on the first attempt (or first-time pass rate) or the number of attempts required to pass.

Likewise, the Level 3 application rate sounds easy. Just ask participants if they applied what they learned. In many cases, however, the amount of content dictates that it would be better to ask the percentage of content they applied using a decile scale from 0%-100%. Then there is the issue of when to ask. Ideally, you will have a question about intent to apply as part of the post-event survey and a question about the actual application in a follow-up survey two to three months later.

Rounding out the challenge of being knowledgeable about measures is the category of metrics that are hard to understand, let alone measure. I think Level 4 impact falls into this category. To make life interesting, we have two very different versions of Level 4 in learning. Kirkpatrick defines Level 4 as results, meaning the business results and he advocates creating a compelling chain of evidence to show that learning contributed to the results. Phillips defines Level 4 as the isolated impact of learning—many in the profession don’t believe this can be measured. Jack and Patti Phillips provide five methods to isolate the effect of learning from all other factors, but some of these require statistics and good experimental design, which many don’t have.

2. The Number of Measurement Metrics Can Be Overwhelming

My second reason to reconsider the ease of L&D measurement is the sheer number of metrics. We have about 200 metrics for L&D alone and more than 700 for HR. This is a lot for anyone to master.

3. There Are Not Many Good Coaches/Teachers on the Topic of Learning Measurement

My third reason is the measurement staff and their leaders. Most are not exposed to L&D or HR measurement at university, so they learn it on the job. For them to learn on the job, they need a good teacher. True, many take workshops and read 100+ books on the topic, but this is usually not enough to master the concepts and practice of sound measurement. They need a good teacher or coach in their workplace to show them how to apply what they have learned and answer all the real-world questions that are going to come up.

Sadly, in many organizations, there may not be a good coach or teacher to help the person new to L&D measurement. Without this coaching, the staff is unlikely to master the measurement practice truly. Consequently, they will not be good coaches for those coming after them. So, as a profession, we do not seem to have reached a sustainable equilibrium where the experienced can teach and mentor the less experienced. In other words, those who should know either don’t know or don’t pass on what they know. This may explain why we continue to talk with so many who have only the most basic understanding of measurement and reporting, even though thousands have taken workshops and read books.

What do you think? Is good measurement difficult? What can we do about this?

We will continue this discussion at our Virtual Conference on November 2^nd. Join us to learn more and share your thoughts.