CTR Exchange and Blog

Future-Focused Training: Are You Committing Learning Malpractice?

This article is reprinted from Chief Learning Officer, January 10, 2022

The temptation is strong. Over the past year many articles, webinars, and conference sessions have highlighted the need to focus on the skills your future workforce will need. Moreover, your CEO may be pushing you to provide training to meet these future needs. After all, who wants a workforce with outdated skills? Most would say it just makes sense to start this important training now. But does it?

My answer is no, and I believe many who proceed down this path are guilty of learning malpractice. First, it is very difficult to identify true future needs. Second, even if future needs are correctly identified, the participant cannot immediately apply their learning since the need does not exist today. This second objection cannot be overcome, and I assert that any training which cannot be immediately applied is a clear case of learning malpractice. Let’s examine each issue in more detail.

Identifying future skills, knowledge, and behavior is hard. Very hard. First, you have to define the “future”. Most writers seem to talk in terms of two to five years so let’s go with that. Second, how would you go about identifying the skills needed in two to five years? Many of us struggle today to have productive performance consulting discussions with our stakeholders about their current performance needs.  Can you imagine having this discussion about their performance needs two-five years out? Remember, we are not talking about what their employees need to be able to do today—we are talking about what new skills and knowledge they will need to meet future requirements. Most will not be able to identify new organization-specific skills needed in two to five years.

Consequently, many practitioners instead rely on survey results about future skills and recommendations by consultants and vendors. You have seen these lists. They are not organization-specific and they highlight primarily soft skills like leadership, creativity, critical thinking, teaming, and presentation skills although the lists generally include a few “hard” skills like data literacy. But are these truly new skills needed for the future workforce? Aren’t these very same skills needed right now? In fact, I have not seen a generic list of “future skills” which is different from much needed “current skills”, although it is certainly possible that some organizations could identify a few specific new, hard skills.

For the sake of argument, let’s suppose we could identify skills and knowledge not needed today but very much needed in two to five years. Now we run into an insurmountable problem and that is application. As a profession, we already struggle with low application rates or conversely high scrap rates. Recent research by Jack and Patti Phillips shows a scrap rate of 80% for learning during the pandemic. That is 80% – not 8%. Even prior to Covid, researchers were reporting scrap rates of 40%-80%. So, even with learning which can be applied immediately to meet an existing need, most learning today is not applied on the job.

What application rate could we expect for learning designed to meet a future need which by definition cannot be applied until some future date? The answer is simple – about 0%. Participants will have forgotten what they learned before they have a chance to apply it. In other words, the scrap rate will be 100% and the learning will be a complete waste of time and effort This is the very essence of learning malpractice.

In summary, identifying truly new skills and knowledge needed two to five years in the future is nearly impossible, and even if it were possible, the application rate will be 0%.  Therefore, it is learning malpractice to provide training today to meet a need 2-5 years in the future. Instead, your first priority should always be to meet today’s needs and here we must focus on raising the application rate. Your second priority should be to emphasize training that will become increasingly important in the near future but which can be applied today. If you agree, let’s try to reset the conversation so we have greater clarity and transparency about what we can and cannot realistically accomplish.

Simplify Your Measurement of Leadership Development

Many find measuring the impact of leadership development especially difficult. They try to tie leadership training results to organizational goals which may be impacted by leadership. Unfortunately, this is a long list since one can reasonably argue that better leaders should contribute to achieving ALL the organization’s goals, including higher sales, greater productivity, more cost reduction, better quality, higher employee engagement, and lower turnover. I have never seen anyone do this successfully since the impact of leadership on all these goals is indirect and very difficult to quantify.

There is a better and much easier approach. Start by stepping back and asking why you are doing leadership development. If the answer is to address one goal, like sales, then measure its impact as part of your suite of learning programs to improve sales. In this case, you should have designed the leadership training specifically to help achieve that one goal (like sales), and the target audience would be only those leaders in sales.

Chances are your needs analysis also found other ways learning could help achieve the sales goal so it is likely you are doing more than just leadership training to improve sales. When it comes time to measure impact, ask the participants (and perhaps their bosses) what impact the suite of leadership training programs had on the improvement in their sales. If you are using the Phillips methodology, there are five methods you can use to isolate the impact of learning from other factors.

More often, the leadership program is developed to help improve leaders across the organization. In other words, the leadership training is not developed to help achieve a single goal but instead to increase leadership competency in general which should help achieve all the goals. In this case, the best approach is to measure the impact directly on leadership. Ask the leaders whether the training improved their leadership, but also ask their employees and the leader’s boss whether they saw an improvement in the participant’s leadership. The key is to measure impact as directly as possible.

In addition to these measures, most medium and large organizations have some version of an employee engagement survey and most of these surveys have questions about leaders. At Caterpillar, we had a 50-question survey with seven questions about leaders. Create a leadership index based on the leadership questions and use this index to gauge the improvement in leadership.

If your leadership program is well designed to address your organization’s leadership needs, the index should improve after leaders have completed the program. If you wish to isolate the impact of the leadership program on the change in the index, then you could use a naturally occurring control group (if available) or ask the participants and their bosses to estimate how much of the change in the index was due to the leadership training.

If you would like to calculate an ROI on the program, you could easily ask each leader to identify an increase in income or productivity or a decrease in cost. Then you would apply the Phillip’s methodology to convert the impact to dollars and calculate the ROI.

There is another method, which is easier at scale, which can work if a goal of the leadership training is to make leaders more efficient in terms of interactions with employees. Better leaders often will conduct more efficient staff meetings and require less time helping employees set goals and review performance. Simply add a question about time saved due to better leadership to the survey given to the participants and their employees. Then find the value of the time saved by multiplying the hours saved by the labor & related rate of the participants. (Needless to say, don’t use this method if the goal of your program is for leaders to spend more time with their employees!)

Bottom line, take the most direct approach possible to measuring leadership impact. This will simplify your measurement and will be much easier than trying to measure impact on all the organization’s goals.

The L&D Industry’s Answer to Measuring Leadership Development

by Kent Barnett, CEO, Performitiv

I think we would all agree that as an industry we do a very poor job of measuring, communicating, and improving the value of leadership development programs. I think most, if not all of us, would also agree that leadership development is one of the most strategic business processes in our respective organizations.

So, why is it that we do such a poor job in this area? The answer is because it’s hard and confusing. The good news is that dozens of leading learning organizations and experts have come together to create a systematic framework to measure leadership development. It is designed to work for large and small organizations and be flexible and easy to get started.

As may of you know, the Talent Development Optimization Council (TDOC) was created over threee years ago to address two primary issues:

  1. Better Communicate Learning’s Value
  2. Optimize Learning’s Impact

TDOC created the Net Impact System (NIS) to provide a systematic approach to address these two issues. Organizations that have started to apply the NIS principles are seeing huge gains in business results, effectiveness, scrap reduction, outcome indicators, and Net Promoter Score. TDOC is now focused on leadership development.

Attend Kent Barnett's Session, Measuring Leadership Development

Join Kent Barnett at the 2021 CTR Annual Conference where he will teach you how to build  world-class dashboards and management reports for leadership development. If you are interested in learning how to use metrics to demonstrate value, then you can’t miss this event!

When: November 2, 2021
Time: 3:00 – 3:35 pm (EDT)
Speaker: Kent Barnett, CEO, Performitiv

Register for this Event Now!

About Performitiv

Performitiv provides the technology, methodology, and expertise to help understand learning effectiveness, business impact, and continuous improvement opportunities. Their technology helps to streamline, automate, and upgrade evaluation processes to save time, simplify data collection, and improve the overall effectiveness of L&D operations.

Make or Buy? Not as Simple as It Sounds

We are often faced with a decision whether to design, develop and deliver a course in-house or pay an outside party. This is a classic “make or buy” problem which turns out to be more complicated than most think. Since some critical costs are often omitted in the calculation of the make option, many practitioners are choosing the make option when the buy option is actually less expensive. Of course, there are other reasons than the cost for developing a course internally, but we will only focus on cost for this article.

I’ll begin with an example. Suppose I want to develop and deliver a two-hour course which will take 100 hours of staff time to design, develop, and deliver. The labor cost for the 100 hours is $30 per hour for a total of $3,000. Assume the instructor travel expense is $1,500 and materials are an additional $500. We will assume there is no room rental charge and no other direct expenses. Furthermore, suppose I have a vendor willing to deliver the same course for $7,500. Based on cost alone, which option should I choose?

At first glance, the answer appears obvious. The internal cost comes to $5,000, and if I choose this option, I save $2,500. Is that the correct decision though? Maybe not. First, the $30 per hour may represent only salary neglecting benefits like the employer-paid portion of FICA, health care, pension, unemployment, and other costs associated directly with the employee’s salary. These are called “related costs” and typically are about 25 percent of the hourly rate for management employees. Adding these in, we have a labor and related rate of $37.50 per hour and a labor and related cost of $3,750. You may be thinking that even with the additional $750, the decision still favors the make option, but we’re not finished.

We also need to add the burden rate and here is where it gets complicated. The burden rate takes two things into account. First and the more obvious of the two, burden accounts for the overhead required to support the department. This includes office rental space, utilities, computers, copy machines, office supplies, telephone, etc. as well as items like travel for conferences and education, subscriptions, fees, and consulting not directly related to a course. We include as overhead only those costs not directly associated with or attributable to projects.

This cost needs to be spread across all of the hours attributable to direct work. Typically, an employee works around 70 percent of their 2080 annual hours (52 weeks x 40 hours per week). The rest is spent on holiday, vacation, sick time, staff meetings, performance reviews, general planning, training and development, managing others, etc. So, we divide the total overhead burden for the department by the sum of the attributable hours, not the sum of all hours. This is often close to the average hourly rate, so for this example let’s say it’s $25.

But, we’re still not finished adding internal costs. The second component to burden is the cost of the non-attributable hours discussed above. In other words, what are the labor and related cost of the 30 percent of total hours employees do not spend directly working on task? Employees still need to be paid 40 hours per week even though they are not working on task for all of these hours. This money has to come from somewhere. Those of you who are consultants understand this well. You have to generate enough revenue to cover your time when you’re not on a billable project. This becomes an important part of burden and is not a small number. We divide this additional burden by the number of attributable hours to get the second component of the burden rate. Let’s say this come to $15 per hour.

In this example, the two components of burden add up to $40 per hour, which is more than the labor rate and not unusual. Now, our fully-burdened labor and related rate is $30 + $7.50 + $25 + 15 = $77.50 per hour, which is more than double the labor-only rate of $30 per hour. This pushes the make option to $7,750 for labor plus travel ($1,500) and supplies ($500) for a total cost of $9,750. It turns out that the less expensive option is to buy rather than make.

As you can see, there is more to the calculation than at first glance. Are you making these calculations in your organization? Do you know your total burden cost and the number of attributable hours? Many don’t and consequently think they are saving their organization money by developing and delivering courses using internal staff when it would be more cost-efficient to hire out.

Attend the Make or Buy Session at the CTR Annual Conference

I’ll be covering make or buy in more detail during the CTR Annual Conference on November 3. Click here to learn more and to register. Hope to see you there!

Why Does L&D Measurement Remain So Difficult?

Why does L&D measurement remain so difficult? Put a different way, why hasn’t the profession made greater progress over the last ten years? ATD research shows clear progress was made from 2001 -2009 in the percentage using higher-level program evaluations (3, 4, and 5), but little has changed since then. We continue to hear from a majority of colleagues how difficult measurement is.

Many explanations have been offered, however, the most common is that practitioners lack the knowledge. They don’t know what to measure, how to measure, and what to do with the measures once they have them. For program evaluation, though, this is hard to understand because there are so many books written and workshops on the subject.

At a broader level, however, I think the profession has lacked a comprehensive framework that addresses how and what to measure for all the reasons to measure—not just program evaluation. We know that most measurement and reporting activity is focused on informing and monitoring, not on program evaluation. Just think of all the scorecards and dashboards you use; these are not for program evaluation. And there has been no guidance whatsoever on reporting and on how to choose the most appropriate report based on the user and reason to measure.

Talent Development Reporting Principles (TDRp) were created specifically to meet the need for an overarching measurement framework. Peggy Parskey and I wrote the book Measurement Demystified: Creating Your L&D Measurement, Analytics, and Reporting Strategy to share this framework and guidance. Our hope is that practitioners will now have the answers to their questions about what and how to measure, and how to report their results once they have them.

Knowledge and lack of a comprehensive framework, however, probably don’t explain all the discomfort with measurement. For some, measurement implies accountability, and many don’t want to be accountable for results. After all, what if we measure and find that application rates are low, impact is negligible, and ROI is low or negative? What if we find we have poor rates of on-time completion for development and delivery or a high cost per learner? This is a culture issue within L&D and more likely within the entire organization that cannot be solved by reading books or attending workshops. Leaders, starting with the CLO, need to make it clear that they want to measure in order to deliver better results and learn what is working so that opportunities for improvement can be identified.

Even if the issues of knowledge and culture are addressed, there may still be a hesitancy to measure and report due to a lack of confidence. This is typical for analysts and others new to their position who simply don’t have much experience with measurement and reporting. They have read the books and attended the workshops, and they may have a supportive culture, but they just aren’t sure about the right measures to use or how to calculate and report on them. Unfortunately, the only way to gain confidence is to do the work. Mistakes will be made, but we all make mistakes and the goal is that we learn from them. A supportive boss or a good coach or mentor can help, but often that is difficult to find.

Another possible explanation is turnover. Like other professions, it takes years to acquire all the practical knowledge needed to excel at a job. I am speculating, but could it be that those responsible for measurement and reporting don’t stay in the position long enough to truly master it with confidence?

Finally, I’ve heard that measurement and reporting is difficult because they don’t have the resources to do a good job. They have a small budget for measurement and not enough staff. In many cases, this reflects a lack of appreciation for measurement and reporting by the CLO and other senior leaders which in turn may reflect a lack of knowledge or accountability on their part.

By the way, these same issues apply across all of HR regarding measurement and reporting, so it is not just L&D.

If you’re committed to making measurement and reporting a part of your L&D strategy, please join us at our virtual conference November 2-4 to explore this issue in more detail. Kimo Kippen will lead a panel discussion on this very topic on November 3 at 2.45 pm ET, and Michelle Eppler from the Human Capital Lab at Bellevue University will share results from a survey she just completed to probe the reasons. Registration is free.

Mid-Year Check Up on Measurement and Reporting

Since we are well into the month of July, I thought it the perfect time to reflect on what we have accomplished so far this year in the world of measurement and reporting.

If you are running learning like a business, that means you set specific, measurable plans or targets for your key measures. How are they doing compared to the plan? Are they where they need to be if you are going to make the plan by the end of the year? If you have started to provide forecasts for how the year will end, you can just compare the plan to the forecast. If you are not yet at the stage of making forecasts, just ask yourself if you are on track to make the plan by the end of the year. Even if it appears you will not make the plan on your current trajectory, the good news is that you still have six months to do something about it. What steps can you take now to get back on plan?

Even if you did not set plans for measures, this is a perfect time to see what you can learn from the first six months. Here are some measures to check:

  • Completion rate: Are you getting the completion rates for formal learning programs that you would expect? Are there significant differences between programs or audiences?
  • Reach: Are you reaching as many employees with learning as you hoped? What percentage of employees have been enrolled in formal learning? How many have engaged in some informal learning?
  • Cost: Do you know what your costs are for the first six months? Are you on budget? Where are the surprises and what can you learn from them?
  • Total Participants: While reach measures whether an employee has taken at least one course or touched at least one informal learning asset, total participants allow for employees to take more than one course or interact with more than one learning asset. So, your reach may be meeting expectations while total participation is not. Are you satisfied with your total participation?
  • Usage: Closely aligned to participation is your usage of online and virtual courses as well as informal learning like communities of practice, portal content, and performance support tools. Are you getting the usage you want?
  • Level 1: Is your participant’s reaction to the learning at an acceptable level? Did it drop when you switched from ILT to VILT? How do your L1 scores compare for VILT and online? Are you measuring L1 for your informal learning assets? How do they compare to formal learning? Where are your opportunities for improvement?
  • Level 2: Are you satisfied with the test scores for your VILT and online courses? Are the VILT scores lower than you were getting for ILT? If so, does the content or instruction need to change?
  • Level 3: Are your application rates where they should be or do have too much scrap learning? Did L3 decline during COVID? If it did, what could you change to improve?
  • Employee Engagement: If you are measuring employee engagement quarterly, has employee engagement with their learning changed with COVID? Are employees happy with the learning opportunities available to them? If it dropped initially after employees began to work from home, has it come back to an acceptable level?

These are just some of the questions that you should consider answering with your data. Remember that a key aspect of running learning like a business is learning from the data. Mid-year is a perfect time to see what you can learn from the first half of the year and then use that knowledge to make improvements in the second half for a strong finish.

Are We Over Using Dashboards?

Dashboards have become increasingly popular, especially those with well-designed visual elements. For many applications, they do represent a great advance from the more boring scorecards filled with rows of data. That said, the question now is whether we have gone too far and are relying too much on dashboards when in fact other types of reports would be better. My answer is yes.

Many practitioners today appear to believe that dashboards are the best, if not the only, way to share data, and this is a problem. It would be better for us as a profession to utilize many different types of reports and tailor the type of report to the specific need of the user, which in turn requires us to think more carefully about the reasons for measuring in the first place. We describe four broad reasons to measure in our new book, Measurement Demystified, and each of these four reasons is linked to the type of report best suited to meet the user’s needs. The dashboard is suited to only two of these four reasons.

The first reason to measure is to inform. This means the measures will be used to answer questions from users and discern if trends exist. The question may be about the number of participants or courses, or perhaps about the participant reaction or application rate. In any case, the user just wants to know the answer and see the data. If they want to see it by month and especially if they want to see subcategories (like courses by type or region), a traditional scorecard will be best with rows as the measures and columns as the months. If the user is interested in year-to-date summaries and more aggregate data as well as some visual representations, a dashboard will be best. So, even for this one reason (inform), the best report depends on what the user wants to see. These may be one-off reports or reports that are regularly updated.

A second reason to measure is to monitor. This occurs when a user is happy with how a measure is performing and wants to ensure the value remains in an acceptable range. For example, participant reaction scores may average 80% favorable and the CLO wants to ensure they stay above 80%. In this case, a dashboard with thresholds and color coding is a perfect way to share the measures. This may be the only element in the dashboard or it may be combined with some other elements. This type of dashboard should be generated monthly.

The third reason to measure is to evaluate a program and share the results. In this case, a program (like sales training) has been completed and the users desire to evaluate and share the results with others. This is a one-off report designed to be used at the end of a program or perhaps at the completion of a pilot. In this case, a dashboard should not be used. Instead, a program evaluation report would be best which takes the audience through the need for the training, the planned results, the activities completed, the actual results, and lessons learned. The report will probably be a PowerPoint but could be a written document.

The fourth broad reason to measure is to manage. In contrast to monitoring, managing means that a goal has been set to improve the value of a measure, perhaps increasing the application rate from 40% to 60% or reaching an additional 2000 employees with learning. If monitoring is about making sure the status quo is maintained, managing is about moving the needle and making progress. In this case, a dashboard should definitely not be used because it would not convey the key information or the detail needed to make management decisions.

Every month a manager needs to know whether their efforts are on plan and whether they are likely to end the year on plan. For this, they need a management report which includes the plan or target for the year, year-to-date results, and a forecast of how the year is likely to end if no additional actions are taken. This type of information is very difficult to share in a dashboard format which is why special-purpose management reports have been designed for L&D. These are generated monthly and focus on both specific programs and aggregated department results. In contrast to dashboards and program evaluation reports, these management reports are not meant to be “presented” but to be used in working sessions to identify where action is required.

In conclusion, dashboards have their place but should not be the only type of report generated by L&D. Dashboards are recommended in two cases: 1) when the reason to measure is to inform and the user wants summary data along with visual elements, or 2) when the reason to measure is to monitor in which case thresholds will need to be included. Dashboards are not recommended when the reason to measure is to inform and the user wants detailed, monthly data. In this case, a scorecard is preferred. Nor is a dashboard recommended to share program evaluation results or to manage programs or the department. In each of these cases, better report types exist and should be employed (program evaluation and management report respectively). Bottom line, it is important to use the right type of report which should match the reason for measuring.

When it Comes to Learning Measurement—What is Good Enough?

When it comes to learning measurement—what is good enough?

This is a question that needs to be asked much more frequently. This is especially true in the case of isolating the impact of learning (Phillips’s level 4) or talking about the accuracy of an ROI calculation. Context is key. Business economists think about the answer to this question all the time because the one thing they know for sure is that their forecast (for GDP, sales, housing starts, commodity prices, exchange rates, etc.) will be wrong. Only by chance will the forecast be exactly right. So, the question is not whether the forecast will be exactly right but rather will it be close enough to make the right decision.  For example, should we raise production, should we hire more workers, should we invest in A rather than B?

We need to apply this same type of thinking in learning. We need to start with the context and the reason for measuring. What decision are we trying to make or what will we do with the estimate once we have it? Given the answers to these questions, how close does our estimate of impact or ROI need to be? I cannot think of a single instance where the estimate needs to be perfect. It just needs to be good enough to help us make the right decision or take the right course of action.

So, let’s step back for a minute and ask why we might estimate impact or ROI? First, I think we would all agree that we want to identify opportunities for improvement. If this is the context, how accurate does our estimate of impact need to be?  In this case, the estimate just needs to be roughly right or “in the ball park”. For example, if the true (but unknown) ROI is 20% we would like an estimate to be in the 10%-30% range. Typically, we would conclude that an ROI in this range has opportunity for improvement. Similarly, if the true ROI were 100%, we would want our estimate to be in the 70%-130% range, and we would likely conclude that no improvement is necessary.

Will the standard methods to isolate impact (control group, trendline analysis, regression or participant estimation methodology) be good enough for this purpose?  I believe so. We simply need to know whether improvement is required, and we want to avoid making improvement when non is needed or failing to make an improvement when improvement is needed. In other words, if the ROI is truly 10%, we don’t want an estimate of 100% And vice versa. The standard methods are all good enough for us to make the right decision in this context.

Now, suppose the reason to measure impact and ROI is not for improvement but to demonstrate the value or effectiveness of the program. At a minimum we want to be sure we are not investing in learning that has no impact and a negative ROI. This is a bit more demanding but the same logic applies. We want the error margin around the estimate to be small enough that we can use the estimate with confidence. For example, if the estimate for ROI is 10%, we want to be confident that the error margin is not plus or minus 10% or more. If it is, we might conclude that a program had a positive ROI when in fact it was negative.

In this context, then, we want to be more confident in our estimates than in the first scenario. Stated differently, we want smaller error margins. We will use the same four methods, but we need to be more thoughtful and careful in their use. We would have the most confidence in the results obtained using a control group as long as the conditions for a valid control group are met, so extra care needs to be taken to make sure the control group is similarly situated to the experimental group. Trendline and regression also can produce very reliable estimates for the “without training” scenario if the data are not too messy and if the fit of the line or model is good. All three of these methods are generally considered objective and, when the conditions noted above are met, should produce good estimates of the impact of learning with a suitably narrow error margin.

The participant estimation method is the most widely used because no special statistical expertise is required and because often there are no naturally occurring control groups. However, it does rely on the subjective estimates of the participants. Accordingly, we will want to be sure to have 30 or more respondents and, ideally, we will obtain their estimates of impact about 90 days after the training. It is also critical to adjust their estimate of impact by their confidence in the estimate. When this methodology is used as described by the Phillips, it, too, should produce estimates reliable enough to be close to the actual but unknown impact and ROI.

The common theme in both scenarios is good enough. At Caterpillar we conducted about three impact and ROI analyses per year using the participant estimation method. We used the results to both show the value of the programs and to identify opportunities for improvement. We always presented the results with humility and acknowledged the results were estimates based on standard industry methodology with adjustments for self-reported data. We had confidence that the estimates were good enough for our purposes and we never received any pushback from senior leadership.

So, remember that your results do not need to be perfect, just good enough.

Where Measurement and Reporting Strategies Go Wrong

measuring and reporting

by Peggy Parskey, CTR Assistant Director

Imagine you are a manager in a Learning and Development function in your company. Each month, you receive several reports (or links to a dashboard) with a plethora of data. The reports focus on the function overall, but with a bit of digging you can find the data relevant to you. While the reports and dashboard provide a lot of information, you lament that they don’t help you manage your operation. Beyond the inadequacy of the reporting, you believe the organization doesn’t measure the right things or provide enough insights about improvement opportunities. You suggest to your CLO that the organization needs a measurement and reporting strategy.  The CLO agrees and delegates the job to your measurement person if you have one, or lacking that, to you. (Since you asked, you own it.)

Having never created a measurement and reporting strategy, you conduct a Google search.  Before November 2020, your search would have surfaced ads for product or service companies at the top, followed by a few white papers from these same companies or blogs with best practices and advice. Some content focused on measurement while others only addressed reporting.  Few, if any, provided a checklist of elements to include in your strategy. Lacking any meaningful guidance, you cobble something together from the disparate pieces of information found on the web.

This theoretical manager exists in organizations across the globe. Over the past 5-10 years, I have reviewed dozens of client-generated strategies. Most have some of the components of a strategy but nearly all lack the critical elements to advance measurement capability in the organization.  In November 2020, David Vance and I published Measurement Demystified with the express purpose to provide practical guidance on how to create a robust measurement and reporting strategy.

In this post, I’ll give you an overview of what comprises a well-designed strategy. But before sharing that with you, I’d like to review what a measurement and reporting strategy is not. Each bullet below represents something I’ve seen over the years as clients have shared their strategies.

  • Focused on reporting rather than measurement and reporting: You need both: what and how you measure as well as what and how you report. Reports are an output. Without the appropriate inputs (that is, the measures), you will be challenged to meet user needs.
  • Kirkpatrick’s four levels or Phillip’s five levels: Both frameworks provide important guidance and processes to select and report effectiveness measures. However, neither framework addresses the efficiency measures you should choose. Moreover, while both address outcome measures, they don’t provide detailed guidance on when and how to use them. They are an essential element of your strategy, but they are not the strategy.
  • One size fits all approach to measurement: To avoid implementing an unsustainable complex approach to measurement and reporting, many organizations have opted to go in the opposite direction and adopt an overly simplified approach. They identify a single suite of measures, methods, and reports across all programs regardless of their purpose. While this approach may meet the needs of some users, inevitably, it will frustrate others who lack the data to manage their operation.
  • A focus on one type of report: Dashboards are extraordinarily popular as a reporting tool and enable “speed of thought reporting”. The data is up-to-date, users can filter the data, and they can drill down to look for root cause of specific program issues. But dashboards can only take you so far. A robust strategy adapts the reporting to the user needs and their reasons for measuring.
  • A tool to gather the data. If you are a large organization, you will need a tool to enable you to aggregate and disaggregate large volumes of data. The tool is not the strategy.

Now that you have insight into what a measurement and reporting strategy is not, let’s turn to what it is.

  1. Begin your strategy with a clear articulation of why you are measuring. Different purposes to measure influence what you measure and what, how and when you report.
  2. Next, identify your users and their needs. Who will consume the data? What do they need? What decisions might they make? What actions might result from the information you provide? A strategy needs to be grounded in their requirements.
  3. Specify the measures overall that you will use. However, don’t stop there. Identify the specific measures for key learning programs or department initiatives. Include a balanced suite of measures including efficiency and effectiveness measures for all programs and outcome measures for strategic, business-aligned programs.
  4. Define your data collection approach. Where will you get the data? Where should you consider sampling? How will you ensure that you get sufficient responses to make meaningful inferences from survey data? Where can you automate to reduce effort, increase data reliability, and improve speed time to insight?
  5. Specify the types of reports you intend to use. When will you employ scorecards vs dashboards? When should you use program evaluation reports or management reports?
  6. Plan how and when you will share reports. What are the decision-making cadences and how can you align reporting to them?
  7. Finally, define the resources you need to execute and sustain the strategy. Be clear about what funding, capability, and tools your organization will require to build sustainable measurement.

Creating a measurement and reporting strategy will take time and effort.  You will need to meet with the CLO as well as senior L&D leaders and perhaps key business goal owners. The payoff for this effort will be significant and will enhance the value the L&D function delivers to the organization.  Don’t hesitate to reach out to us at the Center for Talent Reporting. We are here to help you in your journey.


Measurement Demystified: Creating Your L&D Measurement, Analytics, and Reporting Strategy

As some of you may be aware, the title of this blog is the name of the book Peggy Parskey and I recently published with ATD. It is the culmination of all our work since 2010 to create a framework and a set of standards and best practices to make it easier for L&D professionals to create a measurement and reporting strategy. It is also the result of input, feedback, and suggestions from the thousands of professionals who attended our webinars, workshops, and conferences. So, in essence—the book is a joint effort by everyone to advance the profession.

For those unaware of the backstory of the book, here’s a little history…

It all began in September 2010 at a CLO Symposium networking event. Kent Barnett, at the time CEO of Knowledge Advisors, and Tamar Elkeles, at the time VP of Organizational Learning for Qualcomm, were discussing the state of our profession. They both agreed that the time had come for measurement and reporting standards— similar to what accountants have in the Generally Accepted Accounting Principles (GAAP). Accountants go to university and learn about the four types of measures (income, expense, assets, and liabilities), the specific measures in each category, how to calculate the value for each measure, and what to do with the measure once they have its value (i.e., in what report it should be included and how it should be used). They wondered if a similar framework would benefit L&D.

Kent and Tamar set forth to form an advisory council to pursue their vision and ideas. They engaged leading practitioners as well as thought leaders in the field, about 28 in all. They also recruited both Peggy Parksey and me to lead the effort. We produced a draft document in early 2011, which would end up going through over 20 iterations as we incorporated feedback from the council and others. The final document was completed later in 2011, and we then began to expand the principles to all areas of HR, which was completed in 2012. The standards were named Talent Development Reporting Principles (TDRp). Similar to GAAP, TDRp provides a framework for measurement and reporting for L&D.

By mid-2012, the foundational work was complete, and TDRp needed a home. We established the Center for Talent Reporting (CTR) as a 501c(6) nonprofit organization in August of 2012 to continue to develop and promote the principles of TDRp. We hosted our first conference and our first workshops the following year. We also started our monthly webinars and blogs. And we never turned down an opportunity to speak and share TDRp with others.

It’s hard to believe that we have been at this for ten years. We’ve learned a lot along the way, so we captured all of our learnings in the book, Measurement Demystified: Creating Your L&D Measurement, Analytics, and Reporting Strategy. It is our intention that the book reinforces the standards of the TDRp framework and will do for the L&D profession that GAPP has done for accounting, which is to provide a common language; categories, names and definitions of measures; standard reports; and guidance on what to measure and how to report.

The TDRP framework begins with four broad reasons to measure: inform, monitor, evaluate, and manage.  There are many more detailed reasons to measure, but we believe a framework should have no more than five elements, or it becomes too difficult to remember and use. These four categories make it easier for us to have discussions about the reasons to measure, which is the starting point for any measurement strategy.

TDRp recommends three categories for measures. Historically, practitioners had divided measures into efficiency and effectiveness buckets which is what I did when I worked at Caterpillar. However, research and practice showed the need for a third category, not because of the number of measures in it but because of its importance. The third type of measure is “outcome,” which means the results or impact from the learning on the organization’s goals. This is the type of measure CEOs want to see the most, and yet it is seldom reported. This is the measure required to make the business case for learning.

Lastly, TDRp identifies five types of reports which contain the measures. Moreover, TDRp ties the type of report to the reason for measuring, which provides much-needed guidance for practitioners. Scorecards and dashboards are great for informing and can also be used for monitoring if thresholds are included. Program evaluation and custom analysis reports are ideal for sharing the results of a program or research. Management reports are the fifth type, and these have a special format to help leaders manage programs and initiatives to deliver planned results. They come in three varieties depending on the user and the need, but all have the same format, just like the basic accounting reports do.

It is our sincere hope that the TDRp framework and the guidance in Measurement Demystified will help the profession advance. Some are just beginning and should benefit significantly from a framework and guidance on how to select measures and reports. Others are further along and can benefit from the definitions of over 120 measures and the advanced report formats. Even experts may benefit from the detailed discussion of how to create plan numbers, use year-to-date results, and create forecasts.

And we are not done yet. We just submitted our draft manuscript for Measurement Demystified: The Field Guide to ATD for publication this December. We provide over 100 exercises to improve your understanding of the concepts and to give you practice in applying them. Hopefully, the Field Guide will significantly increase your skill level and your confidence, enabling you to create a much more robust measurement and reporting strategy.

We look forward to your continued engagement and feedback.

Running Learning Like a Business

The concept of “running learning like a business” continues to gain traction. It means different things to different people but in all cases involves bringing a more business-like perspective to the L&D function. Personally, I like to focus most on its implications for the actual management of learning programs and the L&D department.

So, you ask, what does this mean specifically? What would someone do differently? How can you tell if someone is running their operation like a business? I believe there are two primary elements to running learning like a business. First, you need to set specific, measurable goals or targets for every important measure. This becomes your business plan for a program or the entire department. This plan should be created just before the start of your fiscal year or no later than the first month of the new fiscal year. Second, once you have a business plan and the year is underway, you need to compare progress against plan every month and answer two key questions: 1) Are we on plan year to date? and 2) Are we going to end the year on plan? If you are not on plan or if it appears you may not end the year on plan, then you need to discuss options to get back on plan and decide whether to take corrective action. This is what we mean by disciplined execution.

Let’s look at each in more detail. First, you need good plan. Of course, there is lot that goes into a good plan beginning with an understanding of the organization’s goals and discussions with goal owners about whether learning has a role to play. Think of this as proactive, high-level performance consulting where you seek out the senior leaders and engage in good, open discussion to explore the possibilities for learning to help them achieve their goals. Learning will not always have a role, but at Caterpillar we were able to contribute to each of the CEO’s top seven goals each year.

If learning can make a contribution, then you need to reach agreement with the goal owner (like the SVP of Sales) on what the learning program will look like: target audience, timing, learning objectives, type of learning, design, etc. More than that you and the goal owner need to agree on measures of success (like the impact of learning on sales) and on targets for key efficiency and effectiveness measures to deliver the agreed upon outcome (like impact of learning). This would include agreeing on plans for number of participants, completion dates and rates, learning (level 2), and application (level3). This needs to be bolstered by agreeing on roles and responsibilities – what each of you needs to do to deliver the planned measures. For example, how will the goal owner communicate the importance of the learning before the program and reinforce desired behaviors after the program. (We had a written roles and responsibilities document at Caterpillar signed by both parties.)

Second, once the program is underway, you will begin to receive monthly data on the measures you and the goal owner agreed were important. You will need a monthly management report (called a program report) to compare year-to-date results to plan. If you are not on plan, you need to understand why. Perhaps it is just a matter of timing (for example, it took longer to launch than planned) and left alone the program will get back on track and meet plan expectations. This is why the forecast for how the year is likely to end is so important. If the forecast shows you delivering plan, no further action is required. However, if the forecast shows you falling short of plan by year end, then you need to consider taking corrective action as soon as possible. This may involve actions such as redirecting resources toward this program or coming up new plans for the goal owner to reinforce the learning.

Our focus so far has centered on what we call strategic programs. These are programs directly aligned to the goals of the CEO. However, there are typically other programs of great importance, in some cases even more important, than strategic programs. Examples would be onboarding, basic skills training, leadership, compliance, etc. The same thinking applies. Start by creating a plan for each program with specific, measurable goals. Then execute the plan throughout the year with discipline to deliver the plans.

We have just discussed what it means to run learning like a business from a program perspective. The same discipline, however, can be applied to your CLO’s initiatives to improve results across all programs (like improving the overall application rate) or to improve internal process or system efficiency or effectiveness (like improving customer satisfaction with the LMS or reducing helpdesk wait time). In this case, the CLO needs to set specific, measurable goals for each measure she chooses to manage. These measures are captured in the monthly operations report which she and the directors can use each month to determine if the initiatives are on plan and likely to end the year on plan. If not, they need to decide what action to take.

So, this is what “running learning like a business” means to me. This the hard work of managing L&D with the same type of discipline your colleagues use in sales or manufacturing where they set annual targets and do the best they can to meet those. For many, this is a very different way of managing than they do today. Most have plans of some sort, but many are not specific and measurable. Even some who do have good plans do not execute them with discipline by using the monthly program and operations reports to compare progress against plan.

If you are managing this way today, congratulations! I know you see the value in this and ask you to share your stories with others so they can be inspired by your success. If you are not managing this way, consider giving it a try. You can start small. Try it for one program. And/or try it with the CLO to better manage a short list (3-5) of measures for improvement. I am convinced that running learning like business was key to our success at Caterpillar, including support and funding from our CEO and senior leaders.

If you would like to hear more about this concept, join me for a panel discussion on March 23 at 4.15 ET hosted by Corporate Learning Week.

2020: The Year of Disruption

I think we’ll all be happy to put 2020 behind us. As of December 18, 2020, Covid has claimed more than 300,000 lives in the U.S. and disrupted the economy, our personal lives, and learning along the way. Despite the pandemic’s destruction, some good has come from the disruption—like the SEC rule mandating human capital disclosure.

The impact of the pandemic on learning has been massive. Organizations have been forced to shift from instructor-led learning (ILT) to virtual ILT (vILT) supplemented by more eLearning and portal content. Some learning couldn’t be converted immediately, so learning opportunities and productivity gains were consequently lost. Since companies had to convert learning so quickly, much of vILT didn’t meet quality standards. But despite this, learners were appreciative of the effort and happy to have more learning than none at all.

During this time, the good news is that learning departments have demonstrated their value by offering up valuable strategic advice on how to manage remotely. This has been a huge win for many L&D departments and has dramatically increased their credibility and stature within the organization.

Just as the pandemic has caused us all to reconsider our work-life balance and the possibility of an ongoing work-from-home arrangement, it has also forced us to reconsider the mix of ILT versus vILT, eLearning, and content available in the LMS portal. Actually, this reconsideration is long overdue. Many organizations had little to no vILT before March 2020. Now, they have a lot of vILT, and the quality of the content is slowly improving as it is redesigned for virtual delivery. Live virtual training offers significant benefits, such as travel and facility cost savings. And, if properly designed, it may be able to accomplish the same objectives in less time. I don’t see any chance of organizations reverting to a pre-pandemic model that realized so heavily on ILT.

There is also an opportunity here to think even more broadly about restructuring your learning programs. Many organizations going into the pandemic had long onboarding and basic skills ILT programs. Some went on for a month or more, with a few lasting three, six, or even 12 months. Is this really the best way to learn? And how much of the content will be remembered and applied? Instead, why not use this opportunity to restructure the learning entirely? Reduce ILT or vILT at the beginning to just the basics that learners will need to get started. Supplement this with additional vILT or eLearning combined with performance support in the workstream at the time of need. For example, following the initial (now shortened) course, participants might take a short vILT course or eLearning module every month over the rest of the year. And this is combined with performance support, which is easy to access and provides them just the tools they require at the time of need.

Last, let’s not forget the SEC’s new rule mandating human capital disclosure. Over the coming years, this transparency will spread to all types of organizations and radically alter the expectation of both investors and employees for information on important human capital metrics like diversity, pay equity, leadership trust, and employee engagement. By 2030 people will wonder why this type of information was not always available. Disruption is coming, and there will be no going back.

So, take this opportunity as you plan for 2021 to continue to think outside the box, just as you have been forced to do this year. Don’t go back to the old ways even when offices reopen.  Double down on new ways of learning and embrace the opportunity to disclose your important L&D and HR metrics publicly.

Best wishes for the holidays and may everyone be healthy and safe.

SEC Publishes Final Rule on Human Capital Reporting

by David Vance, Executive Director, Center for Talent Reporting

The U.S. Securities and Exchange Commission (SEC) just published its final rule on human capital reporting on August 26, 2020. This follows the proposed rule issued one year ago. The final rule makes very few changes to the proposed rule and mandates for the first time, public reporting of human capital metrics by companies subject to SEC reporting requirements, which includes all US companies issuing stock, bonds, or derivatives. The rule becomes effective 30 days after publication in the Federal Register which should happen in September, 2020.

Today, companies have to report only one human capital metric—number of employees. The new rule will still require the reporting of full-time employees, but additionally, encourage companies to report part-time and temporary employees if they are important to a company’s financial results.

More importantly, the new rule mandates for the first time that companies provide “to the extent such disclosure is material to an understanding of the registrant’s business taken as a whole, a description of a registrant’s human capital resources, including any human capital measures or objectives that the registrant focuses on in managing the business.”

The qualifying word: “material” means anything that an investor would want to know before buying or selling a stock, bond, or derivative. The SEC goes on to specifically call out the areas of “attraction, development, and retention of personnel as non-exclusive examples of subjects that may be material, depending on the registrant’s business and workforce.”

SEC Chair Clayton commented in the public release, “I cannot remember engaging with a high quality, lasting company that did not focus on attracting, developing, and enhancing its people. To the extend those efforts have a material impact on their performance, I believe investors benefit from understanding the drivers of that performance.”

In other words, the SEC expects to see these three areas discussed and reported if they are material, and it is hard to imagine companies where they are not. Consequently, public companies will need to start disclosing and commenting on them as early as  October , 2020. And this is just the starting point. The rule calls for all material matters to be disclosed; therefore each company will need to decide what other human capital matters might be considered material by an investor. Depending on a company’s  situation, this might include total workforce cost or productivity, diversity (especially at the leadership level), and culture (revealed by employee engagement and leadership surveys). Discussion may also be required about the implementation of a new performance management system or a significant change in compensation and benefit philosophy.

Where can companies get guidance on specific metrics they might use to meet the new rule? The 2018 recommendations by the International Organization for Standardization (ISO) includes 10 metrics for public reporting by all organizations and an additional 13 for reporting by large organizations. ISO also recommends 36 other metrics for internal reporting, which are organized by area or cluster. The recommended metrics for the three SEC focus areas with metrics recommended for all organizations include:

  • Attraction: Time to fill vacant position, time to fill critical vacant positions, percentage of positions filled internally, percentage of critical positions filled internally
  • Development: Development and training cost, percentage of employees who have completed training on compliance and ethics
  • Retention: Turnover rate

These metrics provide a starting point for a company’s human capital reporting strategy to meet SEC requirements. Additional metrics are available from ISO for the area of development, including percentage of employees who participate in training, average hours of formal training per employee, percentage of leaders who participate in training, and percentage of leaders who participate in leadership development.

In addition to the metrics for the three focus areas, ISO recommends a number of others that could address areas of material concern such as workforce cost,  productivity, diversity, and leadership trust. Companies will need to identify their own metrics not addressed by the ISO and may wish to supplement with other data (such as employee engagement). They will also need to report on material initiatives which may not have metrics.

Publication of the final rule by the SEC, combined with the comprehensive recommendations by ISO, ushers in a new era of transparency in human capital which will fundamentally change the way organizations operate. And these changes will go far beyond US publicly traded companies. In 5-10 years, privately held companies, nonprofits, and other types of organizations will be compelled (or shamed) into adopting the same level of transparency.

In the future of human capital transparency, what investor will buy stock in a company that refuses to disclose material human capital information, which is already the primary driver of value in many companies? What employee will work for an organization that refuses to share its key human capital metrics because they are embarrassed to share? Why would anyone work for an organization where the culture is terrible, they don’t invest in their people, turnover is high, and there are not enough employees to do the work? Today, you can say you didn’t know. Tomorrow, you will know.

Change is coming. The new world of human capital transparency has arrived.

Learn More

The final rule on human capital reporting is part of a much larger revision to what is called Regulation S-K which governs what companies must disclose to investors initially (form S-1), quarterly (form 10-Q) and annually (form 10-K). The name of this effort, which has been underway by the SEC for four years, is Modernization of Regulation S-K items 101, 103, and 105. Item 101 governs description of the business and this is where new rules for human capital reporting are found. (Item 103 covers legal proceedings and item 105 risk factors.) Read the rule at here (pages 45-54) for human capital.

The ISO 30414:2018 Human Resources Management—Guidelines for Internal and External Human Capital Reporting is available for purchase. ISO is just beginning to release technical specifications on how the metrics are defined and used. By mid 2021, specifications should be available on all 59 metrics. The specifications for five of the eight training and development metrics are now complete and in publication. They should be available soon.

Attend Our Free Webinar On Public Reporting

Join us September 16 for our Public Reporting webinar, where cover both the SEC rule and ISO recommendations in greater detail.

Attend our Free Virtual Conference

We will be covering the subject of Human Capital Reporting at length at our Virtual Conference, October 27-29. Jeff Higgins, Adjunct Professor – Data Science(DSO), Human Capital Analytics at University of Southern California – Marshall School of Business, has been actively involved in both the SEC and ISO efforts and will be keynoting on the subject. We will also have a panel on the SEC and ISO as well as a follow-up session on the recommended metrics.

Why We Need Human Capital Metrics and Reporting in Company Disclosures

by Jeff Higgins
Founder and CEO, Human Capital Management Institute

Can we agree that the coronavirus pandemic is above all a human crisis requiring people-centric measurement and solutions?

The last great crisis in the U.S. was the financial crises of 2008-2009. If the pandemic is a uniquely human crisis, the most powerful driver in our economic recovery is what we do next with our human capital and people practices.

Given the importance of people and human capital practices in organizations, does it not make sense that such a critical resource and source of value creation be included in public disclosures?

Without standard human capital metrics, everyone is flying blind (risk planning, investors, policy makers, and corporate executives).

Human capital risk planning is a new focus for many companies. It’s a topic frequently mentioned but has seen little action beyond companies adding boilerplate legal risk language in public reporting.

Reliable, validated human capital reporting has long been a black hole for investors, including institutional investors representing employee pension funds. This may change, as investors and policy makers press for insightful evidence of risk mitigation and future sustainability including global standards like ISO30414 Human Capital Reporting Guidelines (December 2018) and US SEC proposed rule on human capital disclosure (August 2019). Recently the SEC has reaffirmed its want for enhanced disclosure and more focus on workforce health and welfare.

“We recognize that producing forward-looking disclosure can be challenging and believe that taking on that challenge is appropriate,”  explained the SEC in an April 2020 letter.

While critical for investors, human capital reporting is just as important for employees. Think about a person deciding where to work. Wouldn’t they want to know key human capital metrics to make their decision, such as the diversity of employees and leaders, the amount invested in developing employees, the percentage of employees and leaders who take advantage of training, the retention or turnover rate, and culture measures, such as leadership trust index or employee engagement score?

It is critical to note that organizations disclosing more about their human capital have historically performed better in the market.

EPIC research shows:

Reliable human capital data is often lacking yet raw data is readily available, along with proposed ISO HC reporting guidelines.

The enhanced usage of people data and analytics has been a positive business trend long before the COVID-19 crisis. But stakeholders, including executives and investors, have not always had information that was relevant, valuable, and useful for decision making.

Turning people data into decisions requires skill, effort, discipline, and money, which is lacking in many HR departments. Priorities have often been elsewhere in building employee engagement and experience metrics, along with other “hot” technology driven vendor offerings.

Even experts acknowledge that the value of human capital is not always easy to measure. Nevertheless, ISO guidelines do exist for internal and external human capital reporting. Until now, relatively little has been offered publicly in most U.S. companies.

“We may not always want to know what the data tells us,” said one member. As we have seen with the COVID-19 crisis, not wanting to know what the data tells us, can be a recipe for disaster and  not one that leads to economic recovery.

The Need for Transparency Has Never Been Greater

Many of the critical metrics and practices needed most are not transparent to the very stakeholders (such as employees, investors and government policy makers) who must start and sustain the economy recovery. 

Moving forward, enhanced transparency around workforce management and human capital measures will be needed. Without such data, organizations will have a difficult time adapting to changing markets and workforce needs and building critical partnerships. Such data is vital to building trust within the workforce and attracting, engaging, and retaining talent.

Increasingly, as organizations collect more data about their workforce, employees will want to benefit from that data.

“We saw less concern about privacy when we saw the workforce data was shared and being used to protect the workforce,” said one member.

Investing In and Developing the Workforce Has Never Been More Important

The power of learning, development, and people investment has never been greater nor more directly tied to economic results. A greatly changed world requires a new level of learning investment, adoption and innovation.  

Will organizations see the need for learning investment and people development as a continuing need? Will they measure it to better manage it? Will they begin to see human capital measures as a means for learning about what is working and what is not? Will they begin to see human capital reporting as a leading indicator of recovery? 

2020 CTR Annual Conference


Join us at the 2020 Virtual CTR Annual Conference for this session where industry experts share the definitions of the 23 ISO disclosure metrics and how to use these to in your human capital reporting strategy.


Three Measures You Should Capture Now During COVID-19

I hope everyone is healthy and managing through these very trying times. And, I hope the same for your family and friends. I know this pandemic has not been easy.

All of us by now are well into our Covid-induced new world where we have shifted from instructor-led training (ILT) to virtual instructor-led training (vILT) and eLearning, supplemented by content available through our employee portals. On top of all of these changes, you have probably had to create new courses on safety and return-to-work procedures. Perhaps you have provided guidance to your leaders about how to manage remotely in stressful times. Some have even gone into cost reduction mode to help offset the loss of income.

While you’ve been really busy, I hope are ready to ‘come up for air’ now. If so, here are a few measurements to think about. If you have not already implemented them, there is still time to collect the data before it gets lost.

Make Time to Collect and Analyze Data

First, be sure you are collecting data on your vILT and any new courses you have put online. You may be using a new platform like ZOOM, that were not set up to connect with your learning management system (LMS), and the number of participants may not be recorded and not be receiving typical follow-up survey. In this case, be sure to set up a system to capture the number of participants manually—perhaps by asking instructors to send in counts. And, consider sending a survey out to at least a sample of participants—even if you have to do it manually. If you cannot easily generate the email list of participants, send to a sample of all employees and make it more general. For example, ask whether you have taken a vILT course, online course, or downloaded content in the last three months. Then, get feedback on each.

Feedback, especially for vILT, is important so you can compare to ILT and understand what participants like better about it as well as what they dislike. Surprisingly, some organizations are reporting higher or equal scores for vILT. And these results are from vILT which was probably rushed into production (i.e., presentations from ILT that were used and repurposes for vILT). Imagine how much better vILT could be if it were actually designed for virtual delivery. This is the data you and your leaders will need to decide if you want to permanently change your mix to less ILT in favor of more vILT when the pandemic is over.

Capture Your Efficiency Savings

Second, be sure to capture the efficiency savings as a result of switching to vILT and online courses. Typically, vILT and eLearning will be less expensive because there is no instructor or participant travel and the course may be shortened. To be fair you should really only count the savings where an alternative to ILT was provided. Of course, there will be savings from simply cancelling ILT with no replacement, but that is not really a savings. It is just a reduction in offerings. You can estimate the cost of ILT and the cost of vILT so you can calculate the difference. Don’t be afraid of estimating—it’s a common business practice.  If you have too many offerings to make a calculation for each course, come up with the  average ILT and vILT cost, find the difference, and multiply it by the number of participants.

Capturing the dollar savings from switching to vILT is important because there is real value there and it should factor into your decision about continuing with a mix of learning modalities after the pandemic ends. The savings will be especially large for organizations where participants or instructors incur travel and lodging costs.

Calculate Your Opportunity Costs

Third, calculate the reduction in opportunity costs by switching to vILT and e-learning. Opportunity cost is the value of participants’ time while in class and traveling to and from class. This adds up—especially for half-day or full-day courses and can easily exceed the accounting costs (i.e., room rental, supplies, instructor, etc.) for some courses.

Calculating opportunity cost is simple. Take the average hourly employee labor and related cost (including benefits, employer paid taxes, etc.) from HR and multiply it by the reduction in hours from switching to vILT and e-learning. For example, suppose you replaced an 8-hour ILT with a 5-hour vILT, which also saved participants one hour of travel time.  The reduction in hours would be 3+1=4 hours. Multiply the 4 hours by the number of participants and by the average hourly compensation rate. You can use the average hours for an ILT course and compare it to the average for vILT or e-learning to make the calculation at an aggregate level.

It’s important to account for savings in travel time—because employee’s time is valuable. By eliminating travel, you are giving time back to employees when you move to vILT or e-learning. Share the opportunity cost savings with your senior leadership, CFO, and CEO along with the savings in course costs. The opportunity cost savings, in particular, is likely to be an impressive number. Think about this savings when you decide whether to go back to ILT or keep using vILT.

I hope you find these measures helpful. You really need them to make an informed decision about what to do when the pandemic ends. I think the profession has a great opportunity to use this experience to become much more efficient and effective in the long run.

Forward to the Past—Reflections on How Our Profession Evolves (Or Not)

If you follow this blog, you know that I’ve been struggling this past year with the profession’s new focus on upskilling employees for future needs. I have been trying to understand two things:

  1. How future needs can be reliably identified now, and
  2. Assuming needs can be identified, why would anyone want to provide training now, when the skills and knowledge will not be needed or applied for years to come—resulting in scrap rates of 100%.

To me, this seems to go against everything we have learned over the last 20 years in terms of good performance consulting and designing with application and impact in mind.

I was excited to see multiple topics on upskilling and reskilling at a recent CLO Symposium. I looked forward to learning more about this topic, given its incredible popularity.

During the sessions, I had expected to learn that the identifiable future skills would be tangible skills—like programming or new manufacturing techniques (which would be difficult to predict with enough specificity to train for today). Instead, the future skills identified by the presenters were all soft skills like communication, teamwork, innovation, creative thinking, and problem solving. This reminded me of current research that cited soft skills are very important to future success.

That same research also indicated that employers are looking for those same soft skills in their employees today. If these skills are needed today, we can use our performance consulting tools to identify the gaps and design learning to address them. This all means that these soft skills can be applied today, which means we are really not talking about future skills—we are talking about skills to meet current needs, which are considered important for the future.

The CLO Symposium presenters shared these skills in the context of  being ‘new’—as in organizations should start to train employees on them. This is where I felt I was fast forwarding to the past. When Caterpillar U was founded in 2000, we had a brand new LMS containing hundreds of courses. In the ‘old days’ training organizations used to brag about how many courses they had in their ‘paper’ catalog. Guess what was in that catalog…soft skill courses like communications, team building, writing, and problem solving. The catalog also included a lot of job function skills too, but the point is, soft skill courses are not new. They have always been in demand and always will be. So, can we please apply a little historical context and be more careful about how we define  ‘future skills’? What we are really talking about is ‘forever skills.’

As I reflect on the past, it’s interesting to note what happened to learning leaders bragging about how many courses they offered. Company executives began asking for results. They wanted to know how these hundreds of courses specifically aligned to business goals and needs. Executives demanded measures beyond how many courses were offered. This led to ‘strategic alignment’ where the emphasis is how courses proactively and specifically align to meet the goals of the enterprise.

For many years, strategic alignment had been a hot topic at industry conferences and many books have been written on the topic. How many times did I hear strategic alignment mentioned at the CLO Symposium conference? Zero. I understand that focus areas change and evolve, but we don’t seem to be building on our past. Instead, we’re re-inventing it as with identifying soft skills as future skills. We will advance more quickly and soundly as a profession if we have a sense of our past and can skip the re-learning phases.

Let’s remember that soft skills will always be important as will strategic alignment. There is no need to cycle back and forth and rediscover the value of each. A holistic future can include all that we have learned which will provide the strongest foundation upon which to build, and then we can focus more of our energy on inventing what is truly new and special.

Thoughts on the Coronavirus and What It Means for L&D

The situation surrounding COVID-19 is evolving on a daily basis. Uncertainty is high and it may be weeks or even several months before we know how all this plays out. Several things are clear, though.

First, the US and world are likely headed into an economic “timeout” and quite possibly a recession. Hopefully, this will be brief but even a brief downturn will cause significant harm. Many in the service industries are going to lose their job or have their hours severely reduced. Other companies will experience falling sales as people cut back or delay their spending. Most organizations have already restricted travel. Even if the worst of the virus is over by summer and travel restrictions are lifted, companies may implement cost reduction measures for the remainder of the year to partially offset a decline in revenue for the first half of the year.

So, if your organization is one which is negatively impacted, now is the time to implement the recession planning that we have talked about for years. Be prepared to start reducing variable expenses, including the use of vendors and consultants. Where appropriate, shift from instructor-led learning to virtual ILT, e-learning, and performance support. Prioritize your work and be prepared to focus on only the most important. Get advice from your governing body or senior leaders on where they would like you to focus your efforts. If you have not already done so, clearly align your programs to the top goals and needs of your CEO.

Second, COVID-19 presents an excellent opportunity to make significant progress in moving away from instructor-led learning and towards virtual ILT and e-learning. Most organizations have been moving in this direction for years but say they are not yet at their desired mix of ILT to vILT and e-learning. Well, now is the perfect opportunity to make this transition. Travel restrictions will prevent employees from traveling to a class and will also prevent instructors from flying to class locations. And social distancing discourages bringing employees together for training which is why so many universities have announced that they are shifting entirely to remote learning for at least the next month or so. The private sector should be equally responsive. Use the existing platforms to conduct classes virtually and ramp up the marketing of your e-learning and portal content.

Third, the virus also presents a once-in-a-lifetime opportunity to highlight performance support. What can we in L&D do to help our colleagues adapt to this new world and still do their jobs? People will be working remotely from home, perhaps for the first time. What performance support do they need? Most will not need a 30-minute e-learning course on working remotely but they will need steps to take to get connected. They will need help setting up virtual meetings. What can you provide them to make their life easier and to prevent a flood of calls to IT or HR for support? And, building on this, moving forward what opportunities do you have to replace some of your existing ILT or e-learning with performance support? Take this opportunity to truly integrate performance support into all that you do.

On a personal note, I wish each of you the best and hope you stay healthy. The situation will stabilize and eventually return to normal, but in the meantime let’s see what we can do to help each other through these challenging times.

Top Five New Year’s Resolutions for the Learning Profession

January is always a good time to reflect on the past and ponder the future. It’s also a good time to make some resolutions to improve in the coming year and decade. Here are my top five suggestions for the profession in the areas of measurement, analytics, reporting, and running learning like a business.

1. Resolve to measure more at levels 3 (application) and 4 (impact).

In November, ATD released its latest survey on evaluation that showed 60% of respondents evaluated at least one program at level 3 which is up from 54% in the 2015 study. Likewise, 38% measured at least one program at level 4 which is a slight increase from 35% in 2015. The good news is that the numbers are moving in the right direction. The bad news, is that they should be much higher. Over 80% measure levels 1 and 2. This should be the target for level 3 as well. And, we should measure level 3 for more than just a few programs. It should be measured and managed for all important programs. Once we start measuring application at a higher rate, we can turn our attention to measuring more programs at level 4. Phillips has shown us that this measure is what CEOs most want to see from us and we’re not giving it to them. We need to fix this.

2. Continue our shift from order takers to strategic business partners.

This will require us to have a discussion with the CEO before the year starts, to discover the goals fro the coming year and the goal owners. We need to talk with the goal owner to determine if learning has a role to play, and if it does, work with their staff to recommend an appropriate program. The goal owner then needs to approve the program and we need to reach agreement on the planned impact, other key measures of success, timing, and roles and responsibilities. One of the most important responsibilities of the goal owner will be to ensure that supervisors reinforce the learning with their employees in order to achieve a high application rate. All of this should be completed before the new year starts.

3. Be more disciplined and honest about how well aligned courses are with your organization’s goals.

Most learning departments would say they are well aligned but really, they’re not. What do I mean by “alignment”? It means the courses were designed (or purchased) in response to an identified need in support of a goal. If you say you are “aligned” to your organization’s top five goals, you should have a strategic alignment table which lists the top five goals—and under each one, you show the programs designed (or purchased) specifically to help achieve that goal. Instead, most use backward mapping, which starts with the course catalog and then seeks to map a course to all relevant goals. Since most courses indirectly support most goals, many conclude they are well aligned. For example, you might find that courses to improve communications skills are “aligned” to each of the top five goals (like increasing sales). The two approaches are very different—which explains why in surveys such a high percentage report they are aligned.

4. Be more sophisticated in your analysis of data.

Historically, data has been aggregated and reported as a total or average. Say for example, we report that the average participant reaction (level 1) is 76% favorable or that the average application rate is 51%. These are excellent summary measures but often do not tell the entire story. Perhaps the overall 76% favorable rating is due to the majority of participants being very satisfied but a sizable minority being very dissatisfied. We would want to know this information so we can explore the reasons (i.e., wrong target audience, bad instructor, etc.). Look at the distribution, not just the average or total and then use the information at a “by-name” level to take action (i.e., letting the supervisor know they are going to have to work harder with an individual to get application). This is the potential of microdata, which is really just a fancy term for data at the individual employee level. Ken Phillips is doing interesting work in this area.

5. Create and use more management reports.

Today, the industry primarily generates scorecards and dashboards. Scorecards show historical data in tabular form (like number of participants or courses for the last six months) while dashboards include some visual and/or interactive displays. Both however are designed to inform by sharing historical results. In other words they are “backwards” looking. Of course, it is important to know where we have been so we need scorecards and dashboards—however we also need to look forward. For this, we have management reports that show the plan (or targets) for the year, comparison of year-to-date results with plan, forecast (how the year is likely to end), and forecast compared to plan. They are designed to help leaders manage their initiatives to deliver promised results by answering two questions: “Are we on plan year to date?” and “Are we going to end the year on plan?” Management reports complement scorecards and dashboards by providing that forward look which can be used to actively manage initiatives.


Upskilling Revisited: What Strategy Makes the Most Sense?

In last month’s blog I shared my skepticism about some of the current upskilling initiatives. I have been thinking more about it since then. In fact, it is hard not to given the near daily references to the need for upskilling. Just this morning I saw a reference to a World Economic Forum study on the future of jobs which indicated that 54% of employees will require significant reskilling or upskilling by 2022.

Several things come to mind. First, I wonder about the terminology. How should we define these two terms? The term “upskilling” seems to suggest providing a higher level of skill while “reskilling” suggests a different skill, which may be higher, lower, or the same in terms of competency. Is that what we mean? And how does this differ from what we have been doing in the past? Haven’t you been reskilling and upskilling your employees for years? Isn’t that what the term training means? (Unless you have been “downskilling” your employees!) Is “reskilling” just the new term for “training”? Don’t get me wrong. If you can get a larger budget by retiring the term “training” and emphasizing “reskilling or upskilling”, then go for it. But let’s be careful about how we use the terms.

Second, I wonder about the methodology being used to conclude that upskilling is needed. As I mentioned last month, one of my worries is that we are not doing a needs analysis to confirm this need or identify precisely what the need is. I know some organizations are hiring management consultants to tell them what skills will be needed in the future but how do the consultants know what your needs will be? I know other L&D departments are asking their own organization leaders the same question, but how good is the knowledge of your own leaders? My concern is that this methodology will generate the type of common, somewhat vague, needs we now hear about all the time like digital literacy or fluency.

Third, what exactly do we do once we have identified these vague needs? A traditional needs analysis would identify the performance that needs to be improved and then, assuming learning does have a role to play in improving that performance, recommend training specifically designed to close the gap. If we were to provide training to improve digital fluency, exactly what will be improved? Are we back to increasing competency simply to increase competency? Training for training’s sake? So, my concern here is that even if we increase competency in these areas of “need”, nothing of measurable value will improve.

Fourth, what is the time frame? Put another way, how exactly is the question being asked of leaders about future skills? Is it about “future skills” with no time period specified, or is it about skills that will be needed in one year, five years, or ten years? When does the “future” start? It seems to me that the further out we go, the more likely we are to mis-identify the actual needs, especially given the rapid pace of change. Even if we could identify a need one or two years in the future, do you really want to develop a course today to address it, given that the need may evolve? Moreover, if you begin upskilling employees today, they are not going to have an opportunity to apply the new skill for one or two years, by which time they will have forgotten what they learned.

Here is my suggestion for discussion. Doesn’t it make more sense to focus on the skills needed today and in the very near-future and to deploy the learning at the time of need rather than months or years ahead of time? Ask leaders what skills their employees need today or will need in the very near-future, and then follow up with a proper needs analysis to confirm or reject their suggestions. If we can do a better job meeting the current and very near-future needs of the workforce, won’t our employees always be well positioned for the future? And isn’t a reskilling/upskilling initiative tailored to specific employees and delivered at the time of need likely to be far more impactful than an approach based on vague needs for a large population delivered months or years before the new skills will actually be used?

Reskilling—Are We Thinking About This Clearly?

It seems that “reskilling” is on everyone’s mind these days. How often do we hear about an organization’s need to reskill its workforce for the future? And, this isn’t just coming from L&D—it is also coming from senior leaders. So, reskilling is the shiny new object that is attracting a lot of attention; and as my regular readers know, the brighter that new object, the more skeptical I become that we are on the right path.

Before sharing my doubts about reskilling, let me first acknowledge that there are situations where it makes perfect sense. For example, suppose that a manufacturing plant is going to be modernized in six months. The current employees do not have the skills to run the new automated plant but some would be interested in learning. I think this would be a perfect case for reskilling so the company can keep its existing employees who are loyal and committed to the success of the organization. By all means, provide the training for the new skills they will need to succeed. And, if there are not enough positions in the newly automated factory, provide new skills for the remaining employees to fill other positions in the company. So, I don’t have any issue with this use of reskilling where specific, existing employees need new skills to do a different job because their current job is about to be eliminated.

Let’s explore the other end of the reskilling spectrum, where I do have serious doubts about our current path. Some organizations are looking ahead and trying to define what skills will be needed in the future, often five to 10 years in the future.

In my experience, it is sometimes difficult to get good direction and consensus on skills needed in the short term (like for the next year), yet alone for the future. I have a hard time imagining how the process will work to define skills needed in five to 10 years. Who will L&D rely on to define these needs? Department heads, the CFO, CEO, governing body, a consulting company? How good is their forecast likely to be? Do these people have a track record for forecasting skills or will this be their first time? I can say from my experience at Caterpillar that making accurate long-term economic sand sales forecasts was virtually impossible. So, in a fast-changing world, I am skeptical of our ability to accurately forecast skills five to 10 years out.

Even if we could accurately forecast skills, does it make any sense to start training for those skills today? The assumption here is that the skills are not needed today but will be needed in the future. Given what we know about how quickly knowledge dissipates and about the importance of immediately applying what was learned, why would we start training now for a future need? Wouldn’t it be better to wait until just before the need actually exists and then offer the training, combined with the opportunity to immediately apply it and receive reinforcement from management?

Now, at this point, some may say that the time frame is not five to 10 years but instead just a few years—maybe even just one year ahead. Furthermore, some may say that leaders can readily identify the skills their current employees have and that these skills will be even more in demand in the future. Examples of this may be processes like design thinking, adaptability, communication and teaming. In this case, we can use our standard needs analysis and performance consulting process to identify the skills gaps and address them. Now however, I would contend that we are no longer talking about reskilling, but instead traditional training.

I believe reskilling connotes new skills that are not needed in one’s job today, but will be needed for a different job either today or in the future. And, I think it makes perfect sense to provide these new skills to existing employees whenever there is a good fit and when the need is immediate. I don’t believe we should spend much time trying to identify skills that will not be needed for five to 10 years down the line; and I certainly don’t believe we should start training for those skills today. Instead, let’s focus our reskilling efforts on those employees who need (or want) to find a new job in the next 12 months and who require new skills. For the vast majority of employees who simply want to keep learning to improve in their present position, let’s just continue to call this “training”.

Human Capital Disclosure May Soon Be Mandated by the SEC

In an earlier blog I talked about the ISO standards for human capital reporting which were published in December. These called for the voluntary public disclosure of measures for both large organizations and small/medium-sized organizations. While some European and Asian governments are likely to adopt the ISO recommendations as law, the United States Congress was never likely to follow suit so it appeared that adoption in the U.S. would be voluntary and slow.

The U.S. Securities and Exchange Commission just published proposed rule making which, if implemented, will bring human capital disclosure to U.S publicly traded companies much sooner than anyone imagined. This is a game changer for our profession and a VERY BIG DEAL!

On August 7 the SEC proposed to fundamentally change the way publicly traded companies report. Under the current rules, the SEC specifies 12 items that must be included in the narrative description which accompanies the financial statements. Companies have to address all that apply to them. For example, these items include a discussion of the principle products and services offered, new products and segments, sources and availability of raw materials, whether the business is seasonal, competitive conditions, and the current backlog of firm orders. The last required item is the number of employees, which is the only human capital measure currently required for disclosure.

The SEC proposes to move away from an explicit list of required items and instead adopt a principles-based approach where each company must discuss whatever is material with the understanding that the list of topics will differ by industry and company. The SEC does, however, share a non-exclusive list of items that it believes will apply to most companies. This list includes five items from the current list of 12 and two new items: HUMAN CAPITAL and compliance with government regulations. (The other five items are revenue-generating activities, development efforts for new products, resources material to the business, any business subject to re-negotiation or cancellation, and seasonality.) Furthermore, simply disclosing the number of employees is no longer sufficient. In their own words:

“Item 101(c) (1) (xii) [the current rules] dates back to a time when companies relied significantly on plant, property, and equipment to drive value. At that time, a prescriptive requirement to disclose the number of employees may have been an effective means to elicit information material to an investment decision. Today, intangible assets represent an essential resource for many companies. Because human capital may represent an important resource and driver of performance for certain companies, and as part of our efforts to modernize disclosure, we propose to amend Item 101 (c) to refocus registrant’s human capital resources disclosures. Specifically, we propose replacing the current requirement to disclose the number of employees with a requirement to disclose a description of the registrant’s human capital resources, including in such description any human capital measures or objectives that management focuses on in managing the business.” (page 48 of the proposed rule Modernization of Regulation S-K Items 101, 103, and 105)

Did you just feel the earth shifting below your feet? You should have. The importance of this proposed rule for our profession simply cannot be overstated. Many in the profession have worked years to increase the visibility and use of human capital measures. The time may finally have come for the U.S.

While it is true that the SEC will not prescribe specific human capital measures, it does provide some examples, including measures for attraction, development and retention of personnel. The test for disclosure, however, is clear: materiality. Can you imagine any CEO or CFO telling analysts, the public and their own employees that people are not a material contributor to the company’s success, especially after saying for years that people are the company’s greatest asset? I don’t think so. So, if this rule making is finalized, human capital disclosure is coming and coming soon.

Most companies will rely on their heads of HR as well as accounting for guidance on what to include in their narrative on human capital. If for no other reason than risk mitigation, these leaders in turn will look to the human capital profession for guidance. And they will find ISO30414:2018 which are the human capital reporting standards published in December 2018. These standards recommend the external reporting of 23 measures for large companies and 10 for small medium. These measures will be a natural starting point as companies decide what to discuss so if you don’t yet have a copy, get one, and be prepared to proactively help your organization be a leader in human capital reporting. The ISO document is available for purchase at https://www.iso.org/standard/69338.html.

The proposed SEC rule is available at https://www.sec.gov/rules/proposed/2019/33-10668.pdf. The rule is 116 pages, but the section on human capital is under section IIB7 pages 44 -54.

Can you Justify your Learning and Development Projects?

By Jack J. Phillips Ph.D., Chairman, ROI Institute

Daily headlines in the business press focus on the economy. While it is booming in some areas, other areas are slowing, and economic uncertainty exists everywhere. During uncertainty, executives must take steps to make sure the organization can weather the storm—whenever or wherever it occurs.

One way executives meet this
challenge is to ensure that expenditures represent investments and not just
costs. If an expenditure is considered a cost, it will be frozen, reduced, or
in some cases eliminated. If it is considered an investment that is producing a
return, it will be protected and possibly even enhanced during difficult times.
For example, many learning and development budgets are now being frozen or
reduced, even with record profits. While this seems illogical, it happens. Now
is the time to reflect on your budget and your programs. Can you withstand top
executive scrutiny? Are you ready for ROI?

The most used evaluation system
in the world is the ROI Methodology® to measure impact and ROI for a few major
programs. The ROI Certification® is the process to develop this valuable
capability. This valuable certification provides participants with the skills
needed to analyze the return-on-investment in practical financial terms. The
results are CEO and CFO friendly. Over 15,000 managers and professionals have
participated in this certification since it was launched in 1995, underscoring
the user-friendly nature of this system.

Don’t have the time or budget?
Several approaches are available to reduce the amount of time and cost needed
to develop this capability. For more information on ROI Certification, contact
Brady Nelson at brady@roiinstitute.net.

ROI Institute is the global
leader in measurement and evaluation including the use of return on investment
(ROI) in non-traditional applications. This methodology has been used
successfully by over 5,000 organizations and in over 70 countries.

Bridge the Gap from Training to Application with Predictive Learning Analytics

by Ken Phillips, CEO, Phillips Associates

In my previous blog post, I discussed the concept of scrap learning and how it is arguably the number one issue confronting the L&D profession today. I also provided a formula you could use to estimate the cost of scrap learning associated with your training programs.

In this post, I’ll share with you a revolutionary methodology I’ve been working on for the past several years called Predictive Learning Analyticts™ (PLA). The method enables you to pinpoint the underlying causes of scrap learning associated with a training program. It consists of three phases and nine steps to provide you with the data you need to take targeted corrective actions to maximize training transfer (see figure below). 

While the specific questions and formulae for the scores are proprietary, I hope you can apply the concepts in your organization using your own survey questions and your own weighting for the indexes. Even if you adopt a simpler process, the concepts will guide you and the article will give you an idea of what is possible.

Phase 1: Data Collection and Analysis

Unlike other training transfer approaches which focus mostly on the design and delivery of training, PLA offers a holistic approach to increasing training transfer. Built on a foundation of three research-based training transfer components and 12 research-based training transfer factors (see chart below), PLA targets the critical connection among all these elements. In short, PLA provides L&D professionals with a systematic, credible and repeatable process for optimizing the value of corporate learning and development investments by measuring, monitoring, and managing the amount of scrap learning associated with those investments.

Training Transfer Components & Training Transfer Factors

Phase 1: Data Collection & Analysis

The objective of phase one, Data Collection & Analysis, is to pinpoint the underlying causes of scrap learning associated with a training program using predictive analytics and data. Five metrics are produced and provide L&D professionals with both direction and insight as to where corrective actions should be targeted to maximize training transfer. The five measures are:

  • Learner Application Index™ (LAI) scores
  • Manager Training Support Index™ (MTSI) scores
  • Training Transfer Component Index™ (TTCI) scores
  • A scrap learning percentage score
  • Obstacles preventing training transfer

Data for calculating the first three measures: LAI, MTSI, and TTCI scores, is collected from program participants immediately following a learning program using a survey. The survey consists of 12 questions based on the 12 training transfer factors mentioned earlier. Data for calculating the final two measures are collected from participants 30 days post program using either a survey or focus groups and consists of the following three questions:

  1. What percent of the program material are you applying back on the job?
  2. How confident are you that your estimate is accurate?
  3. What obstacles prevented you from utilizing all that you learned if you’re not applying 100%?

Waiting 30 days post program is critical because it allows for the “forgetting curve” effect—the decline of memory retention over time—to take place and provides more accurate data.

LAI Scores

LAI scores predict which participants attending a training program are most likely to apply, at risk of not applying and least likely to apply what they learned in the program back on the job. Participants who fall into the at-risk and least likely to apply categories are prime candidates for follow-up and reinforcement activities. Examples include email reminders, micro-learning or review modules, and coaching or mentoring to try and move them into the most likely to apply category.

MTSI Scores

MTSI scores predict which managers of the program participants are likely to do a good or poor job of supporting the training they directed their employees to attend. Managers identified as likely to do a poor job of supporting the training are prime candidates for help and support in improving their approach. This help might take the form of one-on-one coaching; a job aid explaining what a manager should do before, during, and after sending an employee to training; or creating a training program which teaches managers how to conduct pre- and post-training discussions with employees.

TTCI Scores

TTCI scores identify which of the three training transfer components and the 12 training transfer factors affiliated with them are contributing the most and least to training transfer. Any components or factors identified as impeding or not contributing to training transfer are prime candidates for corrective action.

Scrap Learning Percentage

The scrap learning percentage score identifies the amount of scrap learning associated with a training program. It provides a baseline score against when follow-up scrap learning scores can be compared to determine the effect targeted corrective actions had on increasing training transfer.

The obstacles data identifies barriers participants encountered in the 30 days since attending the training program that prevented them from applying what they learned back on the job. Waiting 30 days to collect the data allows for the full range of training transfer obstacles to emerge. For example, some are likely to occur almost immediately—I forgot the things I learned—while others are likely to occur laters—I never had an opportunity to apply what I learned. Frequently mentioned obstacles are prime candidates for corrective actions to mitigate or eliminate them.

Phase 2: Solution Implementation

The objective of phase two: Solution Implementation, is to identify, implement, and monitor the effectiveness of corrective actions taken to mitigate or eliminate the underlying causes of scrap learning identified during phase one. Here is where the “rubber meets the road,” and you have an opportunity to demonstrate your creative problem-solving skills and ability to manage a critical business issue to a successful conclusion. Following the implementation of the corrective actions, it is now time to recalculate the amount of scrap learning associated with the training program. You can then compare the results to the baseline scrap learning percentage calculated during phase one.

Phase 3: Report Your Results

The objective of the third phase: Report Your Results, is to share your results with senior executives. Using the data you collected during phases one and two, it is time to show that you know how to manage the scrap learning problem to a successful conclusion.

In Sum

Scrap learning has been around forever, however what is different today is that there are now ways to measure, monitor, and manage it. One of those ways is through Predictive Learning Analytics™. Alternatively, you might employ the concepts to build your own simpler model. Either way, we an an opportunity to reduce scrap learning.

If you would like more information about the Predictive Learning Analytics™ methodology, email me at: ken@phillipsassociates.com. I have an ebook that covers the method and a case study illustrating how a client used the process to improve the training transfer of a leadership development program.

Business Alignment: A Critical Success Factor for L&D Organizations

by Peggy Parskey, Associate Executive Director, Center for Talent Reporting

Business Alignment
Signpost with Alignment wording

If you have attended a Learning and Development (L&D) industry conference within the past three years or listened to a panel of senior L&D leaders, it’s likely that someone has raised the topic of alignment. Numerous blogs from ATD, SHRM or even technology vendors confirm that alignment is clearly top of mind.

Given the attention paid to the topic of alignment, you might think that L&D has nailed down the principles, process, and outputs of business alignment. Unfortunately, you would be wrong. We haven’t nailed down this process by a long shot. And therein lies the problem: The L&D industry does not have a consistent definition of what alignment is, let alone how to achieve it.

Do a quick Google search on L&D business alignment and you will get thousands of articles on the topic. Click some of the links and you will find as many suggested approaches to alignment as search results. Some authors suggest that business alignment is about getting the right KPIs to support business goals. Other suggest that alignment is about doing a gap analysis between the current and future state to identify the needed training programs. Some bloggers recommend conducting interviews with business leaders as input to an L&D strategy. And a few organizations with whom we have worked view alignment as a simple mapping exercise: “We need to grow sales this year, we have a bunch of sales programs. Voila, Alignment!”

With all of these possible approaches, how should an L&D leader navigate the alignment process?

Principles of Effective Alignment

At least four principles should guide leaders who are grappling with achieving business alignment:

  1. Alignment isn’t a nice to have, it’s a must have
  2. Alignment requires engagement and commitment from both L&D and business leaders
  3. Alignment has both strategic and tactical components
  4. Alignment isn’t simply about what L&D offers but also how it offers them

1. Alignment Isn’t a Nice to Have

Strategic and tactical alignment is critical to ensure that L&D is investing its scarce resources in the right place at the right time on the right programs. When L&D doesn’t execute this process or doesn’t manage it effectively, the consequences can be significant. If L&D delivers the wrong programs to the wrong audiences, the organization will lack the needed capability to achieve its goals. Furthermore, other organizations now have to pick up the slack created by L&D. Non-L&D functions may develop shadow learning organizations, or hire external resources to fill the gap left by L&D. If L&D wants to fulfill its purpose to develop the needed capability for the current and future workfoce and improve L&D performance, then effective alignment is the critical success factor.

2. Alignment Requires Mutual Engagement

Discussions about L&D alignment often imply that the business has a peripheral role to play. L&D leaders get the strategic goals from above or they interview a few business leaders to understand priorities. At that point, the business seems to disappear from view.

If L&D leaders want to achieve effective alignment, the business can’t be a bystander. Senior business leaders must engage with L&D not only to communicate their priorities but also to ensure L&D has a role to play in achieving those priorities. Moreover, the business is an influential voice to ensure that L&D has the appropriate resources and support to deliver on its promise. Furthermore, the business will have the primary responsibility for reinforcement, without which there is likely to be little application to the job. If the business doesn’t understand its role, then it is incumbent on L&D leadership to spell it out and create shared accountability for success.

3. Alignment Has Both Strategic and Tactical Components

The alignment process must start at the strategic level. Business leaders establish strategic priorities, set goals and then allocate resources to achieve them. Based on these priorities. L&D then determines if and where it plays a role. If the organization plans to launch a new product line, L&D and the business must agree that L&D should own the process to train employees on these new products. If the organization wants to capture new demographic for its products, the business and L&D may agree that L&D has a minimal role to play. Regardless of the objective, business and L&D leaders need to clarify if L&D has a role as well as the importance and urgency of that role.

Having reached agreement on the strategic priorities, L&D must also ensure it aligns tactically. As an example, imagine that $500K is allocated to develop customer experience competencies. L&D practitioners then dive into the details to assess specific organizational needs. During this process, performance consultants discover that a primary root cause of underdeveloped customer experience capability is the lack of quality resources for call center personnel. At this point, L&D discovers that its requirements have changed. Training, while necessary, is not at the heart of the under performance. At this point, L&D may simply turn over the requirement to the business to develop the needed content for call center employees and invest its scarce resources elsewhere.

The point of this example is that what appears to be development need at the strategic level, may not translate into a development requirement when practitioners study the requirements more fully. It is incumbent on L&D practitioners to adjust their approach to ensure they stay aligned.

4. Alignment Isn’t Just About What L&D Offers

Alignment isn’t just about what programs L&D offers, but increasingly, how it offers them. In a fast-paced world, instructor-led training (ILT) has a long development cycle, is expensive to produce, and requires a large time investment for learners. Yet, according to ATD’s 2018 State of the Industry Report, ILT methods still comprise 67% of all training hours with self-paced at 29% and mobile learning a mere 2%.

Learners need content at their fingertips. They need a rich reservoir of material in different forms (white papers, how-to guides, videos, checklists, case studies) that are easy to find and easy to consume. Content must be high quality and useful on the job.

Alignment requires not simply that L&D builds capability and improves performance, but also does it in the most efficient and effective manner appropriate to the need.


Alignment is critical to ensure the L&D function meets the needs of the business by building the necessary skills to run the business and achieve its strategic objectives. This post focused on the key principles at the heart of alignment. In subsequent posts we will explore the importance of treating alignment as a continuous process and building skills to manage the end-to-end process effectively.

I recommend you learn more about alignment from respected industry leaders who can provide guidance on how to achieve meaningful alignment with the business. Below are four resources you should check out:

The Business of Learning by Dave Vance. See Chapter 4 which discusses strategic alignment in depth.

Attend the CTR Measurement and Reporting Workshop, which addresses the importance of alignment and how to engage business partners in the process.

Read the recently published IDC PlanScape document on the importance of L&D alignment to maximize the impact of training investment.

Read the upcoming 2nd Edition of “Learning Analytics,” which will be published in February 2020. (The current edition may be found here. Look for the second edition at www.explorance.com in February 2020. The second edition has a chapter devoted to the topic of strategic and tactical alignment.)

We’d like to hear from you. If you’re having business alignment challenges, don’t hesitate to email Dave Vance or Peggy Parskey.

The Greatest Issue Facing L&D Today

Scrap Learning

by Ken Phillips

What is arguably the top issue facing the L&D profession today?

The answer is scrap learning or the gap or difference between training that is delivered but not applied back on the job. It’s the flip side of training transfer and is a critical issue for both the L&D profession and the organizations L&D supports because it wastes money and time—two precious organization resources.

Now, you might be wondering, “How big is the problem?”

Two empirical studies, one by KnowledgeAdvisors in 2014 and one by Rob Brinkerhoff and Timothy Mooney in 2008, found scrap learning to be 45 percent and 85 percent respectively in the average organization. To add further credibility to these percentages, I’ve conducted three scrap learning studies over the past few years and found the scrap learning percentages associated with three different training programs in three separate organizations to be 64 percent, 48 percent, and 54 percent respectively. Added together, these five research studies results in an average scrap learning figure of approximately 60 percent in the average organization.

To further highlight the magnitude of the scrap learning problem, consider its effect on the amount of wasted organizational dollars and time. According to the 2018 ATD State of the industry research report, the average per employee organization training expenditure in 2017 was $1,296 and the average number of training hours consumed per employee was 34.1. Using the KnowledgeAdvisors and Brinkerhoff scrap learning percentages mentioned above, you can see in the table below just how much scrap learning costs the average organization in wasted dollars and time.

Cost of Scrap Learning in Wasted Dollars & Time

Average per employee training expenditure (=) $1,296(X) 45% scrap learning(=) 583 wasted $
The average per employee training expenditure (=) $1,296(X) 85% scrap learning(=) 1,102 wasted $
The average number of training hours consumed per employee (=) 34.1(X) 45% scrap learning(=) 15 wasted hours
Average per employee training expenditure (=) $1,296(X) 85% scrap learning(=) 29 wasted hours

Taking all of this data into account reminds me of James Lovell’s famous quote during his Apollo 13 mission to the moon when an oxygen tank aboard the space capsule exploded putting both the flight and crew in great peril: “Houston, we have a problem!”

If you would like to take a crack at estimating the cost of scrap learning associated with one of your training programs, you can use the Estimating the Cost of Scrap Learning Formula below. To gain the most useful insight, you should make every effort to collect the most accurate data possible for each of the input variables. Also, when selecting an estimated percentage of scrap learning associated with the program, variable 4 in the formula, you should get input from several people familiar with the program such as other L&D colleagues, participants who previously attended the program or perhaps even the managers of program participants and then compute an average of their estimates. Gaining the input of others will increase both the accuracy and credibility of the estimate and remove any concerns that the scrap learning percentage is merely your opinion.

Estimating the Cost of Scrap Learning Formula

Wasted Participant Dollars

The length of a learning program in hours _____
X the number of programs delivered over 12 months _____
X the average number of participants attending one program _____
= the cost of scrap learning in wasted time _____
X the average hourly participant salary + benefits _____
= the cost of wasted participant dollars _____ (A)

Wasted L&D Department Dollars

Administrative expenditures (e.g., materials, travel, facility, facilitator, delivery platform, food, etc.) for one program _____
X the number of programs delivered over a 12-month period _____
X the estimated percent of scrap learning (45-85%) associated with the program _____
= the cost of wasted L&D department dollars _____ (B)

Total Cost of Scrap Learning

Cost of wasted participant dollars (A) _____
+ cost of wasted L&D department dollars (B) _____
= total cost of scrap learning _____

With this information in hand, you are now ready to pinpoint the underlying causes of scrap learning and take targeted corrective actions to mitigate or eliminate these causes and maximize training transfer. How to do this will be part two of this blog article.