ROI: Still Evoking Confusion and Controversy

The Return on Investment (ROI) concept for learning has been around since Jack Phillips introduced it about forty years ago. You might think by now that at least the confusion over its use would have diminished even if the controversy had not. Not even close. I continue to see the concept abused and misused on at least a monthly basis.

Here is an example from a recent blog: “Anyone who is involved in the business world is familiar with the concept of ROI. It’s punchy, with its relatively simple calculation, and can make or break a purchasing decision. But when it comes to learning initiatives, gathering the necessary data to calculate ROI is difficult, to put it mildly.” The author goes on to say that learning initiatives implemented as an integral part of business strategy can be measured by the success of that strategy.

There are several issues with the blog. First, the author appears to be confusing ROI used in the business world with ROI used in the learning world. Typically, a financial ROI is calculated to make the business case for an investment, like a new product line or facility. The definition of this financial ROI is not the same as the learning ROI. The numerator of the financial ROI is the net present value (NPV) of the increase in profit due to the investment. The denominator is the cost of the asset required for the project, such as the cost of a new facility. This will be capitalized as an asset on the balance sheet and depreciated each year.

Contrast this with the learning ROI which has (usually) one year of contribution to profit in the numerator (so no need for NPV) and no asset whatever in the denominator. Instead, the denominator is the cost of the learning initiative which is an expense item from the income statement. So, two different definitions and the calculation for the financial ROI is actually more complicated than that for the learning ROI. Interestingly, it was exactly this difference in formulas which led my colleagues in accounting at Caterpillar to tell me that I could not keep referring to learning ROI as “ROI” since it was confusing our leaders. So, I renamed it Return on Learning or ROL in 2002. Take away here is to remember the two are not the same and let those in your organization know that learning ROI is calculated differently.

The next point by the author is that learning ROI is difficult to calculate. The ROI calculation itself is very easy: simply, the net benefit of the initiative divided by the total cost. Net benefit is the gross benefit minus total cost. Generally, total cost is easy to calculate, but the sticky part is the gross benefit which is the increase in profit before subtracting the cost of the initiative. The gross benefit, in turn, depends on the isolated impact of the learning initiative, like a 2% increase in sales due just to the learning. Likely, this is what the author had in mind when complaining about the difficulty of calculating ROI.

However, isolation need not be that difficult. The learning team can work with senior leaders to agree on a reasonable impact for planning purposes. And, once the program is completed, there are several methods available to estimate actual impact which do not require special training or hiring a consultant, Often, it will suffice to simply ask the participants and the supervisors what they believe the impact was and then reduce the estimate some to allow for the inherent error in any estimate. While not precise enough for an academic journal article, this is usually all we need in the business world to make appropriate decisions (like go/no-go after a pilot) or identify opportunities for improvement. It will also be close enough to determine if the investment in learning more than covered the total costs.

Last, the author suggests that if the learning is integrated into the business strategy, its success can be measured by the success of the business goal. I strongly agree that we should always start with the end in mind (the business goal) and design the learning so it directly supports the business goal. Further, we need to work in close partnership with the business goal owner to ensure that the learning is provided to the right target audience at the right time and then reinforced to ensure the planned impact is realized. While this does provide a compelling chain of evidence that learning probably had the intended impact, it does not tell us how much impact the learning had or whether the investment in learning was worthwhile. Instead of a measurement, then, we are simply left with a “mission accomplished” statement.

The question remains how much sales would have risen without any training. If sales would have increased by 10 percent without training, the training clearly was not worth doing. How about if sales would have increased 9% without training? Was it worth doing? We still need to isolate the impact of training and calculate net benefit or ROI to answer this ultimate question of value – not to “prove” the value of the training but to enable us to make better decisions about programs going forward and avoid investing when the return does not justify the cost.

So, the debate goes on. A friend asked me recently if I still believe in ROI. Yes, I do, but we need to use it wisely. It should not be used defensively to try to prove the value of an initiative or an entire department. In other words, it is not an exercise to deploy at the end of a program or fiscal year when you are under fire. Rather it should be used to help plan programs to ensure they deliver value and to identify opportunities for improvement going forward. It should be used to help decide which programs to continue and to identify ways to optimize programs. ROI will never take the place of careful planning, building relationships with goal owners or smart execution. You will always have to get the basics right first. Once you have done that, then ROI can be a powerful tool to make you a better manager.

Comments

  1. Well said Dave. It takes a long time, obviously, to have a new concept understood even when it offers great value. Old ideas and entrenched systems are difficult to unseat. Keep up the great work.

  2. Model Mania is the problem: Messengers of models stick to them when they serve little purpose.

    When a model generates more exceptions than guidance, and creates more confusion than clarity, it’s a bad model. Bad models should be discarded. The ROI Learning Model (and Millennial Generation Model) are hurting learning development credibility. The trouble is once the concepts are used to sell programs, it’s hard for the messengers to say ‘Oops.’

    The messengers of ROI have taken a position and forgotten the issue: Their position – ROI is a useful model. How can a model be useful when it evokes confusion and controversy?

    The issue at hand was increasing credibility and influence for thought leaders dedicated to learning development. I believe ROI was specifically chosen because it sounds financial and executive. Yet, the greatest arguments regarding it happen in executive offices.

    Executive leaders know that figures lie and liars figure. Executive leaders also know that risk is part of the game. ATD released a study a few years back that CEO’s main interest isn’t ROI anyway, it’s innovation. Executive leaders know innovation takes risk. They want plans that mitigate the risk as much as possible, not equations with countless variables. Such equations hurt credibility (theoretical mathematicians excluded), not build it!

    And this problem of Model Mania is getting worse. Time wasted on the ROI learning model would have been better spent contributing ideas and details to the Kirkpatrick Model (Reaction, Learning, Behavior, and Results). It’s a concept model that can be explored in a thousand ways – if the learning development team has the courage to explore.

    Innovation demands new models. The trick is to drop them when they prove unworthy.
    Learning development is served best by thought leaders with the courage to scrap bad models, write off sunk costs, and build on the models providing credibility and clarity.

  3. Daniel Oginga Okal says:

    Confining ROI to only learning initiatives truly evokes confusion and contoversy in some quaters.However,i have noticed that most CEO’s are keen on ROI of all non-Capital initiatives especially projects and other staff costs.

  4. Patti Phillips says:

    Daniel is correct, CEOs are keen on ROI of all non-capital initiatives. In order to compare ROI of one non-capital expenditure to another, there needs to be consistency in models and the standards that support them.

    Innovation does not demand new models; innovation demands new applications of sound models and advancement of techniques used within those models as technology and data availability allow.

    While CEOs are interested in innovation, they are interested in innovation that pays off – hence, ROI. They are not going to fund an ongoing science project — they want innovation because it can lead to movement in the measures that matter to the organization.

    Learning ROI is only difficult to measure when the business measures are not defined upfront. Sadly, this is often the case. Programs roll off the shelf with vague and nebulous business objectives or no business objectives at all.

    If you want to change something, lets change this conversation. ROI is here; its not going anywhere. Decision makers are not confused and I’m betting on the members of the learning community – they aren’t confused either.

    Lets start talking about how to define business measures; how to leverage people data and research methods to determine what needs to change to improve those measures; and how to identify the right solutions. Lets start talking about how people can communicate value in a compelling, logical, and credible way given the audience to whom they are communicating. Lets talk about how to seamlessly integrate it all so that learning drives value that exceeds its cost (ROI) with no confusion.

    https://www.td.org/books/the-business-case-for-learning

    http://www.wiley-vch.de/en/?option=com_eshop&view=product&isbn=978-1-119-24237-6

    https://roiinstitute.net/roi-certification/

  5. The 2017 LinkedIn Workplace Learning Report showed that the number two measure desired by senior leaders, after business impact as the number one desired measure, is return on investment, but only 4% see it. So that says to me, ROI matters. And if it matters to the business it should matter to L&D.

    What I like about The ROI Institute’s methodology is the way in which it measures results end-to-end, including but not limited to ROI. You have to tell the whole story about results. Value, as expressed by return on investment, is an undeniable part of the story.

    I say, don’t deny the ROI story for L&D and don’t run from it because good or bad, it’s a story that can and should be told. The V Model, a driver of ROI impact studies, keeps L&D aligned to the business. Monetizing results is part of accessing the learning solution’s performance. It all works together.

    I’m excited that the conversation is changing and as the LinkedIn Workplace Learning Report shows, return on investment for L&D is an expectation. We can continue the debate on ROI’s efficacy for L&D or we can buckle down and do the work of collecting facts, evidence and data for results, including return on investment. I say, let’s buckle down and do the work!

Speak Your Mind

*