Do We Need New Measurements and Reporting for the Digital Learning Revolution?

Digital learning is revolutionizing the corporate learning environment by making a vast amount of content available to learners. Some of this content can be accessed through internal learning management systems, but the fastest growth is likely to come from content available outside the organization’s firewalls and systems. This is content directly available to anyone with internet access. In a sense we have had digital learning since the first computer-based training (CBT) became available in the 1980s and even more so with the advent of e-learning (WBT) in the late 1990s, but the current revolution dwarfs these past initiatives in terms of breadth, reach, and sheer volume.

Most of our current measures and reports / dashboards were designed for traditional classroom learning and have been applied to e-learning as well, but some have always asked if we need special measures for e-learning. The answer is basically been no. Traditional efficiency measures about number of participants, classes, hours, etc., combined with traditional effectiveness measures for participant reaction, test scores, application, impact and ROI can easily be applied to e-learning. The old post event survey was delivered at the end of class and with e-learning it is generally sent to participants immediately following class. So, no revolution was required in measurement or the reporting of those measures for e-learning.

The question we face now at the start of this new digital revolution is the same as the on e at the inception of e-learning. Will new measures and reporting be required? My initial take is that the answer remains the same as well. The standard efficiency, effectiveness and outcome measures will still serve us well, and the three standard reports recommended by Talent Development Reporting principles (TDRp) will still help us to better manage our programs and departments as well as demonstrate our alignment to and impact on key company goals. However, the measurement strategy itself may have to change dramatically because an increasing amount of content will be accessed outside the company and outside any learning management system (LMS). Think of employees taking free courses at the Kahn Academy and universities like MIT, learning from You Tube or their Google search, or taking advantage of aggregators like Coursera, Udemy, Udacity and others. Since this learning takes place outside the LMS there is no way to automatically capture efficiency measures like number of participants, courses, or hours, and there is no way to automatically send a survey to gauge satisfaction, amount learned, application, or other effectiveness measures let alone any outcome measure.

Instead, much more thought is going to have to be given to what measures are really important for this new digital learning. Do we really need to know how many courses employees take outside the LMS? Do we really need to know how satisfied they were with each course or how much they learned? I don’t think so. So, let’s go back to basics and determine what we want to know, why we want to know it, and what we would do with the data if we had it. Learning and development is increasingly important to millennials so an employer certainly has an interest in finding out whether employees are satisfied with their learning but not necessarily with each micro instance of learning (like a Kahn Academy course). Why not send a quarterly or semi-annual learning survey to employees (or expand your current employee engagement survey) to ask about their satisfaction with learning in general? You could ask about how they have learned in the past ┬áthree months (company offerings, MOOCs, free universities, Coursera-type providers, Kahn, etc.) and perhaps how many times per week they access different sources. A summary question would be something like, How satisfied are you with all the opportunities you have to learn? This should provide the information to determine whether employees are sufficiently engaged with learning.

In conclusion, the digital revolution is not likely to require new measures or reports, but is likely to require rethinking your measurement strategy with regard to data acquisition.

Comments

  1. While I agree with your conclusion 100%, I’m concerned about the quarterly survey solution you propose as a way to collect data to document the distributed learning experiences they are involving themselves in and to cope with the onslaught of Big Data potentially headed our way. It seems that your solution is crafted more to mollify learning professionals who are asking, “do we really have to change how we do things?”

    With xAPI maturing, tools like Degreed emerging onto the market and the ability to use open APIs to gather data from non-LMS tools, why would we revert to a quarterly survey? I think there would be value in knowing how many courses (and from where) employees take outside of the LMS and some gauge of what they learned would be useful. This revolution calls for new weaponry!

Speak Your Mind

*