Training evaluation software for Windows

Training evaluation is the most neglected part of the organisational learning process and often means different things to different people. This difference of opinion and focus on the sometimes conflicting issues by different stakeholders has made it difficult to develop a consensus on just how training should be evaluated.

In this article, we take a look at the objectives that stakeholders seek from training evaluation and how the various training evaluation theories have sought to address these objectives.

Who wants what?

The key stakeholders in the training process are the employees attending the training events (training delegates), their managers and the training providers. The one drive they all have in common is to be able to have an adequate response to the question “Was it worth it?”

When the training delegates look to evaluate their training, their main priority is whether the time spent on the course was worth it. The generally think in terms of personal benefits such as:

• Will it help me do my job better/easier/faster?
• Will it help me increase my earnings?
• Will it improve my career prospects?

At the other end of the scale, management evaluates training in terms of the needs of the organisation; its efficiency, profitability, potential for growth, public image etc. The look to assess the value of training by asking:

• Did it help us achieve our strategic goals/KPI’s? (increased profits, reduced complaints, etc.)
• Was it the most efficient was of developing the skills we need to achieve our goals?
• Was it worth the time and cost to the business?

Sitting in the middle, is the learning provider -the person or organisation delivering (and in many cases, designing) the training. They must consider the learner’s needs in order to engage them and help their development while at the same time steering their learning and development in line with their organisations goals.

A proper training needs analysis will, right from the start, include the question of evaluation and take into account factors such as in-house performance management systems, competence frameworks, current learner skill etc. This will translate into a set of learning objectives for the training which can be easily measured at the evaluation stage.

For training providers, the key question is, are we delivering instruction that meets our clients’ objectives? Having a positive answer to this question is important for a training providers continued economic well-being, as a consequence their attention is focused on questions such as:

• What parts of the training material needs improvement?
• What elements of the training material delivery needs improvement?
• How can we ensure the transfer of learner back to the workplace?

Practical Theories

Much has been written in the last 60 years about evaluating learning and many models and theories now exist. The most influential three come from:

• Donald Kirkpatrick – The four levels
• Jack Phillips – The fifth level
• Paul Kearns – The baseline

The four levels

Donald Kirkpatrick introduced his four levels as a model for evaluating training in 1959:

• 1. Reaction - the trainee’s thoughts and feelings about the training event
• 2. Learning - the extent to which the training increases trainee knowledge and capability
• 3. Behaviour - the extent to which new learning is transferred back to the workplace
• 4. Results – the extent to which the training impacts the business of the trainee

It is said that this model was introduced at a time when organisations were less complex and customer expectations were different from what they are today. Learners had less sophisticated needs and training was mostly trainer led and classroom based. This has led others to seek more sophisticated approaches to the evaluation of training.

However Kirkpatrick’s four level framework is still the model most widely adopted by learning and development professionals today.

The fifth level

Jack Phillips added a fifth level to Donald Kirkpatrick’s model which involved collecting level 4 data, converting the data to monetary values and comparing them to the cost of the training program to show the return on the training investment.

Phillips’s approach can be problematic because of the conversion process includes hypothetical factors and focuses on measurement after the event. It is a retrospective approach which doesn’t necessarily quantify the desired added value at the start of the training process. This can lead to using after-the-fact return on investment (ROI) data to assess how well a training activity worked (or not, as the case may be). Discovering failure after the event can be expensive.

The Baseline

Paul Kearns' Baseline approach seeks to incorporate some innovative and common sense elements into the work of Kirkpatrick and Phillips. Most significantly, Kearns suggests that evaluation should be approached with a philosophy of beginning with the end in mind. As he says:

"The purpose of evaluation is not to prove that a specific piece of training worked, it is to transform the whole learning process and significantly raise the probability of a training investment providing a good return."

In this respect, measuring a "good return" involves looking at how training adds value and quantifying the value in monetary terms – increased revenue or reduced overhead costs.

A lot of the advocates of the ROI method argue that it can be calculated for any learning activity in any environment. However most examples used to demonstrate the ROI approach use sales/profit-making studies to illustrate the process. This is has limited application for public sector and non-profit organisations.

In Closing...

Bringing us up to date, some training evaluation theorists are now moving away from a rigid ROI approach and are working with the view that increased profits or reduced overheads should follow positive changes in individual behaviour. A return to Donald Kirkpatrick’s third or fourth level perhaps?

Marblewire is a London, UK based software company. We make Trellis, a Windows-based training evaluation software application designed to make it easy for you to collect and analyse feedback on training events and training impact.