3.4 Theoretical framework
3.4.2 Phillips’ 5 level framework for ROI in training analysis
example, to believe they were less capable than other races, it would create a sense of weakened self- efficacy, not only for individuals of that race, but also for the collective (i.e. all people who belong to that race, including teachers and learners) (Bandura, 1995, Oettingen, 1995, Tschannen-Moran and Barr, 2004).
reported on via ‘happy sheets’ or programme feedback forms. Often, this level of measurement is either not included in the ROI analysis or a decision is made to consider level one’s objective to have been achieved if the average learner feedback score meets a predetermined criterion, such as an 80% learner satisfaction score using a Likert scale (Kirkpatrick, 1998, Phillips and Stone, 2002, Meyer et al., 2003).
Level 2 (Learning): The objectives identified at level 2 relate to the skills, attitudes and behaviours that are expected to be learned by the participants. In this case, the training relates to database skills and the impact of the programme at this level is measured by the improvement scores of participants on cognitive testing related to the course. The performance improvement scores are obtained by subtracting the mean post-test scores (‘post-training performance’) from the mean pre-test scores (‘baseline performance’) to obtain what the graphic refers to as ‘performance improvement’. In view of the fact that this study seeks to identify the factors that maximise the effectiveness of training and skills transfer at level 2 (and, via the chain of impact, the overall ROI realized), results are grouped in terms of various factors (independent variables), such as teacher student match or mismatch on race, home language or gender, and the improvement scores obtained at level 2 for each of these factors compared. Thus, in the example presented in Figure 3-4, one would conclude that maximum ROI is achieved for Course DB101 when teacher and student are matched in terms of race (improvement score of 13% at level 2) and home language (improvement score of 10% at level 2), as opposed to teacher student mismatch in terms of race (improvement score of 6%) and home language (improvement score of 2%).
Level 3 (Application): The example in Figure 3-4 illustrates the type of level 3 objective that may be identified. For example, management may identify the need for the training to result in improved ‘numbers of cases resolved on a daily basis’ per support agent/student. Thus, the objective might be stated as ‘Number of cases resolved on a daily basis’. This measures the application of skills learnt on Course DB101 in terms of the extent to which students/agents are able to apply what they have learnt at level 2 on the course to their job, as measured in this instance by an increase in cases resolved each day compared with pre-training performance (‘baseline performance’).
Level 4 (Business impact): Objectives at level 4 are either ‘tangible’ or ‘intangible’. In the example in Figure 3-4, an intangible business impact measure may be ‘improved (internal) customer satisfaction ratings’ (the database developers making use of the support agents would be considered internal clients in this case). A tangible impact measure might be
‘monthly database development team revenue from completed projects’, assuming that the
training of support agents to be more effective is considered to have contributed to the ability of the database development team to complete more projects on time and therefore increase their revenue.
Level 5 (ROI%): This final ROI% analysis is only possible if tangible benefits of training were identified at level 4. Furthermore, at least one of the tangible benefits would need to be expressed in financial terms. For example, in Figure 3-4, ‘monthly database development team revenue from completed projects’ was identified at level 4. At the point at which ROI analysis is undertaken, the formula for the ROI% would be:
ROI% =
Total financial benefit of training to date - Total cost of
training X 100
Total cost of training
Thus, in the example in Figure 3-4, we assume that 6 months have elapsed since Course A was completed, and that the training cost totaled R10,000. Monthly revenue in the database development team has increased from an average of R50,000 per month prior to training, to an average of R60,000 per month. Over 6 months, that means that the financial benefit to the company of the training has been R60,000 (R10,000 for each of 6 months since the training was conducted). Using the above formula, therefore, the ROI% is 500% ((R60,000-R10,000)/R10,000, multiplied by 100). This means that for every R1,00 spent on training, R5,00 was realised as financial benefit to the organisation as a result of the training.
The example in Figure 3-4 illustrates Phillips’ ‘chain of impact’. A satisfactory learning experience (level 1) is followed by effective learning and skills transfer occurring at level 2, as exemplified by the increased post-performance scores compared with the baseline performance scores. In turn, the learning that takes place at level 2 allows participants to perform more effectively in terms of the objectives related to ‘application’ in the workplace, as measured at level 3. Application of skills learned results in ‘business impact’ at level 4, viz. improved ‘customer satisfaction’ and increased revenues for the database development team, allowing one to calculate the ROI realized from this training intervention.
The focus of this study is on identifying factors that improve performance at level 2 (learning), on the assumption that, via the chain of impact illustrated above, this will ultimately result in higher overall ROI, whether in the university IS&T classroom or a corporate training venue.
Figure 3-4 Sample ROI analysis on Phillips' 5 level framework (Source: Adapted from Phillips (Phillips, 1997, Phillips and Stone, 2002))
4. Impact
8 3 5
3. Application Number of cases resolved on a daily basis Monthly revenue from completed projects Customer satisfaction rating
R10,000 R50,000
50%
R60,000
80% 30%
500%
5. ROI%
Satisfactory learning environment/experience 1. Reaction
2. Learning
Course DB101 (Database Skills)
Teacher Student Match
(Race) 52% 65% 13%
Teacher Student Mismatch
(Race) 49% 55% 6%
Teacher Student Match
(Home Language) 51% 61% 10%
Teacher Student Mismatch
(Home Language) 51% 53% 2%
Teacher Student Match
(Gender 53% 57% 4%
Level Objectives Independent Variables Baseline Performance
Post-Training Performance
Performance Improvement
Teacher Student Mismatch
(Gender) 54% 58% 4%