CHAPTER 2: LITERATURE REVIEW
2.11 Other Training Evaluation Models
39 | P a g e Incremental Importance of information
Kirkpatrick’s model assumes that data from each of the preceding levels of the evaluation process is more informative and essential than the last (Bates, 2004). Consequently, this assumption has led to perceptions that establishing level four results will provide the most useful information about training program effectiveness. However, the weak conceptual linkages that occur within the model and resulting data generated do not provide a sufficient basis for this assumption (Homklin et al., 2014). As a result of the critiques and pressures in organisations to measure the return on investments of the pieces of training, the model has been continuously improved to overcome difficulties in training evaluations (Al-Mughairi, 2015). The critiques of Kirkpatrick’s model helped in the expansion of the model as well as the development of new training evaluation frameworks. The following section describes the developed frameworks in improving the shortfall of Kirkpatrick’s model.
40 | P a g e only three training outcomes, namely, learning, individual performance, and organisational results.
This model ignores the influence of elements on the reaction level, which is the first level in Kirkpatrick ‘s model. Instead, Holton ‘s model focused on the hypothesis of trainability and the primary variables that affect the ability, motivation, and perceptions of the work environment by trainees (Noe & Schmitt, 2006). As suggested and outlined by Antos and Bruening (2006), the primary moderating or mediating factors that affect the learning outcomes are trainee reaction, motivation to learn, and being able to learn (cognitive ability). The primary moderating or mediating factors influencing individual performance outcomes are motivation to transfer knowledge, transfer conditions within the organisation and transfer design. Finally, the primary moderating factors influencing the results of the organisation are as a result of the linkages between the training and organisational goals, expected utility, as well as external factors (Antos &
Bruening, 2006).
2.11.2 Kaufman and Keller Evaluation Model
Kaufman and Keller (1994) expanded on Kirkpatrick’s four levels of evaluation by adding value to society and continued improvement, rather than a summative measurement (Watkins et al., 1998). The Kaufman and Keller’s model increased the scope of the reaction level of Kirkpatrick’s model by including enabling to the reaction, then renaming the level one ‘input process’, the mode further added the fifth level that measures societal outcomes and the impact of training on the society (Preskill & Russ-Eft, 2005). The five levels of evaluation if applied outside the training, it can allow for consideration of other performance improvement interventions (Kaufman & Keller, 1994). Therefore, the Kaufman and Keller’s model looks into both internal and external results of training, linking to performance and organisational development (Passmore & Velez, 2012).
Kaufman et al. (1996), are of the view that the Kirkpatrick model does not recognise the effects of training on society. As a result, they recommend the consideration of the worthiness of the resources in the evaluation of training. There are five levels in Kaufman ‘s evaluation training model. The first level (Level 1) there is a separation of inputs and process. This level is more concerned with the quality of the available organisational resources, and efficiency of the methods as well as resources invested in the process (Kaufman et al., 1996). Watkins et al. (1998), argue that the separation of input and process at Level 1 is highly useful for training evaluators as it provides managers and decision-makers with valuable information enabling for continuous
41 | P a g e improvement of organisational training and education efforts. Level 2 is focusing on measures and acquisition of competencies; Level 3 is skills transfer in the work environment, and Level 4 measures the training output towards the organisation. Finally, Level 5 evaluates the responsiveness, consequences, and payoffs of the society as a result of the training intervention (Kaufman et al., 1996). Stokking (Stokking, 1998), on the other hand, believes that some of the extended aspects of Kaufman’s model are unclear and lack clarity. As a result, the model needs further adjustments. As an example, “there are differences between the desired chronology of activities and the aspects of levels and importance or no clear distinction is made regarding the implementation condition of training’’ (Stokking, 1998, p. 180). In this model, implementation, learning objectivities, and their achievements are all incorporated into Level 2 (acquisition). These are regarded to be the indicators of training effectiveness and proper training implementation (Stokking, 1998). Furthermore, Kaufman ‘s model is good in theory. However, less has been offered by the model with regards to practical application (Topno, 2012). Therefore, there are similarities in this model with the Kirkpatrick as it provides similar information without consideration of other contextual factors.
2.11.3 Philips’ ROI Model
Phillips (1995) represented the results in terms of money on return on investment (ROI) to the four levels of Kirkpatrick’s model. According to James and Roffe (2000), Phillip’s five-level evaluation approach translates the worth of training into a monetary value which, in effect, addresses return on investment. Phillips framework provides the trainers with a logical framework to view ROI both from human performance and business outcomes perspectives. However, the measurement goes further, comparing the monetary benefit of the programme with its costs (Topno, 2012).
According to Downes (2017), the Phillips model focuses on how to (a) collect data, (b) isolate training impact with other impacts of other factors, and (c) account for intangible benefits.
According to Phillips (1996), many programs have failed to deliver on the set expectations. This failure has led to the sponsors of the programs to request for the return on investment justification.
To justify their contributions, training institutions have increased their interest in the processes of return on investment (Phillips, 1996, p. 11). Kirkpatrick ‘s four-level model fails to measure the economic value of a training programme or its benefits. Therefore, Phillips (1996) developed a way to measure the contributions of training by adding a fifth level to Kirkpatrick ‘s four levels
42 | P a g e (ROI-Return on Investment) and also expanded Level 1 (Reaction) to include trainees ‘intentions to apply knowledge from the training programme to their workplaces. According to Phillips (1996), the return on investment refers to “a ratio based on the monetary benefits to the costs of the training” (McKenna & Beech, 2014, p. 214). Preskill and Russ-Eft (2005) proclaim that defining return on investment is a multifaceted and complex task within a complex system. They further emphasised that the calculation of ROI is more of a subjective measurement. Consequently, it has not produced accurate training investment measurements.
2.11.5 Kraiger, Ford and Salas’s Learning Outcome Model
Kirkpatrick ‘s four-levels evaluation model does not provide reliable measurements at the levels of behaviour and results (Patterson & Hobley, 2003). Kraiger et al. (1993) proposed three learning outcomes that training evaluations must be made up of and these are cognitive, skill-based and affective outcomes. As Kraiger et al. (1993) convey, training organisations often view learning outcomes as a change only in verbal knowledge. Through research, they discovered that this was a restrictive viewpoint on learning transfer and thus developed the classification scheme of learning outcomes based on taxonomies of Bloom’s 1956 and Gagne’s 1984. Bloom’s Taxonomy proposed cognitive-based learning outcomes, those beyond the recall of verbal knowledge, whereas Gagne’s taxonomy reinforced the need for assessment of cognitive, skill-oriented, and attitudinal learning outcomes. As a result of these taxonomies, Kraiger et al. (1993) in characterizing the learning outcomes, they proposed cognitive, skill-based and affective learning outcomes. Furthermore, the constructs were identified for each learning outcome. In Kraiger et al.
(1993), the cognitive results include three indicators which are verbal knowledge, knowledge organisation, and cognitive strategies. Cognitive perspectives focus on the dynamic acquisition, organisation, and application of knowledge.
Once the trainees develop a foundation of verbal knowledge, he or she can begin to focus more on the procedural knowledge acquisition through practice. Once the trainee has built the foundation, then he or she is then able to start to apply the skills to real-life work situations (Kraiger et al., 1993). The second identified category is skill-based outcomes, which focuses on the development of technical and motor skills. In this learning outcome category, the trainees begin to be able to reproduce the trained behaviours through what is known as compilation. In this phase of learning
43 | P a g e transfer, the performance of the trainee becomes less error-prone and operates in a more fluid and focused manner. At this point, individuals are more likely to identify the appropriate situations for using a skill (Kraiger, et al., 1993). The third and final specified category is a useful learning outcome. Kraiger et al. (1993), used the research conducted by Gagne to support that attitudes can determine behaviour and performance. Having to do with motivational tendencies, they identified three other secondary indicators; motivational dispositions, self-efficacy, and setting of goals.
The first two outcomes of this model are similar to levels 1 and 2 of the Kirkpatrick model, however, the levels are not considered to be hierarchical as compared to the Kirkpatrick. This means that one level does not necessarily lead to the next level (Patterson & Hobley, 2003).
Although this model has some advantages over Kirkpatrick’s model, its shortfall is that it does not give or provide guidance on how to determine the financial value or cost-effectiveness of the training (Beech & Leather, 2006). Moreover, the model emphasises on the effects of training only at the trainee’s level and neglects the impact of training at the organisational level. It further ignores the possible delays between training and on-the-job performance improvement. Lastly, on the model, there is insufficient opportunity to enable the collection and incorporation of the subjective views of trainees (or trainers) into the evaluation (Beech & Leather, 2006).
2.11.6 Brinkerhoff ‘s Success Case Method Model
Kirkpatrick ‘s model fails to consider the possibility of multiple variables that can contribute to the impact of learning opportunities; the model assumes that performance results can be achieved only through training (Brinkerhoff, 2003). The success case method is a way to evaluate the impact of training on business that is aligned with and fulfils a deliberated strategy (Brinkerhoff, 2003).
Two fundamental parts involved in a successful case study process are a) participants who are the most successful and participants who are the least successful at applying learned knowledge and skills from the training programme, b) draws a sample from the most and least successful cases (Brinkerhoff & Dressler, 2003).
Brinkerhoff (2003) extended the training evaluation model to six stages (goal setting, program design, program implementation, immediate outcomes, intermediate or usage outcomes, and impacts and worth). This model differs from Kirkpatrick’s by including the earlier phases of the
44 | P a g e training process, need assessment, design, and implementation, into the evaluation phase (Homklin, et al., 2014). According to Downes (2016), Brinkerhoff’s model focuses on qualitative analysis and is not restricted to learning, and can be used to analyse any significant changes in the organisation such as how to implement new processes. This model involves the identification of the most and least successful cases within a learning program and followed by studying such cases in detail. Downes (2015) asserts that comparing the successes to the failures, enables the organisation to learn what to change, and ensures success in future endeavours. As a result of these learnings, the organisation can produce success stories and show the value of the training, based on the lessons learned. However, in overall learning evaluation, Downes (2016) recommends using other training evaluation methods other than the Brinkerhoff Success Case Method.
Although the model has particular worthiness, it still needs refinement on the identification of success factors that relate to a specific type of work by the training experts and instructors. This clarity is necessary because of the difficulties that trainees face when they are back at their work environment (Casey, 2016). The following section discusses the justification for the researcher in adopting Kirkpatrick ‘s model for this study.