CHAPTER 2: LITERATURE REVIEW
2.14 Relationship between the Four Levels of Kirkpatrick’s Model
Kirkpatrick ‘s model assumes that there is a correlation between the four training outcomes:
reaction, learning, behaviour, and results (Santos & Stuart, 2003). The levels represent a causal chain such that positive reactions lead to more significant learning, which produces a higher transfer of knowledge and ultimately, more positive organisational results (Bates, 2004). Bramley and Kitson (1994) contend that each level provides different data and evidence. Thus, analysing all four levels will produce useful information about individual and organisational outcomes as a result of the training. According to Saks and Burke (2012), previous studies have investigated the relationship between the four levels of evaluation outcomes. As an example, a meta-analysis by Hauser, Weisweiler, and Frey (2018) using Kirkpatrick’s framework displayed another study that used ‘Happy sheets’ alone to evaluate training effectiveness. The findings indicate that the approach of ‘Happy sheets’ alone was not enough; consequently, the results were insufficient in determining the programme effectiveness. As a result of the lack of proper evaluations of training programs, the objective of the meta-analysis was to investigate the effectiveness of personnel development programs in academia by using level 1 to level 4 (reaction, learning, behaviour, and results) of Kirkpatrick’s model. In this study, the literature search included one-hundred and twenty-three training programs conducted between summer 2012 and 2014; these training programs clustered into 26 primary studies. The study used repeated measurements design to examine the short and long-term effects of the personnel development program, and these measurements were done one week before, directly after the training and six weeks after the training program. The findings of the study found a moderate to high long-term effect and a high
53 | P a g e short-term impact on learning. These results meant that the training programs led to significant changes in participants’ knowledge immediately after the training, which reflected an immediate knowledge gain. Furthermore, the effect decreased six weeks after the training, indicating knowledge retention and a sustainable effect (Hauser et al., 2018). They further examined effects in terms of participants’ behavioural changes and found a small to moderate long-term impact. The discovery meant that the participants exhibited increased desirable behaviour six weeks after the training. These long-term effects on the learning and behavioural levels led to the conclusion that the positive influence of the training programs did not fade out and lasted several weeks after the training (Hauser et al., 2018). This finding is an indication of the sustainability and effectiveness of the training. The meta-analysis also found medium to high levels of heterogeneity among the primary studies concerning both short and long-term effects.
Borate et al. (2014) in their study to evaluate the effectiveness of continuous quality improvement training in a multinational company in India, found the statistical significance of the four levels of Kirkpatrick in evaluating the effectiveness of training. The study surveyed 330 employees of quality departments within companies across different engineering industries. On level two of the model (learning), the results confirmed that change in learning of the trainees has a significant influence on training effectiveness. Level three (behaviour) had a significant influence on training effectiveness. The level four (results) indicates that change in result has a considerable influence on the training effectiveness. The study went further to investigate how the behaviour of trainees affects the other levels in the model. The results showed that the change in trainee’s behaviour had a significant influence on learning and positive outcomes (Borate et al., 2014).
Lim and Morris (2006) reported their research findings that the training characteristics, especially the training environment and the trainer, influenced the satisfaction of the learner or trainee about the program. They further developed a model for the relationship between learner characteristics, training characteristics and organisational climate characteristics as independent variables, and learner satisfaction, academic achievement awareness and transition awareness as dependent variables. This model explains that trainee characteristics (self-efficacy, motivation, expectations, and abilities), training characteristics (training design, training environment, training content, and trainer) and organisational climate characteristics (mentor support, peer support and organisation
54 | P a g e support) all have a direct influence on learner satisfaction and the achievement of the organisational goals. Yoon (2018) found trainee’s characteristics, training characteristics and organisational climate characteristics to all have a direct influence on the training effectiveness.
Also, Anderson, Ellwood and Coleman (2017), the trainees’ perception of the work environment influences their learning motivation, which in turn influences the degree of his or her learning of the training content. Finally, the learning of the trainee leads to improved organisational results through behaviour change (Yoon, 2018).
The results of the study conducted by Wartenweiler (2018) revealed that learning goes hand in hand with the reaction of trainees. This finding also resonates with the statement by Kirkpatrick (1996), who in his model indicated that reaction criteria could not be separated from the learning criteria. The research findings also confirmed that the program had a significant learning effect as the trainees could well articulate the program. In general, the quantitative findings of the research showed a substantial increase in the survey means scores from program start to program end. In short, students’ cognitive knowledge and attitudes, as well as their behaviour, had a positive impact on education for the socialization of the organisation (Wartenweiler, 2018). The pilot study by Ghofranipour et al. (2018) on improving intern’s patient-physician communication skills found that a theory-driven educational intervention could alter the knowledge and self-efficacy of the trainees. As a result, interns that increased their level of knowledge and self-efficacy were able to communicate better with their patients. This statement supports the study findings by Wartenweiler (2018). Another study by Stiehl et al. (2015) found training effectiveness to be limited if the trainee’s motivation to transfer the learned skills is low. Furthermore, Wartenweiler (2018) found a strong correlation between self-efficacy and communication scores, whereby changes in communication skills score were dependent on the changes in self-efficacy scores of the trainees.
Another study by Calvo et al. (2018) examined the effectiveness of a social enterprise MOOC (massive open online course) program using Kirkpatrick’s model. The study invited learners from different cultural contexts across the North and South countries. The findings demonstrated positive reaction, attitude and learning skills (Levels 1 and 2) of the learners, and this showed evidence of positive results from the training program. The findings of the study also indicated that they were useful towards the reaction and learning of trainees as they were able to demonstrate
55 | P a g e that they were generally happy with the experience of the courses and would highly recommend it. The respondents in their study also claimed that the training provided them with a multidimensional global context involving training content from different industry organisations across the globe (Calvo et al., 2018). This above statement confirms that indeed, the relevance of the training content plays an essential role in the behaviour and learning of trainees. Additionally, looking at the effectiveness of the program at an individual and organisational level in terms of the transfer of learning to professional practice, the findings indicated a positive impact (Levels 3 and 4). “Most of the learners reported a change of behaviour, sense of confidence and increased in their motivation to continue or start with their organisations” (Calvo et al., 2018, p. 18). Evidence to support these changes in learner’s behaviour was supported by the study findings, which showed that learners were generally doing things differently at an organisational level. These findings support the model of Kirkpatrick and Kirkpatrick (2006), which suggest that change in behaviour and motivation to learn can lead to improved organisational performance. Calvo et al. (2018) advise scholars and evaluators to use mixed-methods approach and training evaluation frameworks such as Kirkpatrick to capture outcomes of training effectiveness and impact.
The findings of the meta-analysis conducted by Turner et al. (2018) in investigating the use of reaction criteria as a method to evaluate training, found a moderately positive relationship between trainees’ reactions and performance. In their concluding remarks, Turner et al. (2018) confirmed that reaction is an indirect measure of learning and performance. Likewise, they cautioned that the moderate relationship found between reactions and learning does not guarantee the sustainability of learning in the long-term. As a result, they recommended that attention should be given to environmental and situational factors as they can have an impact on trainee learning and training transfer.
Another study conducted by Gessler (2009) evaluated 43 training courses with 335 participants to investigate the correlation between the first three levels of Kirkpatrick’s model. The results of the hypothesis testing found no correlation between trainees’ reaction and learning. Because of these results, Gessler (2009) concluded that evaluation of reactions alone is not sufficient to justify the quality of rendered services. These findings are consistent with Holton (1996) critiques about the causal assumptions of levels by Kirkpatrick’s model. Conflicting the results of the studies
56 | P a g e mentioned in the paragraph above is the research by Craig, Hall, and Phillips (2016), when they evaluated the outcomes of the interprofessional learning program. In their study, they used all four levels of Kirkpatrick’s model to evaluate the outcomes of the program. They used all the four levels of the model to assess 156 medical students’ reaction, perceptions and attitudes concerning improved skills and transfer of learning to professional service. The results of the research showed a positive change in attitude and collaborative skills, which demonstrated that the program was effective in improving the skills and attitudes of the students. The study showed evidence of a positive relationship between learner satisfaction and skills development (Craig et al., 2016).
Holton (1996) included the trainees’ reactions as a measure for the learning environment that affect learning behaviour. He accomplished this through a complex role of moderating the relationship between motivation to learn and the actual learning.