The SA HIV/IMCI Effectiveness Study employed a mixed methods approach to investigate the effectiveness of IMCI implementation in South Africa. In this study, qualitative and quantitative data were collected sequentially. Qualitative methods were used to explore experiences of IMCI training and implementation, and quantitative methods were used to provide numerical
estimates of IMCI implementation coverage. The two methodologies are complementary in
30 HORWOOD CM
nature, so that the results of the two components add information and clarify the overall findings and, therefore, provide a more holistic picture of IMCI implementation.
3.2.1 Focus group discussions (FGDs)
Implementation of public health interventions demands behaviour change, but factors that determine whether health workers change their behaviours or practises are rarely investigated (92). Developing an understanding of factors that influence the performance of health workers can improve implementation of guidelines, leading to improvements in provision of public health interventions like IMCI. This includes how health workers experience learning, and what
determines their ability to acquire new knowledge, and implement this knowledge in routine practise in the workplace. Their experiences provide critical insight into understanding why some aspects of an intervention work well and others do not. Training is one factor influencing health worker performance, and may lead to development of knowledge and skills required for implementation, but training alone does not result in comprehensive implementation and high coverage of the proposed interventions (65, 93).
Focus group discussions (FGDs) were conducted with IMCI trained health workers and child carers. FGDs are a form of group interview, which are not intended to be objective or
representative, but have the advantage of allowing researchers to elicit a multitude of views, and to explore and contrast the views of participants (94). Focus groups explicitly use the interaction between participants as part of the methodology, so that group processes help participants explore and clarify their views in a way that would not be possible in a one-to-one interview. This method is particularly suitable for exploring participants’ knowledge and experience, and for examining work place cultures (95). This approach was chosen as the best methodology to explore how IMCI trained nurses experienced IMCI training, whether they acquired the skills required to implement IMCI, and what the barriers and enablers for implementation were, in order to develop a deeper understanding of the determinants of health workers ability and willingness to implement IMCI.
Focus groups are particularly useful for exploring attitudes of participants, which often are not easily encapsulated in reasoned responses to direct questions (95). FGDs were also used to explore the attitudes of health workers and child carers to the implementation of the HIV component of IMCI, particularly the inclusion of routine checks for HIV in all consultations with sick children, and to explore the particular challenges related to provision of HIV care. Using this methodology with a sensitive topic like HIV does have possible disadvantages. The presence of other research participants may compromise confidentiality, so that people may be less willing to discuss personal experiences in a group setting, and dissenting voices may not be heard.
31 HORWOOD CM
3.2.2 Comparative survey of IMCI trained health workers
The quantitative component of the study was a comparative survey of IMCI trained health workers, undertaken at PHC clinics. This component provided quantitative data about health workers performance in implementing the IMCI guidelines, when compared to an expert IMCI practitioner. The purpose was to estimate coverage of IMCI interventions, identify gaps in IMCI implementation, and provide quantitative estimates of the findings of the qualitative component.
With the drive to evidence-based medicine, randomised controlled trials (RCT) have been applied increasingly in the field of public health and health policy. Although evidence-based health care is important and desirable, it must go beyond RCTs, which are frequently inadequate for scientific assessment of the performance of large-scale interventions, because it is unlikely that the conditions found in RCTs can be replicated in real world conditions (96). Interventions being evaluated must be carefully monitored and supported during a RCT, and conducted over a well-defined and limited geographical area. The need for control groups, without access to the intervention, limits the use of RCTs to new interventions, so this type of methodology cannot easily be applied to interventions already established over a large area, or to assess the implementation and sustainability of such interventions in routine practise. RCTs may fail to answer some of the relevant questions about large-scale public health interventions, so
alternative and complementary approaches are needed to provide valid, generalisable evidence to add to the knowledge of programme performance. Causal pathways for public health
interventions involve not just biological, but also behavioural steps that need to be understood and measured to demonstrate a logical sequence between intervention and outcome (96).
Therefore, RCTs have limited value in assessment of the effectiveness of an intervention in real- world conditions, and would be unsuitable for evaluating IMCI implementation, which is already well-established and widely implemented in South Africa.
We decided on a comparative survey of IMCI trained health workers, where consultations with IMCI trained health workers were directly observed and the findings compared with a defined standard of performance as demonstrated by an IMCI expert. The technique of direct observation of consultations has been shown to provide the most valid and reliable picture of what health workers do (97). However, even observed performance may not represent routine performance, since health workers know that they are being observed. Without a control group, it may not be possible to attribute any improvements in coverage that are demonstrated as being the direct result of IMCI implementation or infer that IMCI has directly led to improvements in quality of care (91). When this study was conducted it was more important to determine whether the goals of the programme were being achieved and to identify any shortfalls, rather than to establish any causal relationship. The study did not aim to measure impact on childhood
32 HORWOOD CM
morbidity or mortality, or on behavioural indicators. This type of evaluation, where provision, utilisation and/or coverage are measured, can be termed a performance evaluation, as opposed to an impact evaluation, and it assesses how well programme activities have met the expected objectives (91).
Our methodology is different from that used in WHO IMCI health facility surveys or in other published evaluations of IMCI, because the unit of sampling was the IMCI trained health worker, rather than the observed child or the health facility. This allowed a more detailed analysis of the patterns of individual health worker performance in assessment, classification and management of sick children than has been previously reported. We were able to clearly identify
implementation gaps and weaknesses in health worker skills. Although the emphasis of this study is appropriately placed on HIV management, diarrhoea, severe pneumonia, and malnutrition continue to be important causes of death among South African children, and so it was important that all components of IMCI implementation be evaluated. This study provides new knowledge in the field of IMCI evaluation research, as well as about the effectiveness of IMCI as a mechanism for implementation of child survival interventions, particularly the determinants of health worker performance and HIV care in a high HIV prevalence setting.
33 HORWOOD CM