CHAPTER 8: DISCUSSION
8.6 Limitations
8.6.1 Ignoring Team Size Effect
It is observed that CCG team sizes differed in the pre-evaluation and post evaluation period. Under normal circumstances the team size ratio was one supervisor to 25 CCGs. Under study conditions the team ratio is one supervisor to 5 CCGs. The argument presented here is that the RCT was a complex intervention as more than one variable was responsible for the outcomes achieved.. The three variables were the HIV-adapted training, CQI supervision and a change in team size. The control arm
69
represented a change in team size whilst the intervention arm represented all three changes. The importance of team size is supported by the observation made by Liang et al (2008, p799) that larger teams are harder to monitor from a performance perspective and are associated with problems related to coordination and difficulties in communication. Kozlowski and Bell (2001, p12) argue that large teams are associated with losses in motivation caused by a “dispersion of responsibility”. It is argued that smaller teams do not have these problems and that the reduction of teams introduced efficiencies in both the control and the intervention arm. Liang et al (2008, p805) further argue that in attempting to save costs, it is more optimal for firms to decrease team size and hire a lower quality manager whilst keeping the quality of the workers fixed.
8.6.2 Limitations of a Retrospective Cohort Analysis
Retrospective cohort analysis is particularly subject to recall bias as it requires participants to attempt to recall activities that occurred in the past (Drummond, 2005;
Muennig, 2008). In addition, Gold et al. (1996, p75) asserts that retrospective cohort analysis may not collect data pertaining to costs and outcomes in a way that is useful to the cost analysis. In this study, this is particularly evident when considering that discounts and the time allocated to shared resources have not been recorded. The collection of financial costs was accurate, the true costs of goods and services was not systematically recorded, and in particular discounts, received for equipment. This highlights a shortcoming of a retrospective relative to a prospective cohort analysis. In a prospective analysis the required cost and health outcomes information is determined prior to the implementation of the intervention, thus avoiding issues related to the capturing of incomplete information required for a CEA.
8.6.3 “Thin” Information on the Supervision of the Control
CRH-UKZN’s primary motivation when implementing the RCT was to determine the effectiveness of the intervention relative to the status quo. Effectiveness in terms of the number of women exclusively practicing EBF and an increase in knowledge levels of CCGs were recorded in both the intervention and the control group. Unfortunately information pertaining to the monitoring and supervision of CCGs in the control group was “thin”. It would have been more beneficial to have had an in-description of the
70
supervision process in the control arm in order to make more accurate comparisons between the control and the intervention arm.
8.6.4 Duration of the Intervention
It is argued that the 15 month time horizon of the RCT was not sufficient in terms of determining the cost-effectiveness of the intervention. Possibilities do exist that with the initial implementation, its initial cost-effectiveness ratio may significantly decrease at a later date. This can be attributed to the theory of learning by doing where the effect of the experience gathered by CCGs over time results in a more efficient production of health outcomes. It infers that once CCGs have inculcated the principles of CCM and CQI management they may require less supervision resulting in declining costs over time. Counte and Meurer (2001) argue that firms are quick to discount the benefits of CQI without understanding that it is a long term management process.
This process takes time based on Roger’s (cited in Counte and Meurer, 2001) Theory of Diffusions of Innovation which argues that firms go through five steps in the adaption of an innovation. These steps are acknowledgement of a need; an evaluation of how the innovation can solve the problem; the decision stage where team accepts or refuses the innovation; testing the applicability of the innovation; and finally the confirmation stage where there is continued adoption of the new methodology. It is argued that the RCT was terminated at the fourth stage of Roger’s (ibid) steps which is the implementation stage where the focus was on problem solving. It is argued that the confirmation stage was not achieved which is characterized by continuous improvement.
In highlighting how long it takes to embed CQI a process, Counte and Meurer (2001) argue that it takes at least five to seven years to actually implement a CQI management and supervision system although an important caveat is that this is in the context of a large firm with several reporting structures. Whilst the implementation of CQI is expected to take a much shorter period of time in the Nompilo Project RCT which has a much flatter reporting structure, it is important for an organization to understand at what phase of the CQI implementation process they are operating in.
71 8.6.5 The Limitations of Relying on a CEA
Zangwill and Kantor (1998, p913) highlight that the CQI method of focusing on specific actions that have improved in a process, relies on implicit assumptions. These are that the improvements made are separate and distinct, and will not have an impact on each other or furthermore “should not cause any decreased effectiveness other parts of the process”. CCGs have reported an increase in confidence and knowledge as a result of the support and training that they received in the intervention. They stated that they were able to prepare better for their interactions with members of the community. It would have been helpful to determine whether this new found confidence resulted in CCGs increasing the duration of their counselling sessions with mothers in the community and whether this did not affect the number of clients they could have seen.
This highlights the short comings of using natural units as a measure of outcomes as a trade-off may exist where the quality of the interaction between CCGs and mothers has improved whilst the number of mothers reached decreases, thus reducing the cost-effectiveness of the intervention.