CHAPTER FOUR: PROJECT EVALUATION PRINCIPLES
4.3 FORMS OF EVALUATION
Different forms of evaluation, taken together, point to a wide range of roles for evaluation. The evaluation can be conceptually classified into seven categories or forms, namely proactive, clarificative, interactive, monitoring, impact-focused, realistic, internal and external evaluation.
4.3.1 Proactive Evaluation
Owen and Rogers (1999:41) hold that proactive evaluation takes place before designing a programme. It helps planners of the programme to make decisions on the type of programme that they need. Rossi and Freeman (1979:35) observe that the forefront of the process of programme development and efforts of evaluation are largely analytic and are oriented towards a better understanding of the aspects of a social problem, which includes its scope, origin and propensity to intervention. Owen and Rogers (1999:41) maintain that the major purpose of proactive evaluation is to provide input to help deciding how best to build up a programme before the planning stage. Proactive evaluation is an important evaluation form at the conceptualization stage of a project. Failure to carry out effective proactive evaluation will result in a project design that does not meet the requirements of a customer or client.
4.3.2 Clarificative and interactive evaluation
Owen and Rogers (1999:42-44) state that clarificative evaluation focuses on elucidating the internal structure of a programme and its functioning, and studies the causal relationships that link program activities with intended outcomes. Clarificative evaluation should be carried out during the phase of the project design in order to clarify the structure and the project objectives, and during project implementation when there are misunderstandings about how the project should be implemented. As far as interactive evaluation is concerned, Owen and Rogers (1999:44) argue that it gives information about programme delivery and implementation and can be concerned with documentation or incremental improvement of a programme. The evaluator provides findings and facilitates learning and decision-making. Owen and Rogers (1999:44-45) observe that interactive evaluation provides information oriented toward improving the programme. Interactive evaluation targets middle managers and the project team, and is carried out during the project implementation phase.
4.3.3 Monitoring evaluation
The United Nations Centre for Human Settlements (2003) views monitoring as a constant process in all the phases of the project life cycle. This process provides managers with significant indicators on the project impacts, both negative and positive. As Rossi and Freeman (1979:164) indicate, monitoring evaluation assesses whether or not a programme is being implemented according to its design and whether or not it is meeting the needs of its specified beneficiaries. The UNDP Handbook on Monitoring and Evaluation for Results (2002:6) argues that monitoring is a continuing task aimed at providing main stakeholders and management with indicators of performance or of problems that have been identified in the process of achieving expected results. Monitoring deals with the real performance of a project as against what had been planned and generally involves gathering and analyzing data on the processes of the project implementation, releasing results and suggesting recommendations necessary for corrective measures. Owen and Rogers (1999:46) argue that managers undertake monitoring evaluation when they need indications of how successful the programme and to justify its expenditure.
4.3.4 Impact evaluation
Rossi and Freeman (1979:36) write that an impact evaluation is a judgement on the degree to which a programme makes changes in the desirable direction (objectives). Rossi and Freeman (1979:37) state that conducting an impact evaluation requires a plan for the collection of data.
The data will permit the evaluator, in a persuasive way, to demonstrate that changes occurring are a function of a particular programme or project intervention, and can’t be accounted for otherwise. Rossi and Freeman (1979:290-293) further assert that there is agreement that randomized experimental designs are ideally the most appropriate and reliable way of measuring impact, since they provide the best means of controlling a variety of potential biases. However, it can be argued that the above assertion may not be valid for project and programme evaluations today. Experimental designs may be difficult or too expensive to carry out and there is a trend towards employing both qualitative and quantitative evaluation approaches. The authors also point out that impact evaluations are necessary to compare different programmes or test the value of new efforts to address a particular problem in the community.
4.3.5 Realistic evaluation
Pawson and Tilley (1997:216) argue that programmes and projects deal with real problems, and therefore realistic evaluation is a valuable contribution to attaining a realistic formulation of the programme’s policies for the benefit of all stakeholders. Pawson and Tilley (1997:215) indicate that realistic evaluation is primarily focused on the intervention of all stakeholders and the optimal allocation of resources for the programme’s success. Pawson and Tilley (1997:217) state that realistic evaluation sees the project and programme development as an effort to collect the knowledge needed to identify tasks to be done, their beneficiaries and the conditions in which they have to be done. This knowledge can be collected through an understanding of past projects or programmes, and through empirical research. Realistic evaluation is a useful approach in the phases of conceptualisation and planning of a project.
4.3.6 Internal and external evaluation
According to Worthen and Sanders (1987:37), the internal evaluator is almost certain to know more about the programme than any outsider, but he may be too close to the programme to be completely objective. It is not common to question the objectivity of an external evaluator and it can be said that this, in addition to professionalism and experience, is the advantage of an external evaluator. At the same time, the external evaluator will never learn as much about the programme as the insider knows. As Owen and Rogers (1999:49) indicate, it is important for all of those involved in any evaluation to choose the most appropriate way of proceeding.
4.3.7 Lesson from these forms of evaluation
These forms of evaluation can contribute to the project’s effective evaluation if they are properly used throughout the project life cycle, not broken up into separate and independent evaluations but seen as a series of complementary evaluations. This is what O’Sullivan (2004:3-4) says: the project life cycle influences the types of evaluation undertaken for different purposes. Thus, the purpose can be program or project improvement, or to determine its short-term and long-term impacts, to identify its strengths and weaknesses, or to justify additional resources, to obtain support for a particular program/project approach, or to get clarification on the next phase of a project, dissemination of findings, in response to attacks on a program/project or the strengthening of additional resources. With reference to the statement of O’Sullivan, relationships between purposes of evaluation, forms of evaluation and phases of a project can be established as it is illustrated in Table 4-1.
Table 4-1: Integration of evaluation purposes and evaluation forms in the programme/project life cycle
Purpose of evaluation Form of evaluation Phase of the project life cycle
Potential stakeholders Program or project improvement Interactive evaluation Implementation phase sponsors, manager and team
leader Short-term and long-term impacts of a
program /project
Impact evaluation Implementation phase Closeout phase
sponsors, manager and team leader
Identification of a program or a project’s strengths and weaknesses
Proactive or realistic evaluation
Conceptual phase Closeout phase
sponsors and owners Justification of additional resources,
obtainment of support for a particular program or project approach
Clarificative evaluation
Implementation phase sponsors, manager and team leader
Dissemination of findings Monitoring, impact evaluation
Implementation phase closeout phase
sponsors, manager and team leader, senior project personnel Response to attacks on a
program/project and strengthening of additional resources
Clarificative evaluation
Implementation phase sponsors, manager and team leader
Source: the researcher based on O’Sullivan (2004:3-4).
The table indicates that these different purposes can be achieved by different stakeholders in the same programme or project, but at different times of its life cycle depending on the form of evaluation that is to be undertaken. These relationships mean that the project evaluation is not a discontinuous activity but an activity that overlaps all phases of the project life cycle. Evaluation which is done in this way can influence the project management because the processes of planning, executing and controlling can be carried out in iterative way until the project is successfully terminated as Figure 4-2 illustrates.
Figure 4-2: Relationships between the project management processes
Source: Project Management Institute (1996).
Initiation processes
Planning processes
Controlling processes
Executing processes Closing
processes
Reinforcing this point of view, the UNDP Handbook (2002) states, that evaluation is a learning process which continues throughout the project and provides useful information that is integrated into the decision-making process. The United Nations Centre for Human Settlements (2003) points out that the definition of evaluation indicates that a project is regularly and methodically reviewed. Whether the project is completed or in progress, the assessment is particularly concerned with the project design, implementation and result. The aim is to determine the project’s relevance, coherence, effectiveness and impact and, at the same time, to collect relevant information so that it becomes possible to identify mistakes, to learn from these and to make necessary changes. Sanders (2001) adds, that evaluation brings about conceptual clarification, clear and reasonable directions for development, choice of decisions when many viable options are provided and protection when changes are proposed.
The literature does not provide the best form of evaluation, but indicates how the forms can be integrated in the management processes to achieve the project objectives. However, if evaluators wait until the stage of project implementation, their evaluation may end up contributing only a little to an improved quality of project management and of the lives of the project stakeholders.
The forms of evaluation as discussed above inform on the nature of a particular evaluation, the participants, and when it is to be undertaken, depending on the purposes of the evaluation.
Their description also provides orientation as to appropriate approaches and methods that are used to gather relevant information for potential users. The following section deals with evaluation approaches.