CHAPTER FOUR: PROJECT EVALUATION PRINCIPLES
4.4 EVALUATION APPROACHES
Reinforcing this point of view, the UNDP Handbook (2002) states, that evaluation is a learning process which continues throughout the project and provides useful information that is integrated into the decision-making process. The United Nations Centre for Human Settlements (2003) points out that the definition of evaluation indicates that a project is regularly and methodically reviewed. Whether the project is completed or in progress, the assessment is particularly concerned with the project design, implementation and result. The aim is to determine the project’s relevance, coherence, effectiveness and impact and, at the same time, to collect relevant information so that it becomes possible to identify mistakes, to learn from these and to make necessary changes. Sanders (2001) adds, that evaluation brings about conceptual clarification, clear and reasonable directions for development, choice of decisions when many viable options are provided and protection when changes are proposed.
The literature does not provide the best form of evaluation, but indicates how the forms can be integrated in the management processes to achieve the project objectives. However, if evaluators wait until the stage of project implementation, their evaluation may end up contributing only a little to an improved quality of project management and of the lives of the project stakeholders.
The forms of evaluation as discussed above inform on the nature of a particular evaluation, the participants, and when it is to be undertaken, depending on the purposes of the evaluation.
Their description also provides orientation as to appropriate approaches and methods that are used to gather relevant information for potential users. The following section deals with evaluation approaches.
Table 4.2: Possible evaluation activities during the life of a programme/project
Program/project Conceptualization
Program/ project planning Program/project implementation
Program/project completion
• Reviewing relevant literature
• Assessing needs
• Conducting focus groups
• Analyzing cost effectiveness
• Creating personnel projections
• Establishing timelines
• Estimating costs
• Identifying procurement alternatives
• Monitoring program/project activities
• Developing databases
• Assessing program/project functioning
• Determining short- term impact
• Assessing long-term impact
• Determining
program/project strengths
• Identifying areas for subsequent improvement
• Assessing cost effectiveness
Source: O’Sullivan (2004:3).
Projects need to be evaluated and from the above table, it is clear that evaluation must start in the early stages of the project. In the phase of conceptualization, evaluation is required, for example, to assess stakeholders’ needs and the potential success of the project (feasibility study) with reference to the available resources and existing projects in similar fields. In the phase of planning, there is a need to estimate resources and costs. During the phase of implementation, evaluation is conducted to assess the successful implementation of the project components, the degree to which the project goals and objectives are being achieved, and the relevance and impacts of the project.
Pawson and Tilley (1997:215) argue that successful evaluation of the project in all phases of its life cycle is conducted from the perspectives of various stakeholders involved in the project, because these have different interests, expectations and purposes which in some cases are conflicting. O’Sullivan (2004:7) maintains that it is useful to consider them with particular attention as the purposes of evaluation greatly differ. For this reason, O’Sullivan (2004:7) indicates some approaches for evaluation that are useful especially if one wants take the perspectives of different stakeholders into consideration. Various evaluation approaches have been developed to meet these requirements.
4.4.1 Evaluation approaches as developed by O’Sullivan
O’Sullivan (2004:7) found that a meaningful evaluation is based on objectives, management, expertise, adversary and participant approaches as indicated in Table 4-3.
Table 4-3: Evaluation approaches
Approach Primary audiences
Objectives: the focus is on objectives to determine the degree of their achievement
Program/project sponsors, managers Management: the focus is on generating information to assist
program/project decision-makers
Program/project managers, staff Consumer: the approach looks at programs /projects and
products to determine relative merits
Public, program/project sponsors Expertise: the emphasis is on establishing peer and
professional judgments on quality
Peer group, public Adversary: the approach examines programs/projects from
pro and con perspectives
Program /project sponsors, public Participant: The focus of the approach is on addressing
stakeholders’ needs information
Participants, staff, community members
Source: O’Sullivan (2004:7).
The objective-oriented approach considers the extent to which objectives of a program or project has been achieved (O’Sullivan, 2004:62). When evaluators do not work together with key stakeholders during the evaluation design and implementation, it becomes very difficult to achieve the evaluation objectives be achieved, because in some cases those objectives are not clear and need clarification through the intervention of different stakeholders. Usually, the objective based approach can provide relevant information concerning a program or project’s effectiveness. However, the strong stress on outcomes may be an obstacle to collecting information in relation to those aspects, which contributed to the achieving or not achieving of the program or project objectives (O’Sullivan, 2004:7).
The decision-making (management) approach) provides information that administrators of a program or project need to determine future program or project direction (O’Sullivan, 2004:62).
The approach considers the different aspects of program or projects from the perspective of their designers and administrators (O’Sullivan, 2004:7). Stufflebeam (2000) quoted in O’Sullivan (2004:7) and Soumelis (1977:27) argue that the model of evaluation based on context, input, process and output overcomes the weaknesses of the objective-based approach (evaluation based only on outcomes). Program and project evaluation establishes links between inputs, outputs, intended benefits, outcomes and the overall impact for people as Figure 4-3 indicates. The example is a health promotion project, which was planned in more detail, implemented, and evaluated to assess direct changes and effects, and the overall impact on people (better health of children).
Figure 4-3: Links between inputs, outputs, outcomes and impacts.
Overall impacts for people
Children enjoy better health than earlier
Effects for people (Outcomes)
Children are Children are less prone to Children are less prone to better nourished infections from their environment water-borne diseases Relatively direct changes
More nutritious Food supplements more hygienic safer water food is made and served are served household routines is used
are practiced Outputs
Food supplements Unpolluted water are accessed is available from wells
Knowledge about motivation knowledge cleaner water nutrition for change about hygiene is available is enhanced is increased is enhanced from ponds Implementation tasks
Conducting supplying the constructing lining fencing the training food to schools the wells the ponds the ponds preparing
the training procuring providing the recruiting material the food construction the workers materials
organizing deciding on the training
the scope deciding on the of training deciding on the well/ ponds to be done target children for
school feeding
deciding on the assessing assessing
beneficiaries of training the need for facilities the feasibility of options
………..
Inputs
Training technical tools and administrative logistics administrative expertise expertise materials personnel management systems
Source: Dale (2004:58)
As indicated in Figure 4-3, the inputs include, for example, human and material resources, management systems and technical and training expertise, etc. These elements are translated into the recruitment of workers for lining and fencing the ponds, supply of materials for constructing the wells, and procurement of food to schools, etc. The outputs are the availability of unpolluted water and accessed food supplements. Changes made are that the use of clean water contributes to decrease infection diseases and improve children’s health. The example indicates that the evaluation includes all the health processes (from the beginning to the end), not only the outcomes.
As O’Sullivan (2004:8) highlights, the evaluation model based on context, input, process and output, is used to surmount the weaknesses of the objective-based approach because this model provides strategies of evaluation for assessing needs of a program or project (context), determining what is needed to start a program or project (input), monitoring the program or project as it opens out to improve its performance (process), and measuring the program or project impact (product). However, this model has a limitation in that it does not provide the feedback and involve participants or the public at large as a potential audience in the evaluation.
The evaluation is limited to project designers and managers. Attempts to use a systemic approach in evaluation are thereby undermined.
The consumer-oriented approach considers the program or project effects on stakeholders, such as staff and community. For this reason, evaluators often use the consumer-oriented approach as one of multiple approaches that are adapted for the purpose of generating effective evaluation (O’Sullivan, 2004:62). O’Sullivan (2004:7-8) suggests that an analysis of the cost- effectiveness of program or project options can be included in this approach, along with meta- analyses, comparing program or project outcomes across various program or project strategies.
The expertise approach gets leaders of programs or projects to look for peers and other experts in appropriate fields. The peers and experts are invited to review a program or project and to analyze its merits. This in turn helps the program or project implementers to understand events occurring in the program or project, or to solve problems (O’Sullivan, 2004:8).
In particular circumstances, programs or projects may be put on trial. Project leaders present their cases to court for an evaluation, applying the adversary approach. A jury may be selected, witnesses are called, arguments for the defense and the prosecution are heard and finally there is a verdict on the merits of the program/project. An example of such a case is
when a dispute is raised within a program/ project, and neutral judges arbitrate. The approach requires however a thorough investigation of the case concerned (O’Sullivan, 2004:8).
As far as the participant approach is concerned, its central focus is on the value of undertaking an evaluation which involves key stakeholders, so that perspectives of participants are engaged in the process of evaluation (O’Sullivan, 2004:8).
O’Sullivan’s model provides a range of evaluation approaches from which an evaluator can select the appropriate one for the specific evaluation he wants to conduct. He has to take the evaluation purposes, its participants and their perspectives into account. Evaluation can be undertaken in the different phases of the project life cycle. However, a systemic orientation and integration of the environment concerned are lacking in O’Sullivan’s evaluation approaches.
4.4.2 The model of evaluation approaches according to Guba, Lincoln and Zadek
Guba and Lincoln (1989) hold, that approaches including measurements (size, weight, volume, score, value, etc), description and judgment, can be suitably used for effective evaluation. Guba and Lincoln (1989:22) argue that the role of the evaluator is that of a measurer and he/she is required to have the expertise to apply the relevant measurement instruments. This approach is essential for project control and monitoring during the implementation phase. It is also useful at the end of the project for the purpose of carrying out summative evaluation.
Guba and Lincoln (1989:28) find the description approach helpful because it describes weaknesses and strengths in relation to certain stated objectives. The role of the evaluator is that of a describer, although the measurement approach is retained as well. This approach can be applied both during the conceptualization and the closing phases of a project. During conceptualization, the description approach can be used to gather all the details necessary for an adequate design of the project. During the closing phase, description can be used to assess the impacts of the project and whether its goals have been achieved.
Guba and Lincoln (1989:30) acknowledge the judgment approach because they found that there is a great need to make judgements on the basis of available data and in the light of predetermined aims. Although the evaluator should recognize the value of measurement and description approaches, his roles as a measurer and describer are subordinate to his role as a judge. This approach is regarded as a useful tool during project implementation. The project
team, especially the project managers, needs to make appropriate judgements based on facts and descriptions in order to take appropriate actions. But Zadek, (1999) citing Guba and Lincoln (1989), asserts that, although the approaches of measurement, description, and judgment are important for evaluation, they fail to appreciate in particular:
• Real power relations existing and pressuring the evaluator within the evaluation process.
These power structures affect the relationships between those being evaluated and the practical ability of an evaluator to be a neutral outsider.
• The plurality of value bases existing simultaneously within the evaluation process, as well as multiple interests, agendas, and perceptions; and
• The consequential need to cope with multiple perspectives within the evaluation process that are not mediated by resort to the assertion of facts, but to the mediation of perspectives.
Zadek (1999) suggests that the mediation approach can be useful, taking into consideration various perspectives of different stakeholders for a negotiated solution to a specific problem situation. The approach consists in negotiation and mediation between stakeholders. These are effective tools that enhance communication between the parties involved in the project. Zadek (1999), citing Guba and Lincoln (1989), estimates that the evaluator is both a facilitator who elicits the views of different stakeholders, and a mediator who brings the stakeholders to a level of consensus as to what happened in the past and what should happen in the future. The evaluator does not seek to identify, but rather to highlight and mediate between different views rooted in different interests and worldviews. The evaluator does not seek to determine a solution, but rather encourages the various stakeholders to reach an agreement. For this reason, Owen and Rogers (1999:40) emphasize that this approach should be used throughout the project cycle.
On the basis of this discussion it can be argued that no particular evaluation approach is seen as the most appropriate for a specific phase of a project. The challenge is for the evaluator to select an approach or a set of approaches which he/she considers appropriate to the project, depending on the purpose of evaluation, the perspectives of stakeholders and the project environment.
4.4.3 Evaluation approach of UNCHS (United Nations Centre for Human Settlements) UNCHS (2003) suggests that proper evaluation is based on the following elements: preparation of an evaluation plan, indicators development, structuring of indicators, undertaking of evaluation and learning from evaluation as indicated in Figure 4-4.
Figure 4-4: Stages in the Evaluation Process
1. What is evaluation?
The focus Preparation
2. Why is the evaluation being done?
3. Who will the evaluator be?
4. How will the evaluation be managed?
5. Stating what is to be evaluated
6. Choosing the indicators
7.1 Option:
Questioning
7.2 Option:
Watching
7.3 Option:
Reading
8. Providing a comparison
9. Assessment 10. Change in Practice
Design Collecting the information Interpretation Assessment Implementing
the results
11.Return to review evaluation needs and design
Source: Feek (1988).
From Figure 4-4, one can see that evaluation is systematically done through five steps:
preparation of evaluation, design of evaluation, data collection, data analysis and implementation of the results. UNCHS (2003) developed these stages as follows.
The first step involves planning evaluation and provides the following information:
• Identify the real needs of evaluation which are clearly understood by all key stakeholders involved with the project.
• Determine the project objectives, expected outcomes and impacts.
• Identify the actors concerned with evaluation, because the evaluation should be divided into three categories: self-evaluation done by project staff, outside evaluation normally carried out by someone else with no direct connection with project, and joint evaluation done both by project staff and evaluators from outside the project.
• Specify the scope of the evaluation (UNCHS, 2003).
The second step identifies performance indicators, which are regarded as a measurement of project progress in terms of objectives’ achievement. Indicators should be quantitative or qualitative and help to make comparisons between objectives and results. They help actors involved with the project to identify problems, revise objectives, draw lessons and discuss decisions to be made for improving the situation. There are road signs for change in the right direction. These could be used as a way of logically linking inputs, outputs and outcomes of the project (UNCHS, 2003).
The third step structures performance indicators, which have been identified and integrates them into the overall project design and implementation. They should be structured into categories with regard to policy, programme and operational levels (UNCHS, 2003).
The fourth step involves undertaking evaluation, which includes the methodology of evaluation and the design of the evaluation report. The methodology deals with data collection methods which may include groups interviewed, timeframe and methods used such as interviews, meetings, participant observation, questionnaire, etc. For a quantitative and/or qualitative study, the methodology also indicates methods of data analysis that should be used for measuring the project impacts at local, regional and national level. The evaluation report should include the project relevance, efficiency, effectiveness, sustainability, impact, and cost-effectiveness (UNCHS, 2003).
The fifth step is concerned with learning from the evaluation. For evaluation to be effective, it needs to be viewed as under the ownership of all key stakeholders of the project. In that case the implementation of results and joint learning may produce the desired outcomes (UNCHS, 2003).
The Public Health Agency of Canada (1996) suggests that the framework for project evaluation should indicate the following elements:
• Describe activities: clear definition of measurable goals and objectives of the evaluation;
• Identify reasons for success: identification of performance indicators that should be used in measuring the achievement of the project objectives and assessing the project impacts;
• Assess impact: the measurement of the project impacts requires collecting data, indicating the appropriate information needed, data collection and analysis methods to be used, and the participants who hold that information;
• Analyze data and interpret the results: This is about a summary and analysis of data collected and lessons learnt from the findings;
• Use the evaluation findings: These should be used both within the project for making change and by external stakeholders and other projects.
The model of evaluation adopted by the UNCHS (2003) which was designed by Feek (1988) and the model of the Public Health Agency of Canada (1996) are similar, except that the Public Health Agency of Canada added the element of assessment of impact. Like the model of the Public Health Agency of Canada, the model adopted by the UNCHS was limited only to evaluation methods. It did not provide any information about purposes and forms of evaluation.
This model can be used only in the phases of implementation and at the project’s closing down, for example to monitor its progress or to assess its impact. It is not concerned with the project’s feasibility study and planning. A project designed and planned in these conditions is more likely to be unsuccessful if evaluators postpone their evaluation until the implementation phase. In that case the feasibility study and planning are not evaluated and their quality, possibly poor due to inadequate environmental analysis or poor involvement of stakeholders, remains undetected.
The model does not establish links between management and evaluation and, therefore, the importance of evaluation for the project management. The model overlooks other elements such as leadership and the project life cycle, which are important factors for an effective evaluation.
Leadership motivates those who evaluate the project and solves potential conflicts among stakeholders during the evaluation process. The life cycle indicates the phases in which different