CHAPTER FOUR: PROJECT EVALUATION PRINCIPLES
4.7 EVALUATION PROBLEMS
information from monitoring evaluation can affect the quality of the summative evaluation report which constitutes a treasure of lessons gathered for the good of existing and future programs and projects, because the report consolidates all relevant historical events from the beginning to the end of the project.
In the case of NALERP, shortcomings were manifest. The project planning had been inadequate because it was not able to include important elements such as environmental factors, financial resources and effective involvement of stakeholders. This may be one of the causes for difficulties experienced by NALERP during implementation (poor contracting relations, lack of resources, etc). Regarding evaluation, the summative report indicates there were weaknesses in monitoring reports which provided incomplete information on the project’s progress. Although summative evaluation would have been used, other methods of data collection such as interviews, observation, and a documentary review of monitoring reports, could have been a rich source of relevant information for building up a reliable historical account of the NALERP. The lesson learned from this case study is that the process of formative evaluation should be adequately carried out from the beginning, integrating all key stakeholders involved in evaluation, and appreciating the need for, and the importance of, documentation from reports, not only for the benefit of the ongoing project, but also for the good of other existing and new projects.
In fact, all the projects in developing countries that are financed by international organizations or agencies were carefully evaluated, contrary to the small number of domestically financed projects, which were evaluated by consultants who came from overseas, but were hired by local governments or via technical assistance agreements. Hence, many development projects in developing countries were not satisfactorily evaluated before their implementation, because they did not request international support. As a result, waste of resources and irrationalities were common and included even in investment procedures, although many efforts were made to achieve rational, optimal and comprehensive planning. A number of these countries did not have any experience regarding project evaluation practices. Their ordinary ways of deciding on investment issues were primarily based on partial investigation, and on political, social and administrative considerations, instead of on full financial and economic calculations. The lack of local experts in project evaluation resulted in the hiring of external consultants whose services were very expensive.
Another problem, met by Sang (1995:19) in regard to project evaluation, was the inadequacy of statistics and of other information needed for detailed studies of projects as in the case of NALERP. It was also observed that most projects in developing countries are not institutionalized because projects are evaluated and selected on an ad hoc basis without establishing a formal setup. Furthermore, methodologies and procedures for the adequate project evaluation that international aid agencies proposed are generally too time-consuming for, and demanding on, local resources, and too complicated and sophisticated for developing countries to use in cases where there is no presence of international aid agencies.
4.7.2 Resistance to evaluation
Wells, (2007:11) observes that government agencies and non-governmental organizations are increasingly focusing on project or program outcomes, organizational improvement and cost efficiency because of increased competition for decreasing funding, and on the use of business models which imply a move towards accountability and which reflect the desire of society to see people visibly benefiting from social program or project. Wells (2007:5-7) found that this is the reason why people resist evaluation. They perceive evaluation as a way to judge them, to reveal areas of project success or failure, and they fear that failure would result in punishment, for instance, the cutting of funding. They suspect that funders force them to do evaluation for their own benefit, rather than for the good of other stakeholders. They believe that money, spent
on evaluation, is diverted from direct services to the client and that it would be better spent in assisting others. Naturally, people don’t like statistics, especially if they reveal a point of failure that could have consequences. People perceive evaluation as consuming time that could be spent better. They also don’t like to do things in a way that to their mind is not right. .
4.7.3 A case of unethical issues encountered in evaluation
Problems of ethics arise particularly in data collection and reporting. Dale (2004:9) gives an example of such a situation from his own experience. He was made responsible for agricultural development projects in West Africa. Evaluation was high on his list of priorities and he soon began to undertake evaluations of programs and projects, which he was to continue throughout their life cycle. Although it was difficult for him to evaluate large scale projects, especially to do so during the windy and rainy season, he expected that cooperation with others would solve his problems. After all, evaluation was supposed to be participatory and go beyond technical, economic and social-economic considerations.
However, in the practice of evaluating agricultural development projects, he came upon serious ethical issues. He was informed that evaluators normally kept evaluation information to themselves and that the same was expected from him. But this external influence meant that he would be deprived of the exercise of objectivity and independent judgment. His evaluation would not be realistic because evaluators used to re-invent the program or project theory in order to find out what would work well, instead of concentrating on programs and projects as planned by those in charge. He confirmed that evaluation problems often resulted from the failure to make a difference in real world situations but he suspected that his own views would not fully match those of the majority of his fellow evaluators, although he did not dare to openly formulate such a concern. For him, however, “the objective of evaluators should be identical with that of planners and implementers: create sustainable benefits for the target groups of projects, programs and policies”. He knew that evaluators should be part of a team of actors, who work together for a common purpose. But he was convinced that evaluators do not necessarily go along with the objectives, pursued by the beneficiaries, planners and implementers, although this is the case in most projects and programs.
Although Dale does not reveal what he did to cope with the corruption surrounding the project evaluation, he does make clear that there is a need to question the credibility of information from
evaluation reports, the production of which may have involved unethical practices. That is why all the different approaches of evaluation mentioned in earlier sections should be carefully considered in order to decide on the best way of conducting worthwhile and reliable evaluation.
As far as the GADP (Chapter Seven) is concerned, the project suffered from non-transparency in the communication of information from evaluation reports. This could well be because the evaluation was not undertaken in a systemic way. It was conducted from the perspective of a small range of stakeholders and without clear purpose. The GADP environment, which was not favorable, was not considered in the evaluation process. The GADP evaluators were not qualified enough to carry out such an important activity and the methodology they used to collect data was inadequate.