The purpose of the evaluation should be considered in order to identify clear aims and objectives for the process. It is helpful to decide where the bound- aries of the evaluation should lie. How much or how little is to be evaluated?
It can be costly and time-consuming to hold an evaluation. There is a cost involved in collecting information and preparing documentation as well as in holding the necessary meetings. You might save some expense by consid- ering the extent to which already existing information might be used.
The purpose of an evaluation determines, to some extent, the audience for delivery of the results. An outcome evaluation might be for the sponsor of a project but a performance evaluation might be undertaken for a service provider partway through a project. The nature of the audience may also determine the way in which the results of the evaluation are reported and used.
One of the key decisions in the planning stage is who should carry out the evaluation. If, for example, the evaluation was of the outcome of a major project paid for by public funding, an external and independent evaluator would usually carry it out so that the results would be credible to the general public. A formal evaluation of a collaborative project might be held by a group of the key stakeholders, each able to report back to their own group or Evaluating the project 177
organization. An external evaluator might be costly, but an internal evalua- tion will draw on time and energy that might be better devoted to carrying out the project. It is important that those conducting the evaluation should be able to understand the context and the issues that were raised in the project, but it is also important to try to find people who can be open and objective. This may mean seeking evaluators who did not have any direct role in the processes or outcomes of the project, but who know and under- stand your organization well.
In some projects the choice of those who should be involved is constrained by need for confidentiality. Although it is very important to bring a wide range of perspectives into the evaluation, it is not usually appropriate for confidential information to be shared outside the small group that would normally need to access it. It is important to involve key stakeholders in evaluations, but any confidential data must be managed very carefully.
There may be a number of roles to consider, including whether particular people should be involved in considering the questions or only in providing evidence.
Evaluation involves making judgements about the value of the project.
Value judgements are relative and subjective, and it can be very helpful to have some explicit standard against which judgements can be made. In many projects it can be difficult to make comparisons with anything similar. When there are quality standards for any of the outcomes, these provide a frame- work that can be used, perhaps alongside targets for timescales and resource use in achieving the necessary level of quality. Another source of comparable data might be found in benchmarks where these exist for similar activities.
Benchmarks have been established for many processes and are available from industry, sector and professional bodies.
Some of the key questions to consider in carrying out an evaluation of the planning and implementation of a project are:
࿖ Were all the objectives achieved?
࿖ What went well and why?
࿖ What hindered progress?
࿖ What was helpful about the project plan?
࿖ What was unhelpful about the project plan or hindered the work?
࿖ Did we accurately predict the major risks and did the contingency plans work?
࿖ Was the quality maintained at an appropriate level?
178 Managing projects in human resources
࿖ Was the budget managed well and did we complete the project within the budget?
࿖ Was the timing managed well and did we complete the project within the timescale?
࿖ Did anyone outside the project team contribute towards achieving the project?
࿖ Did anyone or any other departments hinder the project activities?
To address these questions, you will need information from a wide range of sources. If you plan to carry out this type of evaluation it is helpful to make a plan to ensure that you collect the appropriate data when it becomes avail- able, rather than expecting to find that it is still all available at the end of the project. In particular, it is usually worth recording the comments and deci- sions made in review meetings and in any meetings held to resolve problems that are encountered.
Example 15.1
Collecting information for an evaluation
The steering group of a financial services staff development pro- gramme decided to plan the evaluation at an early stage in the project so that information could be collected throughout the process. They considered how to collect data about the performance of the project in each of the three dimensions of time, cost and quality. This was to include:
࿖ data about the planned schedules for activities and the completion times of actual events;
࿖ data about the budget, from the estimates and initial forecasts and from the records of financial performance;
࿖ data about the quality of accommodation, equipment and any training materials used;
࿖ data about presentation and content of the programme;
࿖ data about the impact that the training had on performance of par- ticipants.
They recognized that there could be many different perceptions about what was delivered and how it might have been improved. In order to consider the different views, they planned to collect data from the programme providers, from participants and from the line managers
Evaluating the project 179
of participants. Data was also to be collected from other senior man- agers, staff from the HR department and some of the key account clients of the participants. They also planned to assess whether the project had achieved its longer-term objectives six months after the conclusion of the training programme.
There are a number of methods that can be used to collect and analyse data.
Some data collection usually takes place as part of the project activities and can contribute to evaluations. For example, records kept for monitoring pur- poses may be used to make comparisons between activities. Records of meetings and other formal events may also provide useful data relating to the sequence of decisions made and issues discussed.
Other data might be collected purely for the purposes of the evaluation.
For example, interviews or questionnaires might be used to collect a number of different views, or focus groups might be used to explore issues with a group of people together. Observation or role play might be useful if data is needed about how activities are carried out. The balance between qualitative and quantitative data is important because each can supplement the other, and it is difficult to achieve an overall picture if only one type of data is used.
When you are planning the data collection for an evaluation it is usual to try to obtain a range of different types of data. If only quantitative data were available you would only have information about things that could be counted. Although this is often very important, you would have no infor- mation about quality. You would want to know that the project had achieved both formal quality standards and any other expectations identified in the objectives. Opinions of those who are customers of the project are very important if you are evaluating outcomes. The views of the teams who have contributed to the project are important in evaluating the process.
The methods you choose to collect information will be influenced by the availability of resources. However, the key things to take into account are:
࿖ thecost of obtaining the information in relation to its contribution to the evaluation;
࿖ thenumber of sources from which information should be obtained if suffi- cient viewpoints are to be represented to ensure that the results are credible;
࿖ thetime it will take to obtain and analyse the information;
࿖ thereliability of the information obtained;
180 Managing projects in human resources
࿖ thepolitical aspects of the process – for example, some ways of gathering information may help build up support for the evaluation.
Direct contact with those involved in the project might be the only way in which sufficient information can be obtained to make the evaluation of value.