Drawing on the discussion above, the following key principles for evaluation can be set out.
1. Evaluation should lead to policy change
The prime purpose of undertaking evaluation is that it informs key decisions. Such decisions may be to change policy. For example, it may lead to a policy budget being increased, decreased or the policy itself being abandoned. It may also lead to different objectives of the policy being specified and, most likely, will lead to the policy being delivered in different ways – possibly to different target groups. Alternatively, the policy decision may be that no change is required and that the programmes are “on track”.
2. Evaluation should be part of the policy debate
If it is to be at the heart of policy making, evaluation cannot be confined to providing a historical review of previous policies. Instead, evaluation has to lead to policy learning so that current policies may be amended in the light of this knowledge and new policies developed from such learning.
3. Evaluators should be “in at the start”
Evaluation is, as noted above, vital in the formulation of new policies.
Those skilled in evaluation techniques can make major contributions to policy development, most notably in helping policy makers to clearly formulate policy objectives. Without the input of evaluators, policy objectives may not be specified at all, or be expressed in such a way that they cannot fail to be achieved, or be specified in such opaque language that it is impossible to determine at a future point whether or not they were achieved.
The role of the evaluator is to ensure that policy makers specify clear and tangible objectives as policy is being developed. A second merit in evaluation being included at an early stage is that budgets for evaluation are specified when programmes are being formulated. A third advantage of evaluators being “in at the start” is the methods to be used to determine the success or otherwise of the programme are clearly brought to the attention of the policy makers. Finally, policy makers are made aware that a programme is to be evaluated and the criteria of success that will be used to assess effectiveness are agreed in advance by all parties.
4. Evaluation techniques should always use the most appropriate methodology
Subject to the provisos outlined below, the most appropriate evaluation techniques should be employed. The “Six Steps to Heaven” is a simple method of assessing sophistication in this area, and for impact evaluation of medium to large sized programmes we recommend that at least a Step 4 approach should be used. By this we mean that, as a minimum, the beneficiaries of a programme or policy should be compared with a “control group” of otherwise similar firms or individuals, but who did not participate in the programme. By comparing outcomes for the two groups a crude estimate can be made of policy impact. Governments, with their substantial access to statistical data, always have the opportunity to formulate such control groups, even if on some occasions their own statistical agencies are unwilling to collaborate in such work. The importance of evaluation however is such that the statistical agencies should be mandated to develop this aspect of their evaluation culture.
However, in order to understand more deeply the processes of how policy works and to involve stakeholders in policy learning qualitative methods such as peer reviews and case studies also have their place. These methods are not suited for robust impact estimates but are necessary for many other aspects of evaluation.
5. Evaluation should apply to all policies and programmes
It appears that some policies and programmes are evaluated many times, and with some rigour, whereas others seem to either escape scrutiny altogether, or are evaluated in a much less challenging manner than other policies/programmes. This is extremely unfortunate since the overall purpose of policy is to ensure that, at the margin, all policies are equally effective.
6. International comparisons should be made where necessary
Finally this Framework concludes that for some policy areas, evaluation can only be undertaken on an international basis. For example, comparing the impact of tax regimes is best undertaken across countries. As noted above, international comparisons and peer reviews are appropriate for reviewing policies at the regional or local level. In this respect, OECD can serve a valuable function as a “clearing house” for information on policy effectiveness, for producing harmonised data and in having access to experts in this field.
Notes
1. This term was coined in OECD (2004a). It refers to evaluation only being considered after the objectives and targets have been set, and the programme has been in operation for some time. When evaluation is “at the end of the line” it can only serve as a historical auditing function. OECD (2004a) contrast this with the more preferred “Integral Evaluation”. Here Evaluation is integrated into the policy process. So, as the policy is being developed, consideration is given to how it will be evaluated. This has four merits. The first is that it ensures that the objectives and targets of the policy are clearly specified. Second, it ensures that the necessary data collection can begin immediately and is “built in” to the programme. Thirdly, it ensures that the evaluation budget is identified. Fourthly, it ensures that the progress of the programme is monitored so that modifications and improvements can be made in the light of evidence.
2. “Evaluation of SME Policies and Programmes”, background report for the 2nd OECD Conference of Ministers Responsible for SMEs, Istanbul, June 2004.
3. Source: European Commission State Aid Scoreboard, Spring 2004 update.
4. The awards specified by Cooper were 1.107 in 1997, 1.067 in 1998 and 1.097 in 1999 – all in billions of USDs.
5. For formative and prospective evaluations, the opportunities for undertaking quantitative/statistical approaches are generally much less, unless they draw upon previous summative evaluations.
6. Examples of external evaluators are generally those not employed as public servants by the government. These will include private sector consultants and research contractors, such as academics.
7. www.uwex.edu/ces/lmcourse/#.
8. For example, when the objectives of a programme are unclear the evaluator might ask the question “What outcome from the programme would you identify as failure?” Experience of asking this question is that it is initially met with hostile silence. Frequently it is then followed by specifying outcomes that are [close to]
impossible. These might include … Nobody is interested in the policy. After consideration, however, more realistic objections normally emerge.
9. Additionality refers to the proportion of the supported activity that would not have gone ahead without the support, deadweight activity that would have gone ahead anyway, displacement refers to reductions of activity elsewhere (e.g. in other firms) as a result of increased competition from the supported activity (e.g.
the firms receiving assistance).
Policies and Programmes
© OECD 2007