The Political and Organizational Context of Educational Evaluation
4.2 Rationality Assumptions Concerning the Policy-Context of Evaluations
In many applications of educational evaluation, e.g. program evaluation or school evaluation, the evaluation object or evaluandum, is a real-life setting in which (educational) goals are striven for through specific practical activities. Such practical situations can be abstractly described as a set of means and goals.
Goals are particularly important for all kinds of evaluations ranging from program evaluations and school evaluations to the construction of examinations. Goals can be seen as “desired states” or “ideal type processes”, which in their turn can be used as targets and evaluation standards. For example: a certain level of attainment on a math score by at least 80% of the pupils, or use of the computer for at least 20 minutes during 80% of the language lessons. Moreover, goals need not necessarily be defined in such a precise, operational and quantitative form. Even when there is just a general notion of the dimensions on which an existing situation should be improved after a period of program implementation, or, in our case, schooling, we could still see the situation as goal- oriented and assessable. In the latter case an expert committee could be used to make the assessment. Generally, when the evaluation criteria remain more global and “open”, the requirements on the substantive expertise of evaluators should be particularly high, as they could be seen as replacing the rigor of otherwise applicable structured and standardized instruments.
The presence of goals, specific or general, is an important feature of what can be referred to as the formal rationality of the evaluation setting. Where we could take
“evaluation setting” as both the evaluation object and the larger context in which this object and the evaluation itself is taking place. Evaluation itself can be seen as part of the rationality model applied to policy programs or to the functioning of schools.
The main features of this rationality model can be stated according to the following points (note that the concept of “program” which is frequently used in the points stated below should be interpreted in a broad sense, including, for example, a particular phase during regular schooling):
• the program to be evaluated has goals, and the evaluation can be guided by means of these goals;
• the program itself is to be seen as a set of means, for which there exist some reasoning with respect to the likelihood that they will indeed lead to goal attainment;
• planned, or “blue print” programs are also implemented according to plan;
• evaluation has the general form of empirical investigation of whether goals are attained on the basis of the program, i.e. the implemented set of means;
• evaluation activities can be carried out in a relatively undisturbed and unbiased way, free from all type of influence from parties with certain interests, and be conducted according to professional norms (i.e. standards of good evaluation practice);
• the results of the evaluation will be used for decision-making, which may be of a
“formative” or “summative” nature, and in this way practice will be improved.
Oftentimes the last point does not occur so straightforwardly. Evaluation use is often of a less “linear” and “instrumental” nature but rather a gradual and fuzzy process of influencing conceptions of relevant actors.
In a more general way one could say that to the degree that the evaluation setting departs form the rationality model, evaluation, in the sense of systematic, disciplined inquiry, will become more complicated.
Turning back to the goals requirement, it is one thing to say that goals may be general.
But, in a situation where goals are contested among relevant stakeholders, such as, for example, teachers and head teachers, it would be more difficult for evaluators to find a point of reference for designing an evaluation. In the context of large-scale policy evaluations the situation that interested parties differ about the program goals is not at all unlikely. Sometimes, there can also be large discrepancies between the official goals and the “real” goals of stakeholders. For example, in an experimental program for adult education, the official goals stated by the Ministry of Education were stated in terms of learning gains among participants, but for the teachers in the experimental program, prolonged employment and long-term tenure appeared to be more important (Scheerens, 1983).
In the context of policy-evaluations evaluators may find themselves in the midst of a totally politicized context, where partisans will try to use the evaluation to enforce their own position. In such situations a lot will depend on the professional standing of the evaluations, and the organizational independence of the way they can operate. In (internal) school evaluations the situation may not be so openly politicized, though nevertheless, important differences of goals and priorities in a certain domain of schooling may occur as well. And, with respect to external school evaluation, there may
also be differences about the key-function of the evaluator: to inform higher administrative levels, to enhance “consumerism” or to inform the school personnel about strong and weak points of school functioning.
The practical implication of all this for evaluators is that the initial stage of setting the priorities and choosing evaluation criteria and standards is particularly important.
Activities should not just be seen as analytic but also, maybe even more so, as practical and “managerial”, in attempting to come to terms on evaluation priorities with stakeholders, surmounting resistance and building commitment.
The second aspect of the rationality model, the existence of a somewhat explicit rationale about the “means” of the program being adequate to reach the goals, has given rise to a particular type of evaluation, namely “analytic evaluation”. Analytic evaluation is meant to articulate the basic means-to-end structure of a program. Sometimes this process is described as “making explicit the program theory”, or reconstructing the
“underlying program logic” (Leeuw, Van Gils & Kreft, 1999).
Contrary to the rational ideal, in actual practice, the link between goals and means can be considerably loose. Means, for example, can be chosen because they are really ends in themselves—“the medium is the message”, because they serve the more particularistic objectives of some stakeholders, or just because the program was not well prepared or designed. In such situations evaluators can, in principle, save a lot of time, efforts and ultimately money, by pointing out such weaknesses. In “analytic evaluation” the evaluator uses analytic methods (like review of existing research findings on the issue) to get an indication on how likely the proposed methods will lead to goal attainment.
According to the third characteristic of the rationality model, planned programs are also “really” implemented. Again, practice shows that this is not always the case and partial or even no implementation at all may be the reality. In case program objectives are assessed and program implementation has failed, the evaluation is called a “non event evaluation”. In order to prevent this, it is preferable to include an implementation check in the evaluation design. This can be done by means of measuring process indicators or by means of direct observations.
If the program, particularly the means, methods and approaches comprising the program are less straightforward, and implementation is more characterized as a process of “mutual adaptation” between ideas and individual preferences of practitioners, checking implementation becomes more complex. In such situations observational studies may work more like constructing “post hoc” program variants, which then may be associated with outcome measures later on.
The fifth rationality assumption about the evaluation setting is that evaluators will be in a position to carry out their professional job in relatively undisturbed way.
As we already saw when discussing the situation at the outset of program evaluations, where evaluators may become emerged in political debates among partisans, this condition too, is not always met in actual practice. But also when it comes to choosing evaluation methods and carrying out data collection this condition may be violated. When the stakes of the evaluation are, rightly or wrongly, considered high by practitioners, they will not have a neutral stance with respect to the data collection methods that are proposed by the evaluator. One could look upon evaluation methods as varying on a continuum running from “evaluator control” to “practitioner control”. Participatory observation, methods like a teaching writing a “log” of each lesson and open interviews
are examples of methods which are very much under the control of the respondents.
Standardized tests and external observations are largely outside the control of the practitioner. Scheerens (1983) describes a setting where evaluation apprehension of practitioners led them to renounce objective, evaluation-controlled measures, and plead for more open methods in which they themselves were the main providers of information.
In many cases a clear exploration of the purposes of the evaluation will help to overcome resistances. For example, in program evaluations and school evaluations organizational functioning rather than individual functioning of teachers is the evaluation objective. Nevertheless teachers may still think that they are the evaluation object, and they would need to be told explicitly that this is not the case.
Although most of the features discussed in this section are more prominent and are documented within the realm of program evaluation, the recommendations that were provided are also relevant for other kinds of educational evaluation, like school evaluation. The major recommendations are:
• to analyze program objectives carefully and enter a process of illumination of objectives among stakeholders, preferably resulting in overt commitments to goal statements and effect criteria that will be ultimately used in the evaluation;
• to critically assess the attainability of means-end relationships, in other words the likelihood that proposed program means will lead to goal attainment, preferably before empirical evaluation activities start;
• to empirically check the implementation of the program or set of activities that is to be evaluated;
• to be prepared for politically inspired negotiations about more or less reactive data collection methods and also for investing time and energy in communication and presentation of the intended evaluation activities and their objectives;
•to invest in communicating the evaluation results as a general process of illuminating issues to stakeholders, which may or may not lead to immediate impact on decision- making.
4.3 Gearing Evaluation Approach to Contextual Conditions; the Case