The Political and Organizational Context of Educational Evaluation
4.4 Creating Pre-Conditions for M&E
monitoring and evaluation should be fed back primarily to those organizational levels and administrative bodies that have the discretion to rule on the information. An implication of this rule would be that, in a situation where schools are autonomously responsible for classroom instructional processes, evaluative information on these processes should only be fed back to the schools’ management and staff.
• perceived threats at the political top or the top of the government bureaucracy for the possible outcomes of new or improved M&E provisions and activities;
• the degree to which M&E becomes part of a political controversy between ruling party and opposition;
• the stability of the political top during the period when the M&E activity has to be realized;
• a certain antagonism in supporting M&E project managers, based on perceived risks of creating a “critical conscience” in the education system;
• how teachers and school directors perceive of the “stakes” involved in M&E;
• how “user friendly” or “alien” the methods of M&E are perceived by school staff;
• in a more general sense the incentives and “disincentives” of actors like local
administrators, school managers, teachers and students to participate in or be subjected to M&E activities (at the cost side investment of time, loss of status, fear of being criticized, fear of weakening or loss of position should be considered);
• the position of stakeholders in a particular M&E activity in terms of commitment, resistance and “political” preferences for certain methodological approaches (for example when teachers only tolerate qualitative, participatory methods, in other words methods in which their way of seeing things is clearly represented, and resist more external and objective methods of data collection, this could be more than a methodological preference, and be a sign of resistance to critical review).
Since educational evaluation depends very much on the cooperation of people in the situation that is object of evaluation it is particularly vulnerable to distortions and manipulations at this level as well.
Strategies that should be considered for improving political commitment are:
• persuasion, by clearly stating the objectives and clarifying the methods of M&E, also guaranteeing safeguards against possible harmful side-effects, guaranteeing anonymity etc.;
• providing incentives for participation in M&E activities, preferably stimulating intrinsic motivation by exploiting spin-off of M&E to the benefit of actors (for example by making a special effort in feeding back information to schools);
• coercion and close supervision by the government, by making other aspects of reform (like provision of better equipment, teacher training schemes and new curricula) contingent on cooperation with M&E activities.
4.4.2 Institutional capability for M&E
Institutions are “the rules of the game” in a society. They should be distinguished from organizations, which structure “the way the game is played” (Berryman et al., 1997).
Examples of institutions are the legal system, property rights, weights and measures and marriage. But the rules of the game may also be less formal and depend on convention and implicit norms.
In assessing the institutional capability for M&E in a country instances of an
“evaluation culture and tradition” should be looked for.
Indicators of the degree to which a country has a strong or weak evaluation culture that could be considered are:
• whether or not quality and safeguarding quality in education is mentioned in the constitution or other legislation;
• the elaborateness of the system of examinations and certification in education;
• evaluation history (e.g. for how long have educational programs been empirically evaluated, and with what degree of success);
• instances of real use (e.g. evident from referring to evaluation results in public media) vs symbolic use of evaluations;
• participation in international assessment surveys;
• emphasis on accountability in education in public and political debate.
Institutional capability for M&E is most realistically addressed as an assessment activity in order to obtain a notion of the general climate in which M&E activities in a country will “land”. Institutional development in this domain is an endeavor that appears to go beyond the conduct of a particular reform project and should be embedded in a more general and long term country strategy.
4.4.3 Organizational and technical capacity for M&E
Questions about organizational capacity for M&E in a country first of all regard the issue of whether important technological functions have an “organizational home” in the country. For example, initiating a national assessment is the more of a heavy task when there exists no organization that has specialized in the development of educational achievement tests in the country. The same applies when external supervision of schools is considered at a fairly large scale and the country has no educational inspectorate.
Further criteria in determining the organizational capacity concern the wellfunctioning of organizations in terms of effective leadership, ability to mobilize financial, material and human resources and appropriate work practices (Orbach, 1998).
For organizations concerned with educational M&E additional criteria for wellfunctioning are professional standing and a credible degree of impartiality.
Organizational capacity building for M&E should start from a careful analysis of the planned M&E approach and technology and the mix of skills and expertise needed to carry it out successfully. Next, available organizational “homes” should be examined for gaps between the required and available skills and general organizational well- functioning. If no such “homes” are available the creation of new units should be considered. In case of gaps several options are to be considered: narrowing down of the M&E objectives or changing and improving current practices, e.g. by means of training, and provision of additional resources, human resources (e.g. external consultants) in particular.
As far as technical capacity is concerned the required set of skills for successfully carrying out M&E activities depends on the priorities and ambitions of the M&E plan, in the sense of the M&E objectives, general approach and specific methods. Although these are likely to be given most of the attention it should be noted that the required skills do not just pertain to research methodological and technological skills but also to communicative skills and substantive educational knowledge.
Issues of organizational and technical capacity for M&E were documented more specifically in Chapter 3, in referring to the technical and organizational requirements
needed for each of the 15 specific types of M&E that were distinguished in the first chapter).
4.5 Conclusion: Matching Evaluation Approach to Characteristics of