Basics of Educational Evaluation
3.4 Forms That are Based on Systematic Review, Observations and (Self)Perceptions
Technical and organizational capacity required
Introduction requires adaptation in the school-management style, adequate communication platforms and getting used to the ICT aspects. Possibly new roles or job- functions need to be implemented.
Controversial points
Implementation problems may tip the cost-benefit balance in the negative direction. The technological image and formalized methodology may not match well with the school culture.
3.4 Forms That are Based on Systematic Review, Observations and
Technical and organizational capacity required
Countries should be able to put together a good support team that provides the basematerial for the review. Field visits and interviews should be properly planned in order to carry out the visitation as efficient as possible.
Controversial points
Given a systematically conducted review and panel members with strong international reputation governments should be ready to take the results to hart, also in the case of politically unwelcome conclusions.
3.4.2 School inspection/supervision General description
The core activity consists of school visits carried out by the inspectors/supervisors. The range of aspects of school functioning that is reviewed may vary, ranging from a rather formal contact with the school director to classroom observations and talks with students.
In several European countries the work of education inspectorates has become more systematic over the last decades in three ways:
• in the sense of a plan to visit all schools in the country with a specific frequency (e.g.
once every two years);
• in the sense of standard setting;
• in the sense of using more systematic, research-like methods of data-collection, i.e.
systematic observation in schools and classrooms.
Main audiences and types of use of the information
Inspectorates usually have a dual function. On the one hand they are to inform authorities, like central or regional education officers, on the other hand they are usually also seen as a kind of counselors to schools. Emphases between these two functions may differ between countries. For example, in the United Kingdom inspections are predominantly placed in an accountability context, whereas in the Netherlands accountability has less emphasis and school reports have more of a formative function to the schools.
Technical issues
The definition of evaluation criteria and standards is one technical issue, systematic observation by inspectors is another. As to the former specific consensual procedures are used, involving different kind of educational expertise to define key areas of evaluation (criteria) and, next, norms of acceptable performance (standards) on these criteria.
Typically inspectorates will not only consider outcome indicators but they will also look at input and process indicators of school functioning. How to value these processes, when
their explicit association with outcomes is uncertain is a difficult issue in evaluation methodology.
The challenge is to combine systematic procedure and standardization in datacollection with the expert judgement and “clinical look” of school inspectors. Reports may have a quantitative and qualitative component.
Finding a way to inspect all schools in the country with a certain frequency is a technical and practical issue which is of course related to the numerical capacity of the inspectorate and the intensity of school visits.
Technical and organizational capacity required
A school inspectorate, or a structure of school supervisors, which, for example, might be in regional offices or regional educational support centers, is an important institutional and organizational facility in a country’s education system. It requires experienced educators with basic knowledge about systematic data collection methods and evaluation methodology in general.
Controversial points
In situations where building up an inspectorate from scratch is considered, given the high costs, a careful analysis of possible alternatives should be considered (e.g. a combination of a national MIS and stimulation of school self-evaluation). A more loosely organized network of School Panel Inspections exists, for example, in Jamaica (cf. Scheerens, 2002).
3.4.3 School self-evaluations, including teacher appraisal General description
School self-evaluations are internal evaluations of the school as a whole, or of subunits within the school, aimed primarily at school improvement. In fact there is a gliding scale from “completely internal” to extensive use of external capacity in school self-evaluation.
The decisive point being the condition that the school is the initiator and the prime- audience of the evaluation.
There can be several different methodological emphases in school selfevaluation. In previous paragraphs “assessment-based” school self-evaluation and the use of “school management information systems” were discussed.
A third major methodological emphasis is known under headings like schoolbased review or school-diagnosis. School-based review depends heavily on opinions of school personnel on discrepancies between the actual and an ideal state of affairs in schools. In this way a broad perspective, in which all the main aspects of school functioning can be scrutinised, is possible. Usually, respondents are also asked to indicate whether a certain discrepancy should be actively resolved. This approach to school self-evaluation seeks to gear improvement-oriented action to appraisal. The context of application is usually school improvement, which means that a school-based review is carried out when there is a prevailing commitment to educational innovation.
Advantages of this approach are: a broad scope, a user-friendly technology, an explicit linkage between evaluation and action, and a high degree of participation (all school personnel take part in the review). A definite weakness of school-based review is its dependence on subjective opinions and its (usual) neglect of “hard” factual data on school functioning, most notably output data. Examples of procedures for school-based review are the GRID and GILS-systems (see Hopkins, 1987).
A fourth approach seeks to provide schools with self-evaluation instruments (questionnaires) that meet scientific requirements of reliability and validity. In this way the subjectivity in the self-appraisal can be countered; this is also accomplished by using ratings of different categories of respondents on the same phenomena and by comparing the results. This is an example of “triangulation”, a procedure that was developed in qualitative research methodology.
In a fifth approach teams of colleagues from other schools visit schools and do a review of the school’s functioning. This approach of “visitation committees” and peer review can also be a formal part of accountability oriented appraisal.
Finally, several of these forms of school self-evaluation can be combined and integrated with one another. For example assessment information on outcomes, administrative data on student background characteristics and self-reports on the functioning of key processes could be carried out in conjunction, possibly relating the various types of information to one another to obtain more insightful diagnoses.
Teacher appraisal can be a specific emphasis in school self-evaluation. The important issue being that the empirical methods of school self-evaluation would offer a more factual basis to assessments by the school director.
Main audiences and types of use of the information
School management and staff teams are the major audience for the results of school self- evaluation. It is not unusual, however, that (parts) of the results are also presented to other constituencies, like higher administrative levels or stakeholders in the local community. Feedback may also be focused at individual teachers, or subteams, like departments within a school.
Information can be used to redesign school development plans, preferred teaching strategy, grouping of students and targets for professional development of teachers.
Technical issues
All aspects of the science and the art of evaluation of human service programs are also relevant in school self-evaluation. How to combine objectivity and commitment? How to deal with demands for openness to external constituencies and demands for confidentiality and mutual trust at the same time? How to make an efficient choice of the many methodological options? How to deal with resistance to and fear of being assessed?
An additional practical consideration is how to find the time for school self-evaluation. A viable option seems to be an integration of new assessment and monitoring forms with other developments like curriculum redesign and changed teaching strategies.
Technical and organizational capacity required
School teams need to be trained for most types of school self-evaluation. For some forms in particular, external support is required on a more permanent basis (e.g.
testdevelopment, data-analysis, data-feedback).
Carrying out school self-evaluations requires that there are certain communication platforms operational within the school. Support by the school head or director is an important pre-condition.
Controversial points
The key question that remains is the objectivity of self-evaluations. From a very strict methodological position “objective self-evaluation” could even be regarded as a
“contradictio in terminis”. On the other hand objectivity can be supported by providing instrumentation that meets scientific criteria. The degree to which autonomous functioning of schools is a priority in an educational system and “quality care” is decentralized to the school level, is also relevant to this discussion.
3.4.4 School audits General description
As educational institutes (schools and universities) are made to function more autonomously, they may become more like private companies in their managerial and organizational characteristics. An example of this would be a stronger emphasis on strategic planning and on scanning the external environment of the school. It is therefore not surprising that approaches used in management consultancy are introduced in schools. A strong point of these approaches is that it is likely to pay attention to issues that were kept largely unnoticed by the educational province, such as external contacts, anticipation of developments in the relevant environment, and flexibility in offering new types of services.
Screening the organization for quality in its internal and external functioning can be formalized on the basis of quality systems and norms for organizational accreditation like the well-known ISO norms. In this way schools and universities can be formally accredited.
Since this practice is still fairly uncommon, it will not be further developed here.
3.4.5 Monitoring and evaluation as part of teaching
Informal “formative” assessment of students ‘performance and progress has always been part of regular teaching. Students do assignments that are marked, and teaching methods contain progress tests.
Similarly teachers “keep order” and monitor the behavior of students in classrooms.
This aspect of normal teachers’ work should not be overlooked, as it is can be seen as the basis for the application of more formalized forms of assessment and monitoring. To a
degree teacher have always been “reflective practitioners” and evaluating student performance was not invented with the first multiple-choice test.
This point is only made as a reminder that the principle of really-testing and feedback that is at the center of the motivation to enhance and stimulate educational M&E has always been there as an important principle of “good practice” in teaching.
3.5 Program Evaluation and Teacher Evaluation