Basics of Educational Evaluation
3.3 Forms That are Based on Education Statistics and Administrative Data
Controversial points
Perhaps the issue of norm-referenced versus criterion-referenced testing applies to examinations more than to other student assessment forms. Traditionally examinations have been norm-referenced. The main draw-back of this being that norms would differ across years and cohorts.
3.3 Forms That are Based on Education Statistics and Administrative
According to Table 1.1 in Chapter 1 a MIS has a function for accountability as well as for organizational learning and diagnosis.
Technical issues
Educational statistics form the base-material for education indicators. Traditionally education statistics are more readily available in the area of educational inputs, like financial data, number of teachers, and number of schools. Information on the flow of students through the system, participation rates and success rates per age-cohort are also required. Indicators on the stock of teachers and human resources in general are not usually available, but would also be important. Indicators can depend on “single”
statistics or be composites of several statistics, like for example student/teacher ratios or
“school overhead”, measured as the proportion of administrative personnel relative to all staff or all students.
Integrated multi-level indicator systems have additional technological demands attached to them, like data collection on the same or “nested” units, and an integration of statistics with survey-based and/or student assessment data.
Innovative aspects
Thinking in terms of MIS and indicator systems provide an interesting challenge to national statistical bureaus. Innovation is a slow process of gradually covering all main categories of the theoretical model of input-process-output-context indicators with statistics for which basic data is collected at a regular basis. International education indicator projects, like those of the OECD (INES-project, with Education at a Glance as the central, annual publication) and the European Union (the publication of indicators in
“Key Data”) can stimulate these developments at national level.
Technical and organizational capacity required
A MIS requires a bureau for Education Statistics with a specialized unit for developing indicators in the domains where traditional statistics do not fully cover all categories of the theoretical model.
In the case of integrated multi-level indicator systems additional methodological skills are required, encompassing survey and assessment methodology.
Controversial points
The level of detail of an MIS should be matched to the pattern of centralization and decentralization of the educational system in a country. Mismatches could lead to controversy over, for example, “centralistic tendencies” in an otherwise decentralized setting.
3.3.2 School management information systems General description
School management information systems have been inspired by similar systems in private industry. Generally they consist of a careful modeling of information streams and information needs within a company, deciding which data should be available for which purpose on a more or less permanent basis, followed by design and implementation of a computer configuration and software. A school management information system is described as an information system based on one or several computers, consisting of a data-bank and one or several software applications, which enable computer-based data storage, data analysis and data distribution.
A question that could be answered by means of such a school management information system would be: “to which degree has absenteeism decreased after the implementation of specific measures to diminish absenteeism?”
Management information systems have a great potential for supplying important information on a routine basis. At present there are still quite a few practical barriers. One needs to have sufficient and adequate computer hardware and even when professionally developed software packages become available, quite a few specific maintenance functions must be carried out, while new routines and perhaps even new job-functions to guarantee adequate data-entry should be developed.
Main audiences and types of use of the information
Although it would be conceivable to place the use of a school MIS in an accountability context, as for example when a school district or municipality would require specific information yielded, school-based use for purposes of school selfevaluation is more likely. The information could be used for all kind of corrective actions in the running of the school. School-leaders and maybe also the staff-team of the school are the most important user category.
Technical issues
Construction of a computerized school MIS requires considerable development costs.
Development consists of providing functional specifications of information flows, developing adequate software, and development of manuals. The total costs of the development of an MIS for secondary schools in the Netherlands, toward the end of the eighties, were about US$ 500,000. Development up to the stage of a commercially exploitable prototype took about 5 years. Currently there are many commercial products available. Empirical studies indicate that the systems are frequently “underutilized” by schools.
Technical and organizational capacity required
Introduction requires adaptation in the school-management style, adequate communication platforms and getting used to the ICT aspects. Possibly new roles or job- functions need to be implemented.
Controversial points
Implementation problems may tip the cost-benefit balance in the negative direction. The technological image and formalized methodology may not match well with the school culture.
3.4 Forms That are Based on Systematic Review, Observations and