EVALUATION
3. RESEARCH DESIGN AND METHODOLOGY
3.4 METHODOLOGICAL APPROACH
Policy analysis is a form of enquiry which provides either the informational base upon which policy is constructed, or the critical examination of existing policies. The former is termed analysis for policy while the latter is called analysis of policy and is
concerned with:
• analysis of policy determination and its effects, which examines the inputs and transformational processes operating upon the constructs of such policies as well as the impact and consequences of such policies,
• analysis of policy content, which examines the values, assumptions and ideologies underpinning the policy process. (Taylor et al. 1997:36)
Adopting the latter approach, the thrust of this study is the analysis of policy with regard to the performance management of educators as stated in the policy on teacher
evaluation, the Integrated Quality Management System for South Africa Schools (IQMS).
The research focuses on how the IQMS is managed at the school level and questions formulated critically determine the IQMS’ potentiality to evaluate and enhance educator performance. The main objective was to determine whether the IQMS is being complied with perfunctorily as an ‘accountability tool’ or seen as a useful professional
development strategy and has been accepted as such by educators.
Interviews with Department officials and educator unions are used to elicit views on the IQMS. It has been decided to use semi-structured interviews so that there is a clear focus yet the interviewee will not feel restricted in responding to questions. The research combines both quantitative and qualitative approaches to social research. Quantitative responses will relate to the mechanics of the operation of the IQMS and solicit views of principals and educators on their perception of the IQMS. Open-ended questions designed to encourage qualitative responses are also used. Issues raised in the critical questions; namely, the assumptions underpinning the IQMS, the emphasis on
accountability and teacher competence, and professional development and school effectiveness and improvement are also raised.
The researcher will briefly address the following questions:-
• What are quantitative methods?
• What are the ingredients of qualitative methods?
In simple terms, in quantitative methods of research, we use numbers and in qualitative methods of research, we use words to describe outcomes.
Quantitative research methods were originally developed in the natural sciences to study natural phenomena. However, quantitative methods are now well accepted in the social sciences and education. As part of the research, the researcher looks for certain characteristics (variables) and shows something interesting about how they are distributed within a certain population. A variable needs to be measured for quantitative analysis (Berry 2005: 3). Data on various variables in this study are collected through a
questionnaire. Education research has moved away from the numbers approach in recent years. However, the use of numbers can be a very useful tool. With the use of
sophisticated software packages such as SPSS it is relatively easy to deal with the computation side of things and it is possible to come up with numerous tables and charts almost instantly once ones data is installed.
Qualitative research methods were developed in the social sciences to enable
researchers to study social and cultural phenomenon. Qualitative researchers argue that research must be conducted in the setting where all the contextual variables are operating.
Qualitative researchers are interested in how people make sense of their world. The qualitative researcher is the primary instrument for data collection and analysis. Data is mediated through this human instrument, rather than through inventories, questionnaires or machines. The process of qualitative research is inductive in that:
The researcher builds abstractions, concepts, hypotheses and theories from details. Qualitative researchers argue that the ‘objective’ scientist, by coding and standardizing, may
destroy valuable data while imposing his/her world on the subjects.
http://www.statsoft.com/textbook/stathome.html
Mixed Methods Research is used in this study to combine the two methods of research.
The researcher decided that a combination of methods is appropriate for this study as it may help to ‘ask and answer differently conceived or separate questions; answer
questions about connecting parts, segments or layers of a social whole and provide for a close-up illustration of a bigger picture’ (Mason 2006). Mixing methods also help to achieve accurate measurement through triangulation. Triangulation is defined in the following way:
It is the application and combination of several research methodologies in the study of the same phenomenon. It can be
employed in both (validation and qualitative(inquiry)studies. It helps to overcome the biases, weaknesses and problems that come from single method, single-observer, single-theory studies.
http://www.geocites.com/zulkardi3.html
The questionnaires used in this study are primarily a quantitative instrument but it
includes a qualitative element in the use of open-ended questions. The interviews draw on qualitative methods. Brannen (2005: 8) suggests that mixed methods of research be considered in ‘the context of justification’, that is, during the analysis and interpretation of data and this suggestion has been adopted in the analysis of the data in this study.
To obtain data on the issues related to the critical questions formulated in chapter one
which centre around the opposing issues of accountability and professional teacher development, the researcher chose to use both the qualitative and the quantitative paradigms in order to investigate and address the research questions. Quantitative research is used when attempts are made to collect data that can be presented in the form of numbers and which represents some measurement or the other. This approach is generally used when breadth is required. Qualitative data on the other hand, ‘allows the researcher to work with the raw data, to explore the nature of the stories people tell or the way they behave, to look at the different perspectives, understandings and interpretations of social situations in which they participate’(Ross in Carter and Cathryn 1997:39). This approach is used when depth is required. Furthermore, these two approaches are not mutually exclusive and can be combined. By combining these two methods a form of triangulation results which enhances the validity of one’s study (Remenyi et al 1998:192).
Triangulation is a powerful means of demonstrating concurrent validity, particularly in quantitative research. Validity is defined as the best available
approximation to the truth or falsity of a given inference, proposition or conclusion. The real difference between validity and reliability (both are necessary in research) is mostly a matter of definition. Reliability estimates the consistency of measurement while validity refers to the accuracy of measurement (Mason 2006: 12). It has been observed that research methods acts as a filter through which the environment is selectively experienced. They are never atheoretical or neutral in representing the world of experience. Exclusive reliance on one method or approach may bias or distort the researcher’s perspective on what she is investigating.
Creswell (1994:175) advances the following additional reasons for combining methods in a single study:
• Complimentary, in that overlapping and different facets of a phenomenon may emerge (e.g. Peeling the layers of an onion);
• Developmentally, wherein the first method is used sequentially to help inform the second method;
• Initiation, wherein contradictions and fresh perspectives emerge; and
• Expansion, wherein mixed methods add scope and breadth to a study.
The rationale for using a strategy of combining two or more methods is well documented in the literature on research methods as the flaws of one method are often the strengths of the other so deficiencies in a particular method may be overcome. Moreover, a study using both methods is ‘fuller and more comprehensive’ (Neuman 2000: 125).
Thus in keeping with this reasoning, the researcher has decided to use a multi-method approach to this study. The researcher needed to investigate whether the use of the IQMS in schools has been structured around the broad and desirable aims of personnel
improvement and staff development which would offer much to the present South African teaching profession or is there a ‘hidden agenda’ of using the IQMS as a tool for accountability to check on teacher performance and improve school efficiency. To do a fine policy analysis of the IQMS policy and for purposes of data collection, this study has combined the following methods:
• Questionnaires (quantitative and qualitative)
• Interviews (qualitative)
• Document study (qualitative)
Comparing the respondents’ views in the questionnaire and interviews with the document study, especially the IQMS Manual (2005), will hopefully reveal whether the demands of this new performance management system14 exceeds the performance skills of the
majority of educators in South Africa. If this is found to be the case then obviously there would be a greater need to develop educator effectiveness, improve teacher competence and promote professional development than to judge the quality of an educator’s work performance.