INTRODUCTION
In this chapter, I describe the research methodology for exploring the implementation of oral assessment. In the first section, I discuss the reasons for adopting a qualitative research approach, offering an overview of action research as a particular orientation to a study of this nature where an iterative framework was needed to study my assessment intervention with the various groups. I then go on to outline the particular settings, the participants (students and assessors), the various tools for studying the assessment intervention, and the actual design of the study. Finally I discuss the ways in which the issues of validity and generalizability apply to my study and how they were addressed in the study
QUALITATIVE OR QUANTITATIVE RESEARCH?
Elliott (1991: 64) believes that "quantitative methods, which are designed to produce aggregated data in depersonalised and decontexualised form, appear to constitute the perfect solution to the 'insider researchers' dilemma". The researcher is able to produce data that would be quantifiable and unbiased, as they generate "public knowledge" which de- personalises the data. Colon et al. (2000: 2) agree that qualitative research emphasizes
"participant observation" whereas quantitative methods rely on the "research instrument through which measurements are made". Weiler (2001: 415) adds that if teachers want a
"deeper understandings of their students and their learning", they will not be able to achieve this through quantitative research, they will need to be "intimately involved" in the process.
Qualitative research would provide this opportunity. As Labuschagne (2003: 1) says,
"qualitative data provide depth and detail through direct quotation and careful description of situations, events, interactions and observed behavious" or what Jones (1997: 3) describes as
"empathetic understanding". Winter (2000: 8) concurs that while "quantitative research limits itself to what can be measured or quantified", qualitative research "attempts to 'pick up the pieces' of the unquantifiable, personal, in depth, descriptive and social aspects of the world". As Denzin and Lincoln (1994: 4) aptly noted, "qualitative researchers stress the
100
CHAPTER FOUR METHODOLOGY
INTRODUCTION
In this chapter, I describe the research methodology for exploring the implementation of oral assessment. In the first section, I discuss the reasons for adopting a qualitative research approach, offering an overview of action research as a particular orientation to a study of this nature where an iterative framework was needed to study my assessment intervention with the various groups. I then go on to outline the particular settings, the participants (students and assessors). the various tools for studying the assessment intervention, and the actual design of the study. Finally I discuss the ways in which the issues of validity and generalizability apply to my study and how they were addressed in the study
QUALITATIVE OR QUANTITATIVE RESEARCH?
Elliott (199J; 64) believes that "quantitative methods, which are designed to produce aggregated data In depersonalised and decontexualised form, appear to constitute the perfect solution to the 'insider researchers' dilemma", The researcher is able to produce data that would be quantifiable and unbiased, as they generate "public knowledge" which de- personalises the data, Colonet al. (2000: 2) agree that qualitative research emphasizes
"participant observation" whereas quantitative methods rely on the "research instrument through which measurements are made", Weiler (2001: 415) adds that ifteachers want a
"deeper understandings of their students and their leamingH, they will not be able to achieve this through quantitative research, they will need to be "intimately involved" in the process.
Qualitative research would provide this opportunity. As Labuschagne (2003: I) says,
"qualitative data provide depth and detail through direct quotation and careful description of situations, events, interactions and observed behavious" or what lanes (1997: 3) describes as
"empathetic understanding", Winter (2000: 8) concurs that while "quantitative research limits itself to what can be measured or quantified", qualitative research "attempts to 'pick up the pieces' of the unquantifiable, personal, in depth, descriptive and social aspects of the world", As Denzin and Lincoln (1994: 4) aptly noted, "qualitative researchers stress the
socially constructed nature of reality, the intimate relationship between the researcher and what is studied, and the situational constraints that shape inquiry" (see also Denzin and Lincoln 1998). The nature of inquiry in my research suited a qualitative approach so that the richness of the data gathered could be fully explored. A quantitative approach would not have yielded the desired results, as Winter (2000: 9) cautions, "what is certain is that qualitative research sets itself up for failure when it attempts to follow the established procedures of quantitative research such as experimentation, efforts of replication, use of control groups, use of standardised formulas or the use of pre-test/post-test method".
Action Research within Qualitative Studies
Action research is an approach to improving education through change, by encouraging teachers to be aware of their own practice, to be critical of that practice, and to be prepared to change it. It is participatory, in that it involves the teacher in his own enquiry, and collaborative, in that it involves other people as part of a shared inquiry. It is research with, rather than research without (McNiff 1988: 4)
As my research endeavoured to devise a 'new' form of assessment for the South African context, it required techniques and tools that would provide for exploration and that would allow for modification along the way. I would need to determine its structure, format and administration and train assessors to conduct them effectively. Only after the assessment system was devised and trialled, would the shortcomings become apparent. These shortcomings would then need to be addressed and the assessment re-trialled until a good
"fit" (to borrow von Glasersfeld's 1987: 5 term) for all the participants is acquired. Like Joyce Nyof-Young (2000: 472), I too was impressed with the literature on action research, particularly with the use of terms and phrases such as 'shared leadership', 'collaboration', 'empowerment' and 'transformation'. Elliott (1991) and Weiler (2001) believe that the ideals of action research are reflected in constructivist and critical thinking as teaching, learning and research are all part of the same process. After careful consideration, I decided a tool that would allow me to follow such a cyclical method of trial/re-trial is an action research approach.
socially constructed nature of reality, the intimate relationship between the researcher and what is studied, and the situational constraints that shape inquiry"(see also Denzin and Lincoln 1998). The nature of inquiry in my research suited a qualitative approach so that the richness ofthe data gathered could be fully explored. A quantitative approach would not have yielded the desired results, as Winter (2000: 9) cautions, "what is certain is that qualitative research sets itself up for fai Jure when it attempts to follow the established procedures of quantitative research such as experimentation, efforts ofrepJication, use of control groups, use of standardised formulas or the use ofpre-test/posHest method".
Action Research within Qualitative Studies
Action research is an approach to improving education through change, by encouraging teachers to be aware of their own practice, to be critical of that practice, and to be prepared to change it. It is parlicipatory, in thatit involves the teacher in his own enquiry, and collaborative, in that it involves other people as part of a shared inquiry. It is research with, rather than research without (McNiff 1988: 4)
As my research endeavoured to devise a 'new' form of assessment for the South African context, it required techniques and tools that would provide for exploration and that would allow for modification along the way. I would need to determine its structure, format and administration and train assessors to conduct them effectively. Only after the assessment system was devised and trial led, would the shortcomings become apparent. These shortcomings would then need to be addressed and the assessment re-trialled until a good
"fit" (to borrow von Glasersfeld's 1987: 5 term) for all the participants is acquired. Like Joyce Nyof-Young (2000: 472), 1 too was impressed with the literature on action research, particularly with the use of terms and phrases such as 'shared leadership'. 'collaboration', 'empowerment' and 'transformation'. Elliott (1991) and Weiler (2001) believe that the ideals of action research are reflected in constructivist and critical thinking as teaching, learning and research are all part of the same process. After careful consideration, 1 decided a tool that would allow me to follow such a cyclical method of trial/re-trial is an action research approach.
Action research has been an important part of educational theory and practice since the 1940's (Corey 1953, Lewin 1946, Noffke 1992) and in recent years, action research in a variety of forms has been providing teachers at all levels of expertise with opportunities for personal change, improved practice and professional transformation (Noffke 1997, Noffke and Stevenson 1995, Oja and Smulyan 1989, Wells 1994). Definitions of action research are problematic. Carr (1989: 85) stated that "action research now means different things to different people and, as a result, the action research movement often appears to be held together by little more than a common contempt for academic theorising and a general disenchantment with mainstream research'". Habermas's (1972), Carr and Kemmis's (1986) and Elliot's (1991) representation of action research into three clearly distinct types, namely, 'technical', 'practical' and 'emancipatory' was critiqued by Gibson (1985), Lewis (1987), Whitehead (1985) and Whitehead and Lomax (1987) who located the core of the process very firmly with the individual.
A plethora of models of action research evolved in the 1990's adding to the difficulty of precisely defining the action research process. Kemmis' (1990) model incorporated reconnaissance, planning, first action step, monitoring, reflecting, rethinking and evaluation.
In 1952, Lewin's 'spiraling' cyclical process model included planning, execution and reconnaissance. Sagor's (1992) model described a sequential, five-step process that included problem formulation, data collection, data analysis, reporting of results and action planning.
In 1994, Calhoun's model described selecting an area or problem of collective interest, collecting data, organizing data, analyzing and interpreting data and taking action. Wells' (1994) model involved interpreting, planning change, acting and "the practitioner's personal theory" (1994: 27) that informs and is informed by the cycle. Stringer's (1996) model dealt with looking, thinking and acting as a "continually recycling set of activities" (1996: 17).
Although these action research models share some common elements, Jennings and Graham (1996: 276) felt that "a static definition (for action research) is neither feasible nor appropriate in a postmodern world". In an attempt to find a way out of the maze of information, Hollingsworth et al. (1997: 312) suggested that "action research has 'multiple' meanings and uses". They went on to recommend that "what we need to look for is not whose version of action research is the correct one, but rather, what it is that needs to be done, and how action research can further those aims (1997: 312)".
102
Action research has been an important part of educational theory and practice since the 1940's (Corey 1953, Lewin 1946, Noffke 1992) and in recent years, action research in a variety of forms has been providing teachers at all levels of expertise with opportunities for personal change, improved practice and professional transformation (Noffke 1997, Noffke and Stevenson 1995, Oja and Smulyan 1989, Wells 1994). Definitions of action research are problematic. Carr (1989: 85) stated that "action research now means different things to different people and, as a result, the action research movement often appears to be held together by little more than a common contempt for academic theorising and a general disenchantment with mainstream research'''. Habermas's (1972), Carr and Kemmis's (1986) and Elliot's (1991) representation of action research into three clearly distinct types, namely, 'technical', 'practical' and 'emancipatory' was critiqued by Gibson (J985), Lewis (1987), Whitehead (1985) and Whitehead and Lomax (1987) who located the core of the process very firmly with the individual.
A plethora of models of action research evolved in the 1990's adding to the difficulty of precisely defining the action research process. Kemmis' (1990) model incorporated reconnaissance, planning, first action step, monitoring, reflecting, rethinking and evaluation.
In 1952, Lewin's 'spiraling' cyclical process model included planning, execution and reconnaissance. Sager's (1992) model described a sequential, five4step process that included problem fonnulation, data collection. data analysis, reporting of results and action planning.
In 1994, Calhoun' s model described selecting an area or problem of collective interest, collecting data, organizing data, analyzing and interpreting data and taking action. Wells' (1994) model involved interpreting, planning change, acting and "the practitioner's personal theory" (1994: 27) that informs and is informed by the cycle. Stringer's (1996) model dealt with looking, thinking and acting as a "continually recycling set of activities" (1996: 17).
Although these action research models share some common elements, Jennings and Graham (1996: 276) felt that "a static definition (for action research) is neither feasible nor appropriate in a postmodem world". In an attempt to find a way out of the maze of information, HoJIingsworth et al. (1997: 312) suggested that "action research has 'multiple' meanings and uses". They went on to recommend that "what we need to look for is not whose version of action research is the correct one, but rather, what it is that needs to be done, and how action research can further those aims (1997: 312)".
Mills (2000: 5) provides a useful and informative comparison between traditional and action research which briefly describes some of my reasons for choosing the action research approach. Although mention is made below (on page 103), and indeed by many other practitioners and promoters of action research, that action research is conducted by teacher researchers, and principals on "children in their care", my study did not involve my own students. I enlisted the help of lecturers and assessors at two tertiary institutions to enable me to conduct my research. Thus, my research does not fit into the "reflective teacher model"
wherein critical inquiry, based on experience, followed by reflection, is used to continually improve the practice of teaching (McMahon 1999: 165, see also Ashcroft and Foreman-Peck 1994: 3-9).
Who?
Where?
How?
Why?
Traditional Research
Conducted by university professors, scholars and graduate students on experimental groups
In environments where variables can be controlled Using quantitative methods to show to some predetermined degree of statistical significance, a cause-effect relationship between variables
To report and publish conclusions that can be generalized to larger populations
Action Research
Conducted by teachers and principals on children in their care
In schools and classrooms
Using qualitative methods to describe what's happening and to understand the effects of some educational intervention
To take action and effect positive educational change in the specific school environment that was studied (Taken from Mills 2000: 5)
The two main theories of action research according to Mills (2000: 7) are "critical (or theory- based) action research and practical action research". Practical action research "assumes to some degree, that individual teachers or teams of teachers are autonomous and can determine
103
Mills (2000: 5) provides a useful and informative comparison between traditional and action research which briefly describes some of my reasons for choosing the action research approach. Although mention is made below (on page 103), and indeed by many other practitioners and promoters of action research, that action research is conducted by teacher researchers, and principals on "children in their care", my study did not involve my own students. I enlisted the help of lecturers and assessors at two tertiary institutions to enable me to conduct my research. Thus, my research does not fit into the "reflective teacher model"
wherein critical inquiry, based on experience, followed by reflection, is used to continually improve thepractice of teaching (McMahon 1999: 165,see also Ashcroft and Foreman-Peck
] 994: 3-9).
Traditional Research Action Research
Who? Conducted by university Conducted by teachers and
professors, scholars and principals on children in their graduate students on care
experimental groups
Where? In environments where
In
schools and classroomsvariables can be controlled
How? Using quantitati ve methods Using qualitative methods to
to show to some describe what's happening predetermined degree of and to understand the effects statistical signi ficance, a of some educational cause-effect relationship intervention
between variables
Why? To report and publish To take action and effect
conclusions that can be positive educational change generalized to larger m the specific school populations environment that was studied (Taken from Mills 2000: 5)
The two main theories of action research according to Mills (2000: 7) are "critical (or theory- based) action research and practical action research". Practical action research "assumes to some degree, that individual teachers or teams of teachers are autonomous and can determine
103
the nature of the investigation to be undertaken" (Mills 2000: 9). This was not the case in my research as I acted as facilitator guiding the research and the whole process of inquiry. The assessors in my study "are committed to continued professional development" (Mills
2000: 9) hence their willingness to participate in my study. It was hoped that using a combination of the practical action and the critical action approaches would "create opportunities for all involved to improve the lives of children" (Mills 2000: 11). What follows is an attempt to classify my research using Mills' (2000: 9) components or key
concepts of a critical action research perspective:
Action research is participatory and democratic. Based on my experiences as a second language teacher both at secondary and tertiary level, and my experiences as a lecturer at a technikon, I identified the area of assessment in science at tertiary level in South Africa as the focus of my study. Based on data gathered from students, lecturers, assessors, my own experiences, literature searches and current debates surrounding assessment in South Africa, I decided to investigate the impact of my intervention to determine if it would make a difference.
Action research is socially responsive and takes place in context. I was concerned that students (both first and second language speakers of English) are not given adequate opportunity to express themselves fully in the present written assessment situations. I was concerned also about the effect that this has on their results. I therefore decided to explore other strategies and to implement them.
Action research helps teacher researchers examine the everyday, taken-for-granted ways in which they carry out professional practice. I adapted the oral assessment technique and decided to monitor its impact on student performance and students' and assessors' attitudes towards the methods employed.
Knowledge gained through action research can liberate students, teachers and administrators and enhance learning, teaching and policy making. It was hoped that my intervention into assessment in science would liberate all the participants from the constraints of the present system of assessment.
the nature of the investigation to be undertaken" (Mills 2000: 9). This was not the case in my research as I acted as facilitator guiding the research and the whole process of inquiry. The assessors in my study "are committed to continued professional development" (Mills
2000: 9) hence their willingness to participate in my study. It was hoped that using a corn bination of the practical action and the critical action approaches would "create opportunities for all involved to improve the lives of children" (Mills 2000: 11). What follows is an attempt to classify my research using Mills' (2000: 9) components or key concepts of a critical action research perspect ive:
Action research is participatory and democratic. Based on my experiences as a second language teacher both at secondary and tertiary level, and my experiences as a lecturer at a technikon, I identified the area of assessment in science at tertiary level in South Africa as the focus of
my
study. Based on data gathered from students, lecturers, assessors, my own experiences, literature searches and current debates surrounding assessment in South Africa, I decided to investigate the impact of my intervention to determine if it would make a difference.Action research is socially responsive and takes place in context. I was concerned that students (both first and second language speakers of English) are not given adequate opportunity to express themselves fully in the present written assessment situations. I was concerned also about the effect that this has on their results. I therefore decided to explore other strategies and to implement them.
Action research helps teacher researchers examine the everyday, taken-Jor-granted ways in which they carry out professional practice. I adapted the oral assessment technique and decided to monitor its impact on student performance and students' and assessors' attitudes towards the methods employed.
Knowledge gained through action research cem liberate students, teachers and administrators and enhance learning, leaching and policy making. It was hoped that my intervention into assessment in science would liberate all the participants from the constraints of the present system of assessment.