The methods section describes the overall evaluation design and data collection strategies employed in this study, which are based on the Eclectic-Mixed Methods-Pragmatic Paradigm and use a number of research instruments, including anecdotal records, user questionnaires, user interviews, user focus groups, usability observations and online data collection. These methods support the development research paradigm because of their flexible use, particularly with technology, while allowing future educational developments. The research employed both qualitative and quantitative methodologies. Each represented a fundamentally different inquiry paradigm. Qualitative research uses a naturalistic approach that seeks to understand phenomena in context-specific settings and quantitative research uses experimental methods and
quantitative measures to test hypothetical generalizations (Hoepfl, 1997). In order to conduct or evaluate qualitative research, it is important to know the nature of these sometimes hidden assumptions (Myers, 1997). Strauss and Corbin (1990) broadly define qualitative research as,
"any kind of research that produces findings not arrived at by means of statistical procedures or other means of quantification".
5.7.1. Overview of Methods and Analytic Techniques Used
Methods used in this study include case studies, observations, questionnaires, interviews, and paper-and-pencil test as explained below.
5.7.1.1. Case Studies
The case study research method can be defined as an empirical study that uses multiple sources (evidence) to investigate a contemporary phenomenon in a real-world context (Yin, 1984) and emphasise detailed contextual analysis of a limited number of events or conditions and their relationships (Soy, 1997). The weaknesses of case study methods, according to Mikkelson
58
(1995), are that sometimes they serve as a foundation for generalisations, which might be erroneous. In this study, three schools (Buhlebemfundio, Qhakaza and Tholokuhle) and two universities (University of KwaZulu-Natal and University of Zululand) from the KwaZulu- Natal province formed part of the case study. Learners participated in game play where rich data were collected using direct observations, focus group interviews, questionnaires, examinations of records, and paper-and-pencil tests.
5.7.1.2. Observations
Observation on learners engaging with interactive learning system at various stages of its development can be a valuable, if somewhat humbling experience, and time consuming exercise (Reeves and Hedberg, 2003). Direct observation method is important for collecting information on non-verbal behavioural aspects of people. Here, because some students had no experience with computers, observations were made to determine the ease of learning, the speed of user task performance and subjective user satisfaction (Shneiderman, 1987). Observations also included use of the game environment and interaction between players during game play.
Learners from all participating schools were observed to discover the way in which they dealt with the questions, how they reached their conclusions and the way in which they played either Zadarh and yKhozi. One problem with the direct observation method is that the presence of an observer may affect the behaviour of the participants causing them to alter their normal work behaviour (Giacoppo, 2001). In order to avoid the uneasiness caused by being aware that they were being observed, the researcher gave learners many breaks during which notes from observations were made.
5.7.1.3. Pilot Tests
The purpose of the pilot test instrument is to determine whether the survey instrument is effective for use with the people who are potential respondents, that is, to ascertain if there are any survey items that are confusing, ambiguous, or phrased in language unfamiliar to the intended audience (Thompson and McClintock, 2005). Thus, the purpose of pilot testing is to catch potential problems before they become costly mistakes (EvaluationWiki, 2006). Pilot tests were conducted during the development stages of the study questionnaires in order to iron out discrepancies, be they factual or grammatical, and to provide clarity where necessary. For an instrument testing literacy and communication of learners, two pilot tests were performed. In the first pilot test, Digital Media Masters and Honours students took part and their suggestions were incorporated in the second revised questionnaire. In the second pilot test, only first year UKZN
59
students participated and their results were included in the final questionnaire used in the study (Annex 1).
5.7.1.4. Questionnaires
Questionnaires are one of the most frequently used methods of collecting effectiveness data and they can be composed of items that address information and attitudes (Reeves and Hedberg, 2003). According to SIECUS (2004) there are drawbacks to using this method as it (i) limits opportunities to probe or provide clarification; (ii) relies on participants' ability to recall behaviour and/or events; (iii) lacks the capability to measure different kinds of outcomes; and (iv) is difficult to use with low-literacy groups. Another weakness of questionnaires is that because of their structured nature they allow little flexibility to the respondent with respect to response format (StatPac Inc., 2000). Also participants might misinterpret questions, and therefore provide wrong information leading to erroneous conclusion (Mikkelson, 1995). The aim of the two sets of questionnaires was to ascertain the real impact of games on learning and to determine the kind of games that could enhance learning skills. In this endeavour, both qualitative and quantitative data were collected. The participating learners from
Buhlebemfundo, Qhakaza and Tholokuhle schools and UKZN and UniZulu answered POM questionnaires which covered both qualitative and quantitative data. The qualitative and quantitative data were used to assess the socio-economic status of participants who came from township, urban and semi-urban areas. Quantitative data collected were utilised in measuring participants’ literacy and communication skills levels.
5.7.1.5. Focus Groups Interviews
Within the instructional technology field, the focus group method is useful in terms of getting users’ reactions to an interface as they use it over time (Reeves et al., 2003). However, it has the drawback of only collecting information about what users say they do, and not what they actually do. This can lead to less participation from those intimidated respondents. The method is quicker and cheaper than interviewing the same number of individuals and again a greater pool of expertise is tapped than in individual interviews (Mithcell, 2003. Mikklesen (1995), however, says that focus groups interview methods may have their own problems particularly as participants may be overwhelmed by the responses of their peers or where a few people
dominate the discussions. The focus group interview protocol was designed to gather descriptions and details about γKhozi and its impact from the interviewees. A small group of Buhlebemfundo learners was brought together and a structured interview with preset questions
60
was carried out. The focus group interview was intended to evaluate γKhozi by ascertaining learners’ opinions about it and to find out if they think it is a good teaching tool (Annex 4).
5.7.1.6. Paper-and-pencil
Reeves and Hedberg (2003) say, “Paper-and-pencil tests using multiple choice test items are pervasive as evaluation methods, but the reliability and validity of these instruments are often suspect” and therefore, they caution against the use of tests unless there is evidence of their reliability and effectiveness. SIECUS (2004) raises similar drawbacks to those applying to questionnaires for this method, such as; Limited opportunity to probe or provide clarification;
limited question length and breadth; limited capability to measure different kinds of outcomes;
and difficult with low-literacy groups. However, according to SIECUS (2004), paper and pencil methods are: Relatively inexpensive and are a quick way of collecting large amounts of data from large samples in short amounts of time; convenient for respondents as they can complete them in their own time; and offer anonymity which can result in more honest responses. The participating learners from Buhlebemfundo, Qhakaza and Tholokuhle schools and UKZN and UniZulu answered paper-and-pencil test that tried to establish their literacy and communication skills levels. Again, learners from Qhakaza and Buhlebemfundo and UniZulu answered a paper-and-pencil test which covered the topics of photosynthesis and respiration. In this study, paper-and-pencil tests were used to collect quantitative data, which were used to determine the level of learners’ literacy and communication skills.
5.7.1.7. Student Examination Records
This method interrogates the previous academic records of learners. According to Educator's Guide (1998) student records are valuable as such information is easy to obtatin and can be interpreted objectively. However, the other side of this is that these records may not correspond exactly with what the researcher wants; may be incomplete or require additional interpretation;
or one may need special permission to use them. Grade 11 examination records for Buhlebemfundo’s 2003 and 2004 learners were compared to determine if there was any
significant difference between academic skills of the two groups that participated in the research project. This was done to avoid a problem of comparing learners of unequal competences, as this would have compromised the results.
61 5.7.1.8. Games Play
Playing games is one method used for collecting qualitative and quantitative data. Becta (2004) contends, “A striking feature of games software is its power to motivate”. Becta continues to state that it is in provoking and harnessing emotions such as satisfaction, desire, anger, absorption, interest, excitement, enjoyment, and pride in achievement, within the player and their consequences that games software might benefit education. The drawback to games according to Becta (2004) is that they are not universally well designed and many are gender oriented. It is the goal of this research to ascertain if playing Zadarh and yKhozi will provide learners with the motivation to play and improve their literacy and communication skills whilst learning subjects covered by the game.