Chapter 5. Testing of the game environment
5.2 Methods of evaluation
5.2.2 Pre- and post- gameplay questionnaires
Identical pre- and post-gameplay questionnaires (Appendix D) based on those that were used in the definition of the learning objectives for the portal were constructed. Only questions pertaining to content areas that were to be included in the testing version of the portal were included in the questionnaire. An additional eleven open-ended questions were included in the questionnaire. These open questions were concerned primarily with the content areas addressed by the puzzles available for testing in the portal environment. Figure 5.1 shows an extract from the questionnaire, showing one closed and one open question format.
8. What is the most prevalent form of HIV transmission in Africa?
a. Mother to baby transmission.
b. Heterosexual (two sexes) sex.
c. Homosexual (same sex) sex.
d. Blood transfusions.
e. Needle sharing amongst individuals.
Confidence
1 2 3 4
14. What do we mean when we talk about the HIV window period? How would you describe it to someone who’d never heard about it?
_________________________________________________________________________________
_________________________________________________________________________________
_________________________________________________________________________________
_________________________________________________________________________________
_______________
Please rate your confidence in your description
1 2 3 4
Figure 5.1 An excerpt from the pre- and post-gameplay questionnaire showing an example of a closed and open-ended question.
Multiple choice questions were included to facilitate a quantitative aspect to the comparison between pre-and post-gameplay scores while open-ended questions were included to increase the richness of the information obtained from learners. These open-ended questions would be used to assess whether, as a result of interaction with the puzzles and the associated social discourse that resulted, learners had acquired a richer understanding of the topic being addressed.
When comparing the pre- and post-gameplay questionnaires, the multiple choice questions were coded as either a value of 1 (correct answer) or 0 (incorrect answer) and recorded in a SPSS (SPSS for Windows, version 11.5) spreadsheet along with the learners confidence in their answer. The number of questions answered correctly in pre- and post-gameplay questionnaires were then compared as well as the confidences with which these questions were answered.
The open questions were analysed differently to make full use of the richness of the information being portrayed by the learner. What was essentially being measured here was the information that learners volunteered and considered as being relevant at the time of answering the question. In addition to their answers, learners were also asked to specify the confidence they had in the answers they had provided. This was done to ascertain how convinced the learners were that the information they had supplied was correct.
Anderson et al. (2001) define ‘content analysis’ as being a set of procedures that identify a target variable, collect representative examples from the text, and devise reliable rules for categorizing segments of the text.
In a study describing online asynchronous discussions Blignaut and Trollip (2003) formulated a way in which to quantify open-ended responses by placing the responses into different categories, based on the topic they were addressing. They took transcripts from online asynchronous discussions and categorized each response according to the content in that response. They reported that having a taxonomic framework within which to categorise information was a very useful tool for understanding some of the dynamics of that information.
In an attempt to understand the dynamics of the information retrieved from respondents in the open-ended questions for both pre- and post-gameplay questionnaires a similar approach to Blignaut and Trollip was adopted. In the context of this study, however, it was deemed more appropriate to categorise the information from the open-ended questions according to the number of relevant concepts presented in each answer, as well as the number of relevant domain-specific terms used. The reason relevant concepts were identified was because these are the base units of the answers provided and these give insight into the learner’s existing knowledge structures concerning that knowledge domain. Relevant terminology was also measured to see if the interaction with the puzzle interface and the content space of the puzzles resulted in the learner utilizing some of the formal language associated with the knowledge domain.
In order to assess whether learners gained a richer understanding of the knowledge domain during gameplay, all of the pre- and post-gameplay answers were deconstructed into their constituent concepts and then each of these concepts were either categorised as being relevant or irrelevant to the question being asked (Table 5.2).
In addition, the level of terminology was assessed by recording the number of relevant terms used in each response. Relevant terminology tended to be more formal language and often medical in nature.
Once all answers had been deconstructed all the relevant constituent concepts were added up and concept scores were assigned to the pre- and post-gameplay answers provided. Similarly pre- and post-gameplay terminology scores were assigned for the number of relevant terms in each respective answer.
After all the open-ended questionnaires had been similarly scored, these scores were compared and graphed using SPSS to elucidate any trends between pre- and post-gameplay answers provided by the learners.
Table 5.2 An example of the deconstruction, categorization and terminology assessment of one of the pre- and post- gameplay answers provided by a learner.
Question 15: If someone asked you to briefly explain what happens when a person becomes infected with HIV/AIDS what would you say? Please include descriptions of the lifecycle and disease progression in your answer.
Time Learner answer Relevant constituent concepts Relevant terminology
Conc score
Term score Pre- The person will start to lose
strength as his immune system starts to fail. The HIV/AIDS starts to eat or kill the antibodies which fight off sickness and the person becomes ill more frequently. Person’s lifespan after contracting HIV/AIDS varies.
1. person will lose strength 2. his immune system will start
to fail
3. HIV/AIDS eats/ kills antibodies
4. antibodies fight off sickness 5. if antibodies die then become
ill more frequently 6. lifespan after HIV/AIDS
varies
1.immune system
2.antibodies 6 2
Post- The person will start to lose strength when his immune system starts to fail. The HIV virus affects the white blood cells by changing their normal mode of operation thus destroying their function and eventually spreading to other white blood cells.
Because the antibodies are dying the immune system fails meaning the person becomes ill more often.
Expected lifespan after infection depends on person infected and medication but it varies.
1. person will lose strength 2. his immune system will start
to fail
3. HIV virus affects white blood cells
4. HIV changes normal operation of white blood cells
5. HIV spreads to other white blood cells
6. HIV kills antibodies 7. if antibodies die, immune
system fails
8. if immune system fails, get sick more often
9. lifespan after HIV infection with HIV depends on person, 10. lifespan after HIV infection
depends on medication 11. lifespan after HIV infection
varies
1. Immune system 2. White blood cells
3. Antibodies 4. medication
11 4