CHAPTER 3: METHODOLOGY
3.4 Materials used
3.4.3 Comprehension tests
Each video clip had to be accompanied by a comprehension test. The videos have comprehension tests which were used in the previous study (Kruger et al., 2013), which the researcher had access to. Since the current study used clips cut from the original videos and not the full videos, the researcher had to carefully consider which questions from the original comprehension tests could be retained. Furthermore, additional comprehension questions had to be developed so that the comprehension tests for the final clips would have sufficient questions.
More questions were developed by using recent examination papers and the study unit outcomes outlined for each video. The original tests together with PSYC 121 examination papers and module outcomes were used to compile tests for the videos used in this study.
The Psychology lecturer was consulted and requested to check the comprehension tests for accuracy and consistency in terms of level of difficulty. His opinion and input as an expert on the module content were important. The lecturer considered the comprehension tests and confirmed that they were appropriate and relevant to test comprehension for the selected clips. After initial approval by the lecturer, the comprehension tests were pilot-tested with a small group of participants at the Potchefstroom campus of the NWU. This group consisted of first-year non- Psychology students, similar to those who would participate in the main study on the Vanderbijlpark Campus of the NWU, apart from the fact that they did not have to be mother- tongue speakers of Sesotho. Students from the Potchefstroom campus were used so as not to deplete the numbers of students on the Vanderbijlpark Campus who qualify to participate and who may want to participate in the main study. Including Potchefstroom campus students helped increase the pool of participants, which contributed to validating the research findings. The
intention of the comprehension pilot was to check if the level of difficulty of the questions was appropriate for the intended participants. Questions with performance lower than 30% correct answers and higher than 90% correct answers would be deemed too difficult or too easy, and would be altered or discarded from the test. However, the timing of the pilot turned out to be problematic – the participants who signed up could not complete viewing and answering the comprehension tests, which means the full pilot of these comprehension tests could not be completed in the intended manner. To make up for this in the limited time available for the researcher to finalise the comprehension tests and to ensure validity of the test items, other means were devised to ensure that the comprehension tests were appropriate.
The researcher sought the assistance of Academic Literacy (ALDE) lecturers who have experience and skill in setting up comprehension tests. A team of the lecturers with combined work experience of more than 40 years were asked to watch the videos and look at the comprehension tests to identify whether they were appropriate or not, and thus helped with selecting the final questions for each test.
At the initial meeting with the team, they indicated that the questions only tested the lower element of comprehension, which is memory and not the ‘understand (comprehension)’ category of Bloom’s Revised Taxonomy. The team made recommendations to assist the researcher to revise the questions in a suitable manner. The revised questions included activities such as participants paraphrasing the information in their own words, classifying items, comparing and contrasting concepts and explaining concepts.
The researcher revised the questions, taking into consideration the concerns of the ALDE team that random guesswork may be used to answer the multiple-choice questions and the true/false questions. Some of these questions were retained as the researcher took into consideration the fact that the participants were exposed to the content once and did not have the opportunity to revisit the content. However, they were adapted to request students to include a reason for the answer selected in order to ensure that not only memory was tested. This does not eliminate guesswork, however, by asking participants to justify their responses allows them to demonstrate an element of understanding which is relevant for the level of first years according to Bloom’s Revised Taxonomy.
Due to time constraints and unavailability of the ALDE team, the revised questions were submitted to one senior ALDE lecturer who approved the final tests and they were finalised. An example taken from the Video 5 comprehension test demonstrates how the true/false questions were revised following the intervention of the ALDE team:
Original question:
Read each statement and decide whether it is true or false.
8) Community psychology focuses on people on an individual basis.
Final question:
2) Indicate whether the following statement is True or False and explain the reason for your answer.
Community psychology focuses on people on an individual basis.
The purpose of the comprehension tests was to examine whether the different types of subtitle conditions influenced the participants’ understanding of the content in the clips. According to Carlisle (1999:11) recall measurement is a common method for assessing comprehension in the classroom. Carlisle (1999:12) further explains that comprehension depends on long-term memory and working memory processing information presented. In order for the comprehension tests in the current study to assess comprehension holistically, they were designed to include the following types of recall: recognition, cued recall, and free recall. These types of recall may include questions described as typical questions for assessing comprehension of a text such as true/false, multiple choice, cloze and open-ended questions (Cain & Oakhill, 2006:701-705).
Recognition recall entails presenting participants with options from which they are to select the correct option. The questions used include true/false and multiple-choice questions. In true/false questions, participants are presented with a sentence and they have to indicate whether the sentence is true or false (Cain & Oakhill, 2006:702). The authors affirm the input of the ALDE lecturers by asserting that these forms of assessment pose low processing demands on the participants, as no complex verbal response is required. They are good for assessing the memory element of comprehension but fall short in terms of inference making, which is another comprehension indicator. Multiple choice is also a form of assessment which does not require a complex verbal response, however, it poses a higher processing demand because a participant has to compare responses in a set of options and choose the correct one (Cain & Oakhill, 2006:703).
Cued recall on the other hand involves providing the participants with a cue to remind them of the answers to the questions asked. Cloze tasks are included in this type of recall as they consist of a sentence where a single word is omitted and the participant has to either fill in the word or select the appropriate word from a choice (Cain & Oakhill, 2006:701).
The last type of questions used in the study falls under free recall, which involves the participant receiving no cues to assist them to remember the answer to the questions. They are open-ended questions such as asking for explanation of concepts or to motivate their responses. Through this form of assessment, the participants’ memory and understanding are tested (Cain & Oakhill, 2006:703-704). These different levels of comprehension are important to test as they reflect the level of understanding appropriate for first-year students, who are advanced learners.
The different comprehension tests for the different videos all counted a total of 10 marks, with each test consisting of different types of questions as outlined in Table 3-7 below.
Table 3-7: Types of questions asked per video
Recognition Cued recall Free recall
Video 1 2 1 2
Video 2 2 1 3
Video 3 1 3 3
Video 4 2 2 3
Video 5 1 3 3
The comprehension tests were presented in one language only, English. The researcher had considered presenting them in Sesotho as well since Sesotho subtitles were used. The decision to present the comprehension tests in English only was based on the reality that students at HEIs are mainly assessed in the LoLT (English) only. At the NWU some assessments are bilingual, with English and Afrikaans, however this combination would not be suitable for the participants in the current study and the introduction of Sesotho tests could have been a source of frustration as they would be unfamiliar to the participants and could be a potential confounding variable. The comprehension tests for all videos can be found in Appendix D.