CHAPTER 3: METHODOLOGY
3.3 Participants
3.3.1 Participant sampling
In this study, two non-probability sampling methods, which include purposive and quota sampling were used. Students eligible for participation were Sesotho First Language (L1) speakers who completed Sesotho as First Language or First Additional Language in Grade 12. Sesotho as a school subject was a requirement because some of the material in the study was presented in Sesotho, and therefore the participants needed to have developed language abilities in Sesotho in order to engage with the materials in this study. Inability to do so would skew the findings for the study.
Another inclusion criterion was the field of study. The materials used in this study were from the field of Psychology. In order to avoid prior knowledge influencing the performance in the comprehension tests and therefore skewing the findings of the study, participants could be from any field of study except Psychology. Students who are or were registered for a Psychology module in their current or previous studies were therefore excluded from participating.
The last inclusion criterion was the year of study. Participants had to be first-year students only, since the materials used in the study are intended for first-year students.
3.3.1.1 Recruitment of participants
A total of 38 participants were recruited through advertisements distributed on campus (See Appendix A) and brief presentations in classes. The researcher received permission to recruit from Academic Literacy (ALDE) and Communication (ICOM) classes at NWU, Vanderbijlpark Campus. These classes were identified as they had high numbers of students. The researcher targeted the ALDE education and IT groups as they provided the researcher a high likelihood of reaching students who are not enrolled for Psychology modules.
According to the regulations set forth by the Ethics Committee for Language Matters (Humanities), the researcher was not allowed to recruit participants herself. Instead, third- party recruiters assisted and the process for recruitment was planned as follows: the researcher would go to the identified classes and give a brief overview/explanation of the study, after which the third party recruiter would invite the students to participate. The physical recruitment in classes did not go as planned as the third party recruiters were not available to go to every class due to their own work and study commitments. To compensate for this a YouTube video was created with the researcher giving a brief description of the project. The
video ends with a third party recruiter inviting participants to sign up (See Appendix B). This video was posted on the ALDE and ICOM eFundi sites. The researcher also made arrangements to visit some classes to play the video for the students.
In the video and the recruitment letter, the potential participants were informed that the study was about the use of audiovisual material in education with the aim of determining how these materials contribute to learning. In the video, the finer details regarding the nature of the study were kept minimal in order to “avoid conscious reactivity of participants that would make these data worthless” as warned by Bröder (1998:806). For example, the term “subtitles” was omitted in the description and title of the project because it could bias participants’ viewing behaviour by drawing their attention to the subtitle area instead of allowing for natural/preferential viewing and reading; instead, the term “audiovisual material” was used, which includes videos and other materials without being overly specific. The researcher defended this approach in the ethics application, and the Ethics Committee for Language Matters (Humanities) was satisfied that it was necessary to conceal some detail in the general description.
3.3.2 Academic literacy as a grouping variable
The 38 participants were randomly allocated to five groups. The groups had unequal number of participants as demonstrated in Table 3-1. The inequality is a result of problems that were encountered while the experiments were being conducted such as participants dropping out of the experiments.
All the groups were exposed to the same experimental videos, each in a different subtitle condition. The videos are recordings of lectures presented in a module at university. The fact that the videos consist of academic content meant there was a possibility that the results of the study could be influenced by the academic literacy levels of the participants instead of the video conditions. The researcher therefore had to make provision for the issue of the participants’ academic literacy levels. The researcher aimed to have a representative sample in terms of academic literacy levels in the groups, i.e., the groups were to include students with abilities in the lower levels, intermediate levels and higher levels. Academic literacy (specifically the results from the TALL (Test of Academic Literacy Levels) was thus used as a grouping variable in order to achieve a representative sample so as to be able to generalise the results to the sample and to attribute the results to the video conditions. The number of participants on each of these levels is demonstrated in the Table 3-1.
Table 3-1: The number of participants per level included in each group
Participant Group Academic literacy level Number of participants
A (7 participants) Lower 3
Intermediate 2
Higher 2
B (9 participants) Lower 5
Intermediate 3
Higher 1
C (7 participants) Lower 3
Intermediate 2
Higher 2
D (7 participants) Lower 4
Intermediate 3
Higher 0
E (8 participants) Lower 4
Intermediate 2
Higher 0
TALL scores are not available for participants who did not take the test.
2
3.3.2.1 Academic literacy defined
Academic literacy should not be mistaken for language proficiency; it involves much more as demonstrated by Weideman’s definition. According to Weideman (2007:x-xi), academic literacy includes the ability to understand academic vocabulary, understand and make meaning of content, interpret meaning of content, make inferences from and apply information contained in academic texts. Participants in the study were expected to engage with the content presented in the videos and demonstrate understanding, which was tested through comprehension tests. The academic literacy test results were important for this study as they were the only available indicators of the participants’ academic ability. The results enabled the
researcher to ensure that the experiment groups were balanced in terms of the academic literacy levels of the participants.
3.3.2.2 How is academic literacy tested?
Students come to higher education institutions with different levels of academic literacy. Higher education institutions have had to develop tests to assess the academic literacy of students (Van Dyk & Weideman, 2004:14). These tests are considered to be indicators of the students’
ability to access and interact with academic material. One such test that is locally developed for use by multilingual institutions, is the Test for Academic Literacy Levels (TALL) (Le, Du Plessis & Weideman, 2011:116). According to the authors, the TALL is used for student placement and planning intervention programmes in order to prepare students for undergraduate studies (Le et al., 2011:116-117).
The TALL is available in English with the TAG being the Afrikaans equivalent. However, there are initiatives focusing on the translation of the test into African languages including Sesotho.
Butler (2017) at the NWU Vanderbijlpark campus undertook an initiative to translate the TALL into Sesotho. The initiative emanated from evidence that students from various institutions were prepared to use learning support material presented in African languages (Butler, 2017:15). The use of African languages in higher education therefore required reliable tests to assess students’ academic literacy in African languages, hence the translation of the TALL.
3.3.2.3 Access to participants’ TALL results
All first-year students at the NWU write the TALL at the beginning of the academic year.
Students at the Vanderbijlpark campus are requested to complete both the English and Sesotho versions of the test, with the completion of the Sesotho test being optional. Since the current study investigated the impact of Sesotho subtitles on participants’ comprehension of content, both Sesotho and English academic literacy levels were important. Permission to access the TALL results was requested from the students and formed part of the ethics application for this study. The English TALL were requested from the subject chair for Academic Literacy on the Vanderbijlpark campus, who is the custodian of the TALL tests. The Sesotho TALL results, which are part of a study by Prof Gustav Butler, were requested from him. The TALL results in this study were used for grouping purposes only, which is outside the original purpose of the TALL results.
The original aim was to use the Sesotho TALL scores as the grouping variable, however, since the Sesotho version is not compulsory, 50% of the participants did not write the Sesotho TALL.
The researcher therefore used the English TALL scores to allocate participants to groups, ensuring that all groups included the different levels of academic literacy.