CHAPTER 3: METHODOLOGY
3.5 Eye tracking
to complete the activity. The current study’s questionnaire consisted of nine questions, which were mainly based on both questionnaires. The questionnaire retained the Leppink et al. (2013) question regarding the impact of the activity on comprehension, which was excluded in the Leppink and Van den Heuvel (2015) questionnaire. The questions in the current questionnaire focused on the difficulty of the content, terminology and language used in the videos, the impact of the subtitles on comprehension, and the perceived mental effort used to complete the activity.
The participants responded to this nine-item questionnaire at the end of each video viewed, by rating their perception of their experience of viewing the videos in the experiment on a ten-point scale. Similar to the Leppink and Van den Heuvel (2015) questionnaires discussed above, the responses ranged from 1 (“not at all”) to 10 (“is completely the case”). Throughout the questionnaire the response options were presented in a consistent manner, with the negative response on the left-hand side, and the positive response on the right-hand side, with average or neutral responses in the middle. This was done based on the recommendation by Dörnyei and Taguchi (2009:32) so as to reduce possible cognitive effort and distraction involved in the process with participants internally rating and then marking the chosen option on the questionnaire. The task load questionnaire can be found in Appendix F.
The last questionnaire was completed only once, after the fifth video was viewed, and focused on the participants’ general experience of subtitling and perceptions of subtitling in education (See Appendix G). The questionnaire contained questions that measured the participants’ perceptions of the helpfulness of subtitles in education, their preference for subtitle type, and which subtitles they would recommend to others. In addition, the questionnaire focused on the participants’
exposure to subtitled television programmes in general. The participants indicated how often they watched TV, if the TV programmes they watched contained subtitles and the language of the subtitles.
3.5.1 Eye-tracking system: iViewXTM RED
The SMI iViewX RED500 eye tracker was used to monitor participants’ eye movement whilst watching the video clips. The RED500 is a remote eye-tracking device that is table-mounted. The setup of the device consists of the iViewX workstation, a stimulus screen and the eye-tracking module. The eye-tracking module is located at the bottom of the stimulus screen and uses infrared LEDs to illuminate the eyes (SMI, 2011:168).
Figure 3-1: SMI’s iViewX™ RED500 eye-tracking system
This eye-tracking system works with a built-in camera which views and records the eye behaviour of the participants. According to SMI (2011:166) it has a sampling rate of 500 Hz, meaning that it takes 500 photographs of the eye per second, calculating the pupil position and size, and relative head movement. The RED500 allows for the measuring of participants’ eye movements without any physical contact. Given this non-intrusive nature of the device, experiments take place under relatively natural viewing conditions. However, the participants were required to sit as still as possible for the duration of the experiment as movements would influence the data collection.
In eye-tracking studies, mascara can be mistaken for the pupil, thus leading to the camera jumping from pupil to the dark lashes, resulting in inaccurate calibration. To avoid this issue, participants wearing eye make-up were requested to remove it. Eye-makeup remover pads and a mirror were provided for participants who put on mascara and they removed the makeup themselves. The reflection caused by eye glasses can also lead to inaccurate calibration. Four participants who signed up for the study wore glasses and two unfortunately had to exit the study due to calibration difficulties. Participants were excluded if their eye tracking ratio was less than 85%.
3.5.2 SMI Experiment Center™ 3.5
The experiment was set up and run from SMI’s Experiment Center 3.5, a component which is specific to SMI iViewX™ software. In preparation for the experiment, the relevant stimuli were selected and inserted into Experiment Center 3.5. A calibration element is always automatically inserted as the first ‘stimulus’ participants see. The process of calibration ensures that the participants’ eye behaviour is recorded accurately by the system. This is important as people have different eye shapes with varying eyeball radii (Holmqvist et al., 2011:128), which affect where they focus when viewing an object. Calibration therefore ensures accurate recording of the place that participants viewed the screen whilst the stimulus is presented.
During calibration participants are shown a dot that moves across the screen. Holmqvist et al.
(2011:128) explain that the calibration area has predefined points, commonly 2, 5, 9, 13, or 16, which cover the area where the experiment stimuli will be presented. The participants are requested to follow it with their eyes without moving their heads. When the dot stops the participants have to focus their eyes on the centre of the dot until it moves again. This process continues until all points presented are fixated. All the while the system observes the participants’
pupil position at each point in order to select the correct pupil reflections for all points fixated (Holmqvist et al., 2011:129). In order to ensure accuracy a predetermined deviation value should not be exceeded (Holmqvist et al., 2011:128). According to the authors (2011:133), some systems calculate an “accuracy value of the average deviation between markers and gaze position”. A maximum deviation of 0.5° is recommended (Holmqvist et al., 2011:133). The authors recommend that recalibration be conducted should the validation value exceed the required accuracy value.
Once calibration is successful, the recording may start. The remaining stimuli that follow calibration were sorted according to the order in which they will be presented. For the current study the order was the calibration, pre-video, and lastly the five primary videos.
Figure 3-2: Screenshots of calibration process (from Saiz Manzanares et al., 2020:6)
3.5.3 SMI BeGaze™ 3.5
All data was captured and recorded in real-time, and analysed using SMI BeGaze™ 3.5, which is also an element specific to SMI iViewX™. It is used for behavioural and gaze analysis of eye- tracking data (SMI, 2009b:2). It automatically groups data based on individual experiments and participants.
Eye-tracking data was recorded for the viewing of the entire screen, but only the viewing data for the subtitle area at the bottom of the screen was considered for analysis: each individual subtitle was marked as an area of interest (AOI) and data was analysed in terms of these AOIs.
During data analysis, recordings were grouped and presented according to each AOI. The AOIs were marked after the data had been collected using AOI editor in BeGaze™. They were marked with the intention of ensuring that the retrieval of the recording of eye-tracking data focuses specifically on the AOIs (in terms of time spent in the AOI, number of fixations, fixation duration, etc.). Once AOIs are marked, BeGaze automatically clusters the eye-tracking data per AOI. The data may be viewed in BeGaze itself (scan paths, etc.), however for the purpose of the current study, the data was exported from BeGaze in Excel format for statistical analysis.
3.5.4 Eye-tracking measures
Although eye-tracking systems record and allow for the investigation of many measures of eye movement, only a specific aggregated measure was used in the current study, namely the Reading Index for Dynamic Texts (RIDT). RIDT is a score “of the degree to which a particular
subtitle was read by a particular participant” (Kruger & Steyn, 2014:110) and is determined through the following formula:
Figure 3-3: RIDT formula (Kruger and Steyn, 2014:110)
The RIDT score indicates the extent to which a subtitle was read by each participant, thus allowing for conclusions to be made regarding the effect of the subtitle reading on comprehension. As Kruger and Steyn (2014:116-117) assert, we can only understand the role of subtitles in learning when “we consider the correlation between comprehension and the degree to which participants actually read the subtitles”. The RIDT score is interpreted on a scale, with a very low score depicting that subtitles were read to a lesser degree and a high score depicting more extensive reading (Kruger et al., 2014:6).
However, the RIDT formula may be problematic, as Matthew (2019) discovered in his study.
Some participants in his study had a RIDT score of zero for some of the subtitles. This meant that the subtitles were not read. However, the fixations counts were higher than zero, indicating that the subtitles were read. Upon closer investigation of the RIDT formula, Matthew (2019) found that when the participants rescanned a subtitle (regressed), the formula detracted these from the number of fixations. This means that for those subtitles with zero fixations, the participants possibly made numerous regressions. For Matthew (2019:46), multiple regressions were inevitable since the language of the audio and the subtitles (English) is an additional language for the participants in his study. As discussed in the previous chapter, many South African students access education through EAL, however, many enter higher education with poor proficiency in English. Due to this and the lack of reading skill development, there was a possibility that the participants would need to reread some of the subtitles in order to understand and confirm information. In order to avoid a false representation of how the subtitles were processed, Matthew (2019) adapted the RIDT formula in order to accommodate the multiple regressions, which the RIDT penalised. The regressions did not detract from the unique fixations. This resulted in a new formula called the Unique Fixations per Mean Word (UFMW).
The UFMW score was calculated for each subtitle by dividing the number of unique fixations for those subtitles (i.e. fixations without refixations) by the mean number of words (number of words of subtitle divided by the average word length of subtitles in the video) (see Equation 1).
Equation 1: UFMW= Fixations without Refixations / Mean Words (Matthew, 2019:47)
The current study shares similarities with Matthew’s study with regard to the population and their context. The participants in both studies are non-native speakers of English, who are students at the NWU, Vanderbijlpark campus. Furthermore, videos of recorded academic lectures were used and the videos have an English audio track and English subtitles. What is different, however, is that the current study also includes Sesotho subtitles and had a different focus than Matthew’s study. The current study seeks to investigate the effect of L1 subtitles, both full and keyword, on comprehension. The inclusion of L1 subtitles emanates from the findings of some research that L1 subtitles may contribute positively to comprehension (Mahlasela, 2012). The investigation of L1 subtitles will make a contribution to the ongoing discussions about the benefits of mother- tongue education. In order to determine whether subtitles contribute to comprehension it is essential to determine if and to what extent the subtitles are read. The use of the eye-tracking technology makes this possible through the reading indices such as RIDT and UFMW.
The current study therefore set out to determine the extent of subtitle reading by calculating the RIDT as well as the UFMW values for all videos viewed with subtitles. Even though found to be flawed, calculation of the RIDT would enable the researcher to compare the findings of the current study to those of previous studies that used the RIDT formula to calculate the degree of subtitle reading (Kruger & Steyn, 2013; Kruger et al., 2014). In addition to RIDT, the modified UFMW was also calculated since multiple regressions were anticipated in the current study, given the similarities to the study by Matthew (2019). Apart from reading in English, regressions were also anticipated for the Sesotho subtitles since students (1) are not used to Sesotho subtitles and may have never been exposed to them before, given the subtitling culture in South Africa, which provides mostly English-only subtitles on public television; and (2) have primarily received education through English, which is the main LoLT in South Africa. To conclude, eye tracking was therefore used in the current study to gain insight into the extent to which subtitles were processed in order to draw conclusions regarding the effect of subtitles on comprehension.