CHAPTER 3: METHODOLOGY
3.4 Materials used
3.4.2 Subtitled video clips
3.4.2.1 Video complexity as an indicator of video comparability
After the experiment videos were created, the full transcripts were tested for complexity in order to determine whether they were comparable. The videos had to be comparable in terms of complexity and the level of language used, in order to ensure that participants’ performance was not influenced by differences in levels of difficulty. In order to do that the full subtitle scripts, which are close to being the full transcript of the audio track, were used to compare the complexity of the subtitles.
The full subtitle scripts of the videos were analysed for readability and complexity using the following readability tests, which are freely available online: Coh-Metrix, Lexile and Flesch- Kincaid. Various text features account for text readability and difficulty such as text length, grammatical characteristics, cohesion, text abstractness, word concreteness, syntactic simplicity and narrativity (Green et al., 2010; Graesser, McNamara & Kulikowich, 2011). In order to ensure comparability of the subtitle texts and to ensure that the videos are comparable in their level of complexity, the texts were analysed for these features through the Flesch- Kincaid, Coh-Metrix and Lexile indices. These measures help determine each text’s level of difficulty and matches it against the reader’s ability. The scores and comments from the test results were used to compare the levels of complexity of the videos and thus also the suitability and comparability of the videos for the experiment. Since the participants were first-year students, and since the videos are recorded lectures from a first-year module, it was anticipated that the complexity level of the videos would not exceed a Grade 12 reading level.
The scores are demonstrated in Tables 3-3-3-4, and 3-5.
The results of the Flesch-Kincaid analysis are demonstrated in Table 3-3. According to this analysis, the texts across all five videos were easy to read with the highest grade level being Grade 6.
Table 3-3: Flesch-Kincaid reading ease scores
Video Topic Duration Number of
Words
Flesch-Kincaid Reading Ease
Flesch-Kincaid Grade Level
1 Compliance 00:08:56 1109 89.9 3
2 Attraction 00:12:30 1410 82.1 4
3 Aggression 00:09:23 994 72.6 6
4 Social
Psychology 00:10:57 1080 78.4 4
5 Community
Psychology 00:09:08 898 62.3 6
Further analysis was done using the Coh-Metrix measures, which analyse the ease or difficulty of texts on five different dimensions: narrativity, syntactic simplicity, word concreteness, referential cohesion, and deep cohesion. For a given text, each of these dimensions is given an “ease score compared to thousands of other texts” (Nelson, Perfetti, Liben, & Liben, 2012:15).
Table 3-4: Coh-Metrix measures
Video Topic Duration Number of Words
Narrativity
%
Syntactic simplicity
%
Word concreteness
%
Referential Cohesion
%
Deep Cohesion
%
1 Compliance 00:08:56 1109 89 82 34 27 89
2 Attraction 00:12:30 1410 86 73 2 34 59
3 Aggression 00:09:23 996 62 68 5 33 52
4 Social Psychology
00:10:57
1080 69 90 5 20 39
5 Community Psychology
00:09:08
898 58 75 5 20 44
The results of the Coh-Metrix analysis of the texts is demonstrated in Table 3-4 above. The narrativity dimension demonstrates the extent to which a text is story-like and contains familiar words and oral language (Graesser et al., 2011:228). The authors (Graesser et al., 2011:230) explain that syntactic simplicity describes the simplicity level of the sentence structures, with a low value indicating an easier to process text. Since texts are transcripts of speech text, the high percentages for narrativity and syntactic simplicity are understandable. In terms of the degree of narrativity, the scores range from fairly high to very high (58%-89%) meaning that all texts were relatively easy to read and understand. A similar variation was found in terms of syntactic simplicity which ranged from 68%–90%.
In terms of word concreteness, the authors (Graesser et al., 2011:230) explain that text with words that “are concrete, are meaningful, and evoke mental images - as opposed to being abstract” will have higher scores. Table 3-3 indicates low levels of concreteness, from 2% to 34 %, meaning that the texts were more abstract and this is understandable as the content of the videos deals with concepts in psychology.
Referential cohesion, in turn, refers to the degree to which terms and concepts are connected with each other throughout the text (Graesser et al., 2011:230). The authors further explain that science texts have a higher referential cohesion than social studies and arts texts. This explains the lower referential cohesion for the texts in the current study, which range from 20%
to 34%. This low referential cohesion measures mean that for all subtitles the connections between sentences were low, requiring the reader to infer the relationship between the sentences thus making them difficult to read.
With regard to deep (or causal) cohesion, on the other hand, “scores are higher to the extent that clauses and sentences in the text are linked with causal and intentional (goal-oriented) connectives” (Graesser et al., 2011:230). High scores for deep cohesion are therefore beneficial in making texts, particularly those with difficult topics, easier to understand. The deep cohesion scores for the texts ranged from 39% to 89%, as indicated in Table 3-3, meaning that some consisted of connectives, making them easier to understand, whilst others with lower scores did not have this support, making these videos possibly difficult to understand.
Although there is a large difference in terms of deep cohesion, the other categories do not display such great variation. The texts, and by implication the videos, can therefore be considered to be comparable.
A third and final analysis was done using the Lexile Analyzer, which measures the syntactic and semantic complexity of the text by analysing its characteristics, such as sentence length and word frequency (https://lexile.com/). The result is a Lexile measure, which indicates the text complexity, together with details of the word count, mean sentence length and mean log frequency. Table 3-4 illustrates scores from the Lexile analysis of the videos.
Table 3-5: Lexile measures
Video
Duration Number of Words
Lexile Measure
Lexile grade equivalent
Mean Sentence
length
Mean Log Word Frequency
1 Compliance 00:08:56 1109 500-600L 3 7.63 3.81
2 Attraction 00:12:30 1410 400-500L 2 7.44 3.78
3 Aggression 00:09:23 996 600-700L 3 9.03 3.56
4 Social
Psychology
00:10:57 1080 500-600L 3 8.15 3.53
5 Community
Psychology
00:09:08 898
500-600L 3 7.55 3.58
According to the Lexile measures as presented in Table 3-5 the videos were comparable in the sense that not much variation was found between the videos in terms of the Lexile measure, grade equivalent, mean sentence length and mean log word frequency. The Lexile analysis indicated the mean sentence length to be ranging from 7.44 to 9.03.
The mean log frequency calculates the frequency of a word in a corpus of 600 million words (Lennon & Burdick, 2004:3). The authors (Lennon & Burdick, 2004:3) further explain that the measurement is “used to determine the semantic difficulty of a text” with a lower measure indicating the use of less frequent words. The mean log frequencies ranged from 3.53 to 3.81 demonstrating lower frequencies, which may result in difficulty in reading, this can be expected as the texts contain Psychology terminology.
According to the results of the above analysis, the subtitles may generally be described as texts containing shorter sentences with words of lower frequencies. The Lexile measures ranged from 400L–700L for which the grade equivalents are Grade 2 and Grade 3 respectively. According to these measures the texts’ complexity was comparable in that, according to the analysis, the texts were appropriate for the lower grades and may be considered to be suitable for first-year students.
The discussion above reflects on the different types of analyses done with the intention of determining the comparability of the videos. From all the tests done it can be seen that the videos are relatively easy, with the highest grade being Grade 6 for the Flesch Kincaid and Grade 3 for the Lexile. These grades are lower than the grade of the participants, which is anticipated to be at Grade 12 since they have passed Grade 12. A factor that may contribute to the lower grades is the fact that subtitles are actually a transcription of speech and consideration should thus be given to the difference between oral texts and written texts. Furthermore, it should be taken into consideration that the texts are from videos of an introductory module intended for first-year students, thus the use of language will be simple in order to be appropriate for the intended audience. Even though the analyses indicate easy texts, it is important to note that some learners
leave high school with under-developed reading skills as discussed in the previous chapter.
Under-developed reading skills were related to the findings by Hefer (2013:171) that the Sesotho L1 participants spent more time reading subtitles compared to their English L1 counterparts.
These Sesotho L1 participants also exhibited slower reading of Sesotho subtitles compared to their reading of the English subtitles; the Afrikaans L1 participants, on the other hand, demonstrated no difference in the reading of Afrikaans and English subtitles (Hefer, 2013:171).
This finding further confirms that indeed some participants enter HE with under-developed reading skills. Based on this finding it may be expected that some of the participants of the current study may exhibit a similar under-development in their reading skills, therefore making it important that the subtitles are easy to read so that all participants may have an equal opportunity at reading them. The reading of the subtitles is an important consideration if their impact is to be examined.