The most immediately obvious limitation in the study is that the number of readers tested constitutes too small a sample for any generalisations to be made. Although it was originally intended to test a large enough number of readers to reach statistical significance, problems that became obvious in the course of testing the readers and considering their scores made it clear that it would not make sense to test more readers than had been done on reaching this point. However, the problems
themselves are worth noting and discussing, particularly since at least some of them are likely to arise in the context of other possible cross lingual reading studies.
Ensuring that reading is being tested on texts of equal difficulty when comparing reading skills across languages is essential, yet probably impossible. Texts used in this study were parallel versions of the same content on a website offering historical information on the people of KwaZulu-Natal. It seemed to me that the text had been written in Zulu and then translated to English, yet just as it is difficult to say exactly why this seemed to me as a reader to be so, it is at least as equally difficult to compare the level of difficulty of texts across languages. In English the levels of reading
difficulty are measured by using readability indexes that depend on average length of words in terms of syllables, and the average number of words in sentences. These are exactly the areas in which agglutinative and non-agglutinative languages differ, and therefore official indexes such as these are of no help in judging the level of difficulty across languages. In this study I based my judgement that the texts were of a
comparable level of difficulty on the basis that they were parallel versions of
73
particular content on a website, and therefore dealt with the same content at the same level of detail. However, translation is a practice subject not only to the idiosyncrasies of each translator, but also to the vocabulary of each language and the culture of its speakers. For example, the excerpts of texts used in this study related to the armies of King Shaka, which is surely home ground for the Zulu language, but not so for the English language, and a translator might have to use circumlocutions and abstract descriptions of objects and practices unfamiliar to English readers. This would obviously affect the level of cognitive difficulty for readers. While it might be suggested that the use of “culture free” text would solve the problem, a researcher in pursuit of this quickly discovers that culture free text is actually a unicorn of the reading world, and impossible to find. Therefore the limitation that some uncertainty in ensuring a comparable level of difficulty across texts must be admitted.
The effect of observation will always be a factor in studies of reading. The close and invasive observation that characterised this study would probably affect the reading behaviour of absolutely everyone, even reading super-heroes. In the case of this study, occasional malfunctions of the equipment, or difficulties in adjusting the apparatus to suit each reader was a nuisance, but one that could not be avoided in the course of recording eye movements. The scores recorded are those of readers who read while wearing a mask and knew that they were under observation. It is impossible to know how closely these scores reflect unobserved reading under natural conditions.
A more subtle limitation lay in what first seemed like a good idea. This was the decision to assess and record the eye movements of each reader as they read texts in both English and Zulu, so that in effect each reader provided his or her own match for comparison across languages, thus controlling for variables such as age, educational level and personal idiosyncrasies. However, what this strategy cannot control for is differing levels of reading competence between first and additional languages in the same person. It can also not control for carry-over effects of reading habits learned early in life. To illustrate, it might be that as a child learns to read, he or she develops a set of reading skills that serve as the most economical and effective means of reading in the language in which this early learning takes place. It is possible that when these habitual patterns are set, they are applied to the reading of all languages
74
learnt thereafter, whether or not they are the most effective for reading the newly acquired language and its orthography.
The use of number of fixations and extent of span as established measures of competence in English (where a low number of fixations and a wide span are
associated with skilled reading) conflates features that are currently used to judge the extent of reading competence with the eye movements that one would expect of competent readers processing large grain size chunks of text if indeed eye movements are influenced by grain size of textual processing by readers.
The units of measurement and their lack of direct applicability to reading in a language other than English, for which they have all been developed, made comparisons across languages difficult. Measurements that accommodate cross lingual comparisons are essential for studies in this area. For example, measures of reading speed could perhaps be based on characters per minute instead of words per minute, and a cross lingual measure of span of recognition could be expressed in terms of the number of character spaces recognised in a single fixation, instead of as a portion of an English word.
Finally, it had not been possible to adapt the Visagraph programme to map readers‟
eye movements directly onto words read in the Zulu text, which would have
pinpointed the focus of each fixation. Without this it was impossible to judge exactly how readers were segmenting or parsing the text as they read.
Some of the factors listed above may possibly account for an anomaly in the scores that will have been immediately apparent to anyone familiar with this form of reading testing. The anomaly is that the scores of nearly all the readers are consistent with very low levels of reading skill, and yet all these readers had had the benefit of tertiary education, some to the level of PhD. A number of different interpretations of this feature are possible: scores may have been depressed by the intrusive nature of the testing process as described above, or the scores may simply be wrong, or the scoring system may not be related to actual reading skills, or perhaps the scores do reflect the actual reading competence of these second language academics. If this is so, and if it is reflective of a wider pattern, it is possible that the reading difficulties faced by second language academics are underestimated.
75