3.5 Research design
3.5.1 Cohort one
3.5.1.2 Data collection methods and tools
Table 3-6 below provides a cross tabulation of races and home languages represented in the sample for cohort one:
Home Language
Total English Afrikaans Zulu Xhosa Swazi Ndebele
South Sotho
North
Sotho Tsonga Tswana Venda Other
Race Black 7 0 206 10 5 1 1 2 2 3 1 8 246
White 8 0 1 0 0 0 0 0 0 0 1 3 13
Asian 303 1 1 0 1 0 0 0 1 0 0 1 308
Coloured 11 0 0 0 0 0 0 0 0 0 0 0 11
Other 28 0 1 0 0 0 0 0 2 0 0 0 31
Total 356 1 209 10 6 1 1 2 5 3 2 12 609
Table 3-6 Race and home language cross tabulation for cohort one (Perception)
but may not reflect the real impact of learning in the ‘new learning environment’. Thus, a Black student who enters university from a school environment that is inferior to that of his White counterparts may score lower marks than his White classmates, reflecting the legacy of his disadvantaged schooling, but now that the ‘playing field is leveled’, may find his ‘improvement score’
matching that of the White students.
This is best explained by a hypothetical example. Black, previously disadvantaged students may score on average 20% lower in cognitive tests at the end of their first year at university than White,
‘privileged’ classmates. A researcher would draw certain conclusions on the impact of various factors (such as teacher student race match) on the performance of these students. For example, if Black students consistently obtained lower test scores than their White counterparts, despite being taught by Black teachers (with whom they are racially congruent), a researcher might conclude that teacher student race match is not a predictor of cognitive test performance. However, this could be misconceived. If, for example, a comparison of a pre-test score conducted at the beginning of the year was made with a post-test score at the end of the year, it may be found that Black students who were taught by Black teachers improved by a greater margin than their White counterparts. If such previously disadvantaged students were to improve in their cognitive test performance to a greater extent than their ‘privileged’ classmates, for example, it would demonstrate that such previously disadvantaged students were equally capable students at least, despite lower overall test scores and would suggest that any race based performance gaps were indeed the result of a disadvantaged educational environment rather than inherent learning deficiencies. Such insights would not be possible by using a single post test score as a dependent variable. A researcher would draw a very different set of conclusions about the impact of teacher student racial congruence under these circumstances. It is the opinion of the author that this use of an ‘improvement score’ rather a single post test score as the dependent variable in studies of this nature provides a more accurate view of the impact of factors such as teacher student congruence.
In order to compare the results of this study with many of those conducted internationally using a single test score as the independent variable, two analyses of the data are performed: one using a single post test score as dependent variable and another using ‘improvement score’. The results obtained by using each of the approaches are compared.
3.5.1.2.a.2 Overview of cognitive test performance research approach
In respect of the research objectives related to examining cognitive test performance, a pre- and post- training assessment test instrument was developed to assess the students’ cognitive learning in respect
of each of the three courses’ subject matter (see Appendix A: Pre- and Post Assessment Test). To ensure consistent assessment, the same instrument was used as both pre- and post-assessment tool.
The assessment test took the form of multiple choice questionnaires, an assessment approach not uncommon in the field of Information Systems and Technology when assessing technical skills (Roberts, 2006). Although the assessment instrument contains questions relating to six subjects (‘word processing’, ‘spreadsheets’, ‘databases’, ‘using a computer and managing files’, ‘I.T.
concepts’ and ‘information and communication (networks)’, only the results for ‘spreadsheets’,
‘databases’ and ‘information and communication (networks)’ were included in the study. This was due to the fact that there were no specific lectures held in respect of the excluded subjects (‘word processing’, ‘using a computer and managing files’ and ‘I.T. concepts’) during the semester in which the study was conducted, thus precluding any analysis of the impact of teacher student congruence factors in learning outcomes. In the case of the included subjects, ten multiple choice questions with mutually exclusive options were presented for each of the three subject areas, based upon the course content for the semester.
Three separate pre-tests were administered to each student for each of the three courses in advance of any lectures taking place. Post tests (the same instrument) were subsequently administered immediately after completion of the lecture period for each course (at the end of the semester in this case). For each course, each student’s pre-test score was then subtracted from the post test score to obtain an ‘improvement score’. This approach is consistent with Phillips’ (2002) ROI analysis model which uses improvement score as a measure of what Phillips terms the ‘second level of measurement’
(viz. Learning). Phillips’ model refers to a ‘baseline level’ of performance in terms of any objectives identified at level 2 of his framework, and a ‘post training performance level’, which are compared to produce a ‘performance improvement score’. In this study, the pre-test score corresponds to Phillips’ ‘baseline score’ at level 2, while the post test score represents the ‘post training performance’
level of each student (Phillips, 1997, Phillips and Stone, 2002).
Investigating student perceptions of collective self-efficacy
The objectives related to student perceptions of collective self-efficacy (in respect of teaching ability) for cohort one were investigated by means of a survey conducted with the same group of students that participated in the pre and post assessment components of the research (see Appendix B:
Perception Questionnaire). This instrument was designed to measure students’ perceptions of the efficacy of teachers within their own reference groups (in this case, reference groups in respect of race, home language and gender). This is taken to be a measure of what Bandura terms ‘collective self-efficacy’, which refers to a group's ‘shared belief in its conjoint capabilities to attain their goals
and accomplish desired tasks’, (in this case, IS&T related teaching capability) (Bandura, 1986, 1994, 2000). Bandura’s guidelines on the development of instruments to measure collective self-efficacy informed the creation of the instruments used in this study, as described in Table 3-7.
Bandura’s guidelines for the development of inctruments for measuring collective self-efficacy (Bandura, 1994, 1995, 2000)
Justification for ‘Perception/ Collective Self- Efficacy’ Instrument (See Appendix B:
Perception Questionnaire)
Measurement guideline Measurement options Choice Justification
1. Bandura refers to two possible approaches to the measurement and evaluation of collective self-efficacy:
a. The aggregation of appraisals by members of a reference group of their their personal capabilities in terms of the functions they peform in the group;
Since the efficacy of the teacher (not the student) was being measured, option b was selected to inform the development of the
‘perception/ collective self-efficacy’ instrument used in this study.
b. Aggregate appraisals by members of the capabilities of the group as a whole
2. Measurement of collective self-efficacy should occur in terms of one of three dimensions:
a. Perceived efficacy to take action as a group.
The dimension referred to in option b (viz.
‘perceived capability of other community members’) was selected for the ‘perception/ self- efficacy’ instrument used in this study as it was the students’
perception of the efficacy of individual teachers within the reference group that was being measured.
b. Perceived capability of other community members.
c. Perceived efficacy to solve problems as a group.
3. Perceptions of self- efficacy may vary with the tasks at hand and with other contextual factors.
Questions about perceived self-efficacy should be precise and refer to specific circumstances.
The questions selected for this instrument referred specifically to
‘teaching capability’.
Table 3-7 Instrumentation justification based on Bandura’s guidelines (Source: (Bandura, 1994, 1995, 2000)
The perception survey asked the following questions of each student:
1. Which of the following is true about your teacher’s gender?
a. I learn better from a teacher who is of the same gender as me.
b. I learn better from a teacher who is not of the same gender as me.
c. The teacher’s gender makes no difference to how I learn.
2. Which of the following is true about your teacher’s race?
a. I learn better from a teacher who is of the same race as me.
b. I learn better from a teacher who is not of the same race as me.
c. The teacher’s race makes no difference to how I learn.
3. Which of the following is true about your teacher’s home language?
a. I learn better from a teacher whose home language is the same as mine.
b. I learn better from a teacher whose home language is not the same as mine.
c. The teacher’s home language makes no difference to how I learn.
4. Which of the following is true about your teacher speaking your home language while teaching you?
a. I learn better when my teacher speaks my home language while teaching me.
b. I learn better when my teacher does not speak my home language while teaching me.
c. The teacher using my home language while teaching makes no difference to how I learn.
Questions 3 and 4 both relate to home language as a reference group for collective self-efficacy, but differ in focus. Whereas question 3 focuses on the home language of the teacher, whether spoken as a medium of instruction or not, question 4 focuses on the actual language used during teaching.
For each question, and for each of the demographic variable categories (gender, race, home language and language of instruction), a Chi-square goodness-of-fit test was applied to investigate the frequency of response. Table 3-8 shows how response frequencies in respect of specific questions were interepreted in relation to a measurement of collective self-efficacy.
Reference group Question Response option
Interpretation of response frequencies as a function of an estimation of collective self-efficacy (in respect of teaching ability) Higher than
expected response frequency
Lower than expected response frequency
Gender 1. Which of the
following is true about your teacher’s gender?
a. I learn better from a teacher who is of the same gender as me.
High collective self- efficacy
Low collective self- efficacy
b. I learn better from a teacher who is not of the same gender as me.
Low collective self- efficacy
High collective self- efficacy
c. The teacher’s gender makes no difference to how I learn.
Passive (no impact on collective self- efficacy rating)
Passive (no impact on collective self- efficacy rating)
Race 2. Which of the
following is true about your teacher’s race?
a. I learn better from a teacher who is of the same race as me.
High collective self- efficacy
Low collective self- efficacy
b. I learn better from a teacher who is not of the same race as me.
Low collective self- efficacy
High collective self- efficacy
c. The teacher’s race makes no difference to how I learn.
Passive (no impact on collective self- efficacy rating)
Passive (no impact on collective self- efficacy rating)
Home Language 3. Which of the following is true about your teacher’s home language?
a. I learn better from a teacher whose home language is the same as mine.
High collective self- efficacy
Low collective self- efficacy
b. I learn better from a teacher whose home language is not
Low collective self- efficacy
High collective self- efficacy
Reference group Question Response option
Interpretation of response frequencies as a function of an estimation of collective self-efficacy (in respect of teaching ability) Higher than
expected response frequency
Lower than expected response frequency the same as
mine.
c. The teacher’s home language makes no difference to how I learn.
Passive (no impact on collective self- efficacy rating)
Passive (no impact on collective self- efficacy rating)
4. Which of the following is true about your teacher speaking your home language while teaching you?
a. I learn better when my teacher speaks my home language while teaching me.
High collective self- efficacy
Low collective self- efficacy
b. I learn better when my teacher does not speak my home language while teaching me.
Low collective self- efficacy
High collective self- efficacy
c. The teacher using my home language while teaching makes no difference to how I learn.
Passive (no impact on collective self- efficacy rating)
Passive (no impact on collective self- efficacy rating)
Table 3-8 Interpretation model for ‘perception questionnaire’ response frequencies