CHAPTER 5 RESULTS
5.4 RELIABILITY AND VALIDITY OF CONSTRUCTS
The questionnaires used in this study contain different groupings of questions that formed constructs. In the first place, all questions pertaining to each phase (music, context-based inquiry, and computer simulation) formed constructs P1, P2 and P3. Secondly, each phase consisted of four constructs of questions that related to cognitive, affective, behaviour and authentic (AGAU) engagement components. The notations for the four constructs of music (for example) are P1Q1, P1Q2, P1Q3, P1Q4, as they refer to the four questions on phase 1 music (P1) in the pre-test.
Similar notations are used for the other two phases of context-based inquiry (P2) and computer simulation (P3). In the post-tests, the same questions were repeated, and therefore, the same notation was applied. Each of these constructs contains five sub-questions (e.g., 1.1 to 1.5) on a
component of engagement. The validity and reliability of the constructs as determined with the aid of statistics are reported in the following subsections.
Reliability of constructs
The reliability is illustrated in Tables 5-10 to 5-12 for the three phases respectively. Cronbach’s Alpha coefficients statistically determined reliability due to the consistency of participants’
responses to a group of questions. According to Field, Cronbach’s Alpha Coefficient values above 0.6 is an indication of reliable constructs (Field, 2014).
Phase 1 (Music)
The reliability of the constructs of phase 1 (music) follows from the Cronbach’s Alpha coefficients summarised in Table 5-10. The results are given for the four different constructs on engagement (e.g., P1Q1 on cognitive engagement).
Table 5-10: Cronbach Alpha Coefficients on pre-tests for the constructs of phase 1 (music)
Construct N Cronbach’s alpha Reliable?
Cognitive P1Q1 47 0.91 Yes
Affective P1Q2 47 0.88 Yes
Behaviour P1Q3 47 0.84 Yes
Authentic P1Q4 47 0.85 Yes
All Cronbach Alpha values pertaining to phase 1 (music) of the pre-test are above 0.84 (Table 5- 10) thus show excellent consistency in the way the participants answered the questions.
Therefore, the constructs under the engagement strategy, (music) are reliable.
Phase 2 (Context-based inquiry)
Although some of the questions on phase 2 that used context-based inquiry as engagement strategy were similar to those of phase 1 music strategy, there were some slight differences.
Table 5-11 shows the Cronbach’s Alpha Coefficients for phase 2 constructs.
Table 5-11: Cronbach Alpha Coefficients on pre-test for the constructs of phase 2 (context based inquiry)
Construct N Cronbach’s alpha Reliable?
Cognitive P2Q1 46 0.87 Yes
Affective P2Q2 46 0.83 Yes
Behaviour P2Q3 46 0.91 Yes
Since all the Cronbach Alpha values in Table 5-11 were above 0.80, the constructs of context- based inquiry were assured to be reliable.
Phase 3 (Computer simulation)
The reliability of the constructs of phase 3 (computer simulation) follows from the Cronbach’s Alpha coefficient values as summarised in Table 5-12. The results are given for the four different constructs on engagement (e.g. P3Q1 on Computer simulation).
Table 5-12: Cronbach's Alpha Coefficients for phase 3 (computer simulation)
Construct N Cronbach’s alpha Reliable?
Cognitive P3Q1 46 0.90 Yes
Affective P3Q2 46 0.86 Yes
Behaviour P3Q3 46 0.92 Yes
Authentic P3Q4 46 0.86 Yes
Just as the first two phases, the constructs of questions relevant to phase 3 (computer simulations) were also found to be reliable. All Cronbach Alpha values reported in Table 5-12.
are above 0.85. The constructs were thus measured consistently for individual items.
While the consistency of a measure (i.e. reliability) was described in section 5.4.1, it is followed by a discussion of the accuracy of the measure (i.e. validity) in section 5.4.2.
Construct validity
A factor analysis was done to assure construct validity. The same constructs in the pre-test were used as for reliability in the previous section. The results of the factor analysis are given in Table 5-13 for phase 1 (music), Table 5.14 for phase 2 (context-based inquiry) and Table 5.15 for phase 3 (computer simulation).
The number of factors retained for each of the constructs is 1. Kaiser’s measure of sample adequacy (MSA) gave an indication of the inter-correlation among variables (Tabachnick & Fidell, 2007). This index ranges from 0 to 1, reaching 1 when each variable is perfectly predicted by the other variable. The measure can be interpreted with the following guidelines: MSA of 0.80 is meritorious, 0.70 is middling, 0.60 is mediocre, 0.50 is miserable, while a value below 0.50 is unacceptable (Hair et al., 1998).
An assessment of how well these strategies perform can be obtained from the communalities, the measure of variance. The communalities indicate the extent to which an individual item ‘relates’
to the rest of the items.
Table 5-13: Factor analysis for phase 1 (music) for pre-test
Construct N MSA
Number of factors Retained
Percentage variance explained
Cummunalities vary between Cognitive
P1Q1 45 0.87 1 73.96 0.67 and 0.80
Affective P1Q2 45 0.67 1 67.44 0.61 and 0.74
Behaviour
P1Q3 45 0.79 1 61.47 0.74 and 0.86
Authentic
P1Q4 45 0.78 1 63.38 0.76 and 0.86
As seen from Table 5-13, the number of factors retained by the Mineigen criterion is one for each construct (Music cognitive; Music affective; Music behaviour and Music authentic). All MSA values are about 0.7 or above. The communalities that measure the amount of variance shared among the construct questions confirm interrelatedness. Thus, construct validity for the different constructs on music was assured.
Table 5-14: Factor analysis for phase 2 (context-based inquiry) for pre-test
Construct N MSA
Number of factors Retained
Percentage variance explained
Cummunalities vary between
Cognitive P2Q1 45 0.79 1 73.60 0.74 and 0.87
Affective P2Q2 45 0.69 1 84.85 0.64 and 0.77
Behaviour P2Q3 45 0.81 1 91.54 0.75 and 0.89
Authentic P2Q4 45 0.79 1 97.67 0.76 and 0.84
Table 5-14 shows the factor analysis on phase 2 (context-based inquiry) yielding one factor for each engagement component. The high MSA values and low variation in communalities indicate that the constructs contain interrelated questions and construct validity is thus confirmed.
Table 5-15: Factor analysis for phase 3 (computer simulation) for pre-test
Construct N MSA Number of
factors Retained
Percentage variance explained
Cummunalities vary between
Cognitive P3Q1 45 0.80 1 72.57 0.76 and 0.84
Affective
P3Q2 45 0.80 1 85.37 0.77 and 0.84
Behaviour P3Q3 45 0.85 1 92.50 0.80 and 0.89
Authentic P3Q4 45 0.77 1 97.10 0.74 and 0.79
The factor analysis on the constructs of computer simulations (phase 3) also relates to each of the four engagement components (Table 5-15). The variations in communalities are small.
Consideration of the MSA values (0.80, 0.80, 0.85 0.77) which are all above 0.76, the validity of the different constructs on computer simulation was assured.
The results of sections 5.4.1 and 5.4.2 above showed that each construct of related sub-questions in the pre-test was reliable and valid. The constructs could thus be statistically processed and compared to each other, as reported in section 5.5.