3. CHAPTER THREE
3.6 T HE RESEARCH P ROCESS
3.6.2 Data Analysis Strategies
The process of data analysis started soon after the 2012, 2013 and 2014 ANA Grade 9 mathematics tests and 2011 Grade 8 mathematics TIMSS response items were accessed. Firstly, the analysis of the 2012, 2013 and 2014 ANA question papers started after accessing the question papers. Secondly, data analysis continued when scripts were accessed from the schools. Lastly, the alignment index was calculated between the 2012 Grade 9 ANA mathematics test and the Grade 8 mathematics TIMSS response items, the 2013 Grade 9 ANA mathematics test and the Grade 8
108
mathematics TIMSS response items and the 2014 Grade 9 ANA mathematics test and the Grade 8 mathematics TIMSS response items. Documents as records of events were reproduced either by individuals or groups (Cohen, Manion & Morrison, 2011). The trends and patterns that exist in documents are interpreted in a process called document analysis (Creswell, 2014).
ANALYSIS OF ANA QUESTION PAPERS
The analysis of the ANA question was to generate data that responded to the following research question: How are cognitive levels of mathematics tested by ANA reflective of SMP? First, the ANA questions for 2012, 2013 and 2014 were organised into five content areas as stipulated in the CAPS document (DBE, 2011) and these were: 1) Numbers, operations and relations; 2) Patterns, functions and algebra; 3) Space and shape; 4) Measurement; and 5) Data handling and probability.
Second, SMP were explored in the ANA questions and subsequent questioning was coded as per the emerging SMP from each question (see Table 3.3 for codes).
Consequently, Table 3.3 is a synopsis of the generic codes that emerged.
Subsequently, specific codes that emerged from the content areas of ANA and how these emerged are outlined in the next chapter.
Third, these codes were documented in strands which was informed by the theoretical framework, SMP, that they are intertwined, interconnected, inseparable and interwoven (Kilpatrick et al., 2001). Axial coding was followed which allowed this researcher to categorise the emerging codes in strands that outline coherent mathematical activity in each question item (Gibbs, 2012).
Last, the codes were categorised according to their relationship into themes.
Subsequently, axial coding allows emerging codes to be matched against some hypotheses, in this case SMP. Hence, the codes in Table 3, the code SP captured interconnected codes for categories of the following themes: simple procedures, SP for coding of categories for procedures that neither tested computations nor algorithms (Schoenfeld, 1985) for the theme; procedural fluency, PF1-2 denoted coding for categories of fluency in computations and sequence of steps (NCTM, 2000)
109
for the theme, procedural fluency, CU1-2 for categories of comprehending concepts, computations and algorithms the theme, conceptual understanding, SC1-3 categories for problem formulation, problem representation and problem solving (Granberg, 2016; Land, 2017; Stein, Grove & Henningsen, 1996), for the theme (strategic competence), AR1-3 categories for logical thought, explanation and justification for the theme adaptive reasoning. Most studies (Dhlamini & Luneta, 2016; Graven &
Stott, 2012; Maharaj et al., 2015) used the first four SMP except for one study (Graven, 2012). However, that study was limited to mathematics questions that resulted in the development of dispositions. This study closes that gap by using learners’ responses to examine productive dispositions that learners develop as they respond to sampled ANA question. The categories of sense making, utility of maths and valuing mathematics emerged from the analysis of ANA question and were coded as PD1-3, for the theme productive disposition. Subsequently, these categories were coded separately for the identified content areas due to the conceptualisation by Kilpatrick et al. (2001) conception that the strand productive disposition results from learners’ proficiency from the other strands. Conversely, learners who are not proficient in the other strands do not develop productive disposition (Graven, 2012).
To respond succinctly to the first research question, the emerging themes were matched with the NAEP to show clearly the mathematics cognitive levels and levels of complexity (Berger, et al., 2010) posed by the ANA testing in the three consecutive years.
110
Table 3.3: Generic codes for mathematical proficiency
Code Meaning
SP The question requires the learner to write the procedure which requires no calculations (simple procedure).
PF1 The question requires some systematic computations to reach the required answer (computation).
PF2 The question requires a sequence of steps of computations, procedures and relations (algorithm).
CU1 The question requires learners to comprehend a variety of mathematical concepts to reach the required answer (conceptual connections).
CU2 The question comprises of two computations, first compute a value that is subsequently used in the computation required by the question (computational connections)
SC1 The question used familiar problems for learners in the grade (routine procedures).
SC2 The question used a diagram or context to represent concepts, procedures and relations (provision of multiple representations).
SC3 The question requires learners to recall already known procedures, concepts and relations to solve the problem (reproductive thinking).
AR1 The question allows learners to give reasons for their answers, which gives them the opportunity to reflect on their solutions and navigate through concepts procedures and relations (mathematical reasoning).
AR2 The question allows learners to make inferences that are subject to acceptance or rejection (conjecturing).
AR3 The question allows learners to invent suitable commonalities of mathematical relations to make a proof (an analogy).
PD1 A mathematical problem is useful to make sense through the use of diagrams and representations (sense making).
PD2 Using mathematics to solve real life problems (utility of mathematics).
PD3 A mathematical problem is useful and important in solving a reasoning or thought provoking problem, which is regarded as a complex problem (valuing mathematics).
SCRIPT ANALYSIS
The three questions that were sampled from the 2014 Grade 9 ANA mathematics questions were analysed using document analysis and the instrument used was adapted from Luneta (2015). There were a total of n=1250 scripts that were analysed by exploring SMP exhibited by learners and categorising them in the following four variables; correctly answered, partially answered, incorrectly answered and no response. The assumption of the current study was as follows. (1) A correct response in line with the marking guidelines is enough to justify that the learner was proficient in the question and the learner fully exhibited the SMP that the ANA posed. (2) A partially answered question only justifies that a learner still requires additional assistance in the SMP that the ANA examined. (3) An incorrect response is enough to justify the fact that the learner is not proficient in the SMP that the ANA examined.
(4) A no response may mean that the learner skipped the question because of lack of
111
proficiency in the SMP that the ANA examined but on the other hand, a no response might also be as a result of not finishing answering the questions which may be the result of time allocated for the test or slow pace of a learner.
The current study uses the standard deviation, mean bar graph and radar to present findings. The use of these descriptive statistics to present and interpret data has been widely accepted in research (Gorard, 2005; Lathrop, 1961; Saary, 2008).
Bar graphs are often limited to displaying frequencies (Saary, 2008). However, the standard deviation is useful in instances where there is a need to measure variance of data from a comparable point (Lathrop, 1961). The use of mean deviations has more advantages than the standard deviation (Gorard, 2005) as it easier to understand and suitable for distributions that may have minute errors. As such, the trustworthiness of using radar has been documented and justified in terms of its competence, popularity, recency, corroboration and proximity (Nurse, Agrafiotis, Creese, Goldsmith & Lamberts, [Sa]). Additionally, radar are useful in presenting multivariate data (Feldman, 2013; Saary, 2008).
CALCULATING THE ALIGNMENT INDEX
In calculating the Porter’s alignment index, firstly, there was a need to analyse the cognitive levels as well as the content messages conveyed by the Grade 9 mathematics ANA test papers and the 2011 Grade 8 mathematics TIMSS test response items. Secondly, matrices were formed and the hits in the cells were documented using a protocol. This was done to calculate the Porter’s alignment index.
The question totals in the ANA papers were: cell Xi the 2012 matrix with n=59 questions; cell Xj the 2013 matrix with n=62 questions; cell Xp the 2014 matrix with n=61 questions; and cell Yi, the TIMSS matrix with n=90 questions. These matrices are shown in Chapter 4, the presentation of findings.
112