2. CHAPTER TWO
2.2 M ONITORING M ATHEMATICS E DUCATION
2.2.6 Alignment of Systemic Assessments
Content of instruction and cognition are vital in determining what students achieve in learning (Porter, 2002). Policymakers need information on content of instruction and the level at which teaching materials, such as textbooks and assessments, advance the content of instruction (Ndlovu & Mji, 2012). Furthermore, Porter (2002) points out that content of instruction, teaching and learning materials and standards are essential in monitoring curriculum change. Hence, there is a need to align what is taught and what is learned (Nazeem, 2010). The absence of alignment between what is taught in reference to the curriculum and what is tested, may result in teachers focusing their teaching upon what is tested (Graven & Venkat, 2014). Alignment has
32
been described as the extent to which standards and assessments agree. This guides the education system on what learners must learn and achieve (Webb, 2007). The use of Webb’s alignment procedures was widely documented in the United States of America. Researchers such as Martone and Sireci (2009) used Webb’s alignment to view how content, instruction and assessment were linked. Other researchers such as Porter, Smithson, Blank and Zeidner (2007) have modified the Webb’s alignment and justified alignment with a quantitative index.
In an attempt to contribute to the issue of quality, the current study compares the ANA tests with the 2011 TIMSS Grade 8 response items using the Porter’s alignment index to verify the content message that the ANA relate to the learning and teaching of mathematics points as a determinant of student achievement (Porter, 2002). Studies on systemic assessment in America have shown that low performance in America in the 1996 and 2003 was caused by insufficient knowledge of content (Ferrini-Mundy & Schmidt, 2005; Suter, 2000). A study by Schmidt and McKnight (1998) revealed that American learners performed poorly in the 1995 TIMSS, especially in Geometry due to insufficient teaching of content. Furthermore, they pointed out that the American curricula lacked coherence and provided students with less rigour. Another important consideration by Ferrini-Mundy and Schmidt (2005) was that it was significant to compare the richness in message that the NAEP, TIMSS and PISA to make sense of some noticeable progress in mathematics education. This is the rationale that made this researcher decide to align the Grade 8 TIMSS response items and the Grade 9 ANA to compare the content message that these studies are posing in the mathematics discourse.
According to Porter (2002), alignment is a worthwhile tool for measuring content message between the content of instruction and content of instructional materials such as content standards, textbooks and achievement tests. Similarly, in their description of the alignment index, Porter et al. (2011) argued that the alignment index defines the contents of intersections of topics and cognitive demands and also assesses the extent at which two documents have the same content message. In addition, Fulmer (2011) describes the alignment index as the measure of the degree to which assessments adequately measure standards to help schools achieve
33
accountability. Finally, Edwards (2010) described alignment as the degree of agreement of standards and assessments which further inform one another and shape students’ learning, with emphasis on the quality of the relationship between the two.
Polikoff, Porter and Smithson (2011) support coherence in tested outcomes and the curriculum which can only be visible by calculating the alignment index. They argued in the context of systemic assessments and show how aligning assessments to state standards in America has been essential:
‘Twenty years on since systemic reform and the state systemic initiatives, instructional coherence remains an important part of standards-based reform in its current incarnation, the No Child Left Behind Act (NCLB). The text of NCLB echoes the framework for systemic reform laid out in the early 1990s, claiming that improving student achievement and ensuring access to a high quality education for all will be accomplished first and foremost through ‘‘ensuring that high-quality academic assessments, accountability systems, teacher preparation and training, curriculum, and instructional materials are aligned with challenging State academic standards so that students, teachers, parents, and administrators can measure progress against common expectations for student academic achievement’’ (No Child Left Behind Act of 2001, 2002, pp.
1439-1440). NCLB mentions alignment dozens of times, specifically focusing on the alignment of assessments with content standards. States are required to show to the Department of Education that their assessments are aligned with the content specified in their standards, and they are also required to assist local entities in identifying curricula that support those standards.
Clearly, the coherence of the system remains of utmost importance in the vision of standards- based reform under NCLB” (Polikoff et al., 2011: 2).
In their study Polikoff et al. (2011) tried to address the following question; Are state standards and assessments aligned with one another? Their findings led them to conclude that the answer was NO. However, they acknowledged that this was dependent on how alignment was defined. Such discrepancies in calculating alignment made them conclude as follows: (1) when the definition of alignment was relaxed in the levels of cognitive demand, alignment increased to 0.5 or even higher.
(2) Some content is over-tested and other under-tested. (3) State standards and assessment in the specimen of states were not aligned as per intentions. (4) Once states had been made aware of the low alignment, the alignment between state standard and assessments increased. (5) There were many topics that were specified
34
in the state standards that were not tested at all and so in some cases the state standards did not agree with the cognitive levels resulting in misalignment.
In their study, Ndlovu and Mji (2012) calculated the alignment between the TIMSS and the South Africa’s Revised National Curriculum Statement (RNCS). Their findings revealed that there was poor alignment between the RNCS and the TIMSS which shows a lack of attention in benchmarking the RNCS and international assessments. They suggested that policymakers must engage urgently to align the TIMSS and the RNCS to make South Africa’s curriculum reform relevant. This however, is an ambitious recommendation that the current researcher does not fully agree with in view of the fact that RNCS and TIMSS do not serve the same purpose.
When measuring the alignment index, Porter (2002) aligned of the state assessments and the state standards by calculating the alignment index. Cells of cognitive demands and topics were formed which were described as tools for measuring content and alignment. An example of this is a content matrix shown below.
Table 2.1: Content matrix for cognitive demand and topics (Porter, 2002:4).
Topic
Category of cognitive demand Memorise Perform
procedures
Communicate understanding
Solve nonroutine problems
Conjecture/
generalise/
prove Multiple-step
equations Inequalities Linear equations Lines/slopes and intersect Operations on polynomials Quadratic equations
To calculate the alignment index, matrices are formed which match assessments and curriculum in cells and the alignment index is calculated per cell.
The items are assigned in the cells and, once finished, alignment is calculated where:
Alignment index = 1 - ∑ │𝑋−𝑌│
2 where x and y stand for proportion in cell i for documents
35
x and y respectively. The alignment index ranges from 0 to 1, 0-0.50 range depict no alignment to moderate alignment, 0.51 to 1.0 range from moderate to perfect alignment (Ndlovu & Mji, 2012; Porter, 2002; Porter, McMaken, Hwang & Yang, 2011).