Chapter 3: Methodology
3.3 Study Instruments
3.3.2 The Mathematics Literacy Test (MLT)
The last point was about the order of the lessons, the experts suggested teaching the lessons about “quantity” just after “change and relationship” not after “space and shape”. This was because “quantity” might include concepts that are considered basics to other content areas. The experts’ suggestion was met, and the order of lessons was changed as they said.
3.3.2.1 The Mathematics Literacy Test (MLT) Components
Table 6: A map for selected mathematics items in MLT
Part 1: Q1-Q26 Problem Solving
Item number Item name Item difficulty Level of Proficiency
Process Content Context
Formulate Employ Interpret Change and relationships Quantity Space and shape Uncertainty and data Personal Occupational Scientific Societal
1 Charts Q1 * 347.7 BL1
2 Charts Q2 415.0 L1
3 Charts Q5 428.2 L2
4 Which Car? Q1 * 327.8 BL1
5 Which Car? Q2 490.9 L3
6 Which Car? Q3 552.6 L4
7 Garage Q1 419.6 L1
8 Garage Q2.1 663.2 L5
9 Apartment Purchase Q1 576.2 L4
10 Drip Rate Q1 610.0 L5
11 Drip Rate Q3 631.7 L5
12 REVOLVING DOOR Q1 512.3 L3 13 REVOLVING DOOR Q2 840.3 L6 14 REVOLVING DOOR Q3 561.3 L4
15 Sauce Q2 489.1 L3
16 Sailing Ships Q1 511.7 L3 17 Sailing Ships Q3 538.5 L3 18 Sailing Ships Q4 702.1 L6 19 Climbing Mount Fuji Q1 464.0 L2 20 Climbing Mount Fuji Q2 641.6 L5 21 Climbing Mount Fuji Q3 591.3 L4 22 Helen the Cyclist Q1 440.5 L2 23 Helen the Cyclist Q2 510.6 L3 24 Helen the Cyclist (E) Q3 696.6 L6 25 FERRIS WHEEL Q1 592.3 L4
26 FERRIS WHEEL Q2 481.0 L3
Total 26 8 14 4 7 7 8 4 8 4 6 8
Percentage % 100 31 54 15 27 27 31 15 31 15 23 31
Part 2: Q27-Q34 8 Reasoning
*BL1 means below level one
Source: OECD (2014), PISA 2012 Technical Report, OECD, Paris.
The above Table 6 demonstrated the map for selected mathematics items used in this study. This MLT was a PISA-style test where test items released from PISA 2012 were used to better compare with previous results. Those chosen items were spread across all six proficiency levels identified by PISA 2012 of three types of response format (multiple-choice, closed constructed, and open constructed response).
Furthermore, the items were spread across the four content subdomains (quantity, space & shape, change & relationships, and uncertainty & data), and all four PISA contexts (personal, public, educational/public, and scientific). Finally, the three processes (formulate, employ, and interpret) that they had to be activated in order to connect the real world were included.
The test problems primarily attempted to assess the students’ problem-solving in six proficiency levels that were presented in 26 of the test problems in addition to 8 problems that measured their reasoning skills. Moreover, Table 7 describes the distribution of test items based on the dimensions of the PISA framework for the assessment of mathematics
Table 7: Distribution of MLT items by dimensions of the PISA framework
Processes # Contents # Contexts #
Formulate 8 Quantity 7 Personal 8
Employ 14 Space and shape 8 Occupational 4
Interpret 4 Change and
relationship
7 Scientific 6
Reasoning * 8 Uncertainty 4 Societal 8
Total 26 +
8Reasoning
Total 26 Total 26
*Reasoning is the new addition to the PISA 2021 framework of mathematical literacy
In PISA, each question was assigned a difficulty level. Using Item Response theory and these difficulty levels the raw scores of the students were converted to a score on the PISA scale. The PISA scale in mathematics was also divided into six
mathematical literacy levels to represent degrees of proficiency where level six was the highest. The mathematics proficiency levels are detailed in Figure 10 below.
Source: OECD (2014), PISA 2012 Technical Report, OECD, Paris.
The results of PISA 2018 indicated that 45.5% of the students performed below level 2, and only about 5.4% performed above level (OECD, 2019b). This indicated that if the students’ performance improved in levels 2, 3, and 4 was crucial to the overall improvement in ML. In addition, this would be a small starting step towards the goal of being among the best 20 countries in the world and fulfilling the gifted needs too.
Figure 10: Descriptions for the six levels of proficiency in mathematics
The items of the MLT were distributed to cover all the six proficiency levels of mathematics problems as presented in Table 8 as follows:
Table 8: Distribution of MLT items by levels of proficiency Level of proficiency # of items Percentage %
Level 1 and Below 4 15
Level 2 3 12
Level 3 7 27
Level 4 5 19
Level 5 4 15
Level 6 3 12
Total 26 100
These “proficiency levels” described what students at given levels of proficiency typically know and can do. Where students were distributed between the first level, in which students succeeded only in basic tasks, and the sixth level, in which students could solve complex problems and had advanced thinking skills.
It is noteworthy that the Cronbach alpha test was performed to compute to find the reliability of the administered mathematical literacy test and found to be 0.85.
3.3.2.2 The Mathematics Literacy Test Appropriateness
As mentioned earlier, PISA questions are carefully reviewed and only those that have no cultural bias are used. “They also have strong measurement properties, and place an emphasis on authenticity and educational validity” (OECD, 2009b, 2013).
This is a result of the PISA procedure to select test items. Thus, the results of PISA have a high degree of validity and reliability (OECD, 2009b, 2013). Moreover, for this study, the questions were presented to the same committee that reviewed the MEP to decide about the test appropriateness to measure the students’ level of ML. They reported that the test was comprehensive and is appropriate to measure ML that was based on both the MEP and PISA framework. They suggested reordering the questions
so that the test starts from easy to hard questions. The researcher reordered the beginning of the test but did not order it fully from level 1 to level 6 because the questions were on the unit form that means two or more questions might be related to measuring the same context. Another reason, that the students might just skip doing all the test if they recognized that the rest questions were of high level.
Experts initially indicated that students would need more than 90 minutes for 34 PISA problems. However, although all math questions in PISA involved some real- world mathematical problem solving, not all students were required to take a complete problem-solving course due to the limited time students had to answer the questions:
“the average allowable response time for each question is around two minutes, which is too short a period of time for students to go through the whole problem-solving cycle.” (OECD, 2009a, p. 160). Thus, bearing in mind that students should, as far as possible, undergo conditions similar to the original PISA test conditions, the committee decided that the test time was appropriate.