Chapter 3: Methodology
3.6 Research Instruments
3.6.1 Newton’s Second Law of Motion Achievement Test
The test included two-tire multiple choice questions that assess students’
performance in NSLOM at both conceptual and procedural levels. A test was developed in such a way to be suitable and comprehensible for the students.
According to Biggs (1996), multiple-choice test critics claim that it only tests factual knowledge, and not conceptual understanding. As, it focuses on low order thinking skills. One possible solution is to use a two-tier Multiple Choice Question (MCQ). According to Mann and Treagust (2000), this question aims to help students and teachers understand students’ struggles so that they can correct any misunderstandings or difficult areas and deepen their understanding of the topic.
The two-tier MCQ format is similar to the traditional MCQ format, but as the name suggests, it contains a second tier of questions related to the main question. The first tier of the question usually relates to a statement of knowledge. However, the second level of the question is suitable for testing students’ ability to learn and not only remember, but also requires a higher level of thinking (Williams, 2006).
The two-tier MCQs was developed following the procedure used by previous researchers in this field, namely, Treagust (1985, 1998). The development process was divided into three stages and ten steps, see Figure 13. In the first stage, the scope of the research content is defined. The second stage involves identifying student ideas when using multi-level notation (based on the responses of a group of 60 students) to describe and explain NSLOM. Stage 3 includes several steps in designing the test project and reviewing the final version of the two- tier MCQs diagnostic tool.
Figure 13: Scheme of the Development Process of a Two-Tier Diagnostic Test Based on Treagust (1988)
Define the content
• Identifying propositional knowledge statements
• Developing a concepts maps
• Relate propositional knowledge to the concept map.
• Content validation
Obtaining information about
students' misunderstanding
• Check the relevant literature.
• Conduct unstructured interviews with students
• Freely develop multiple options for content items
Development and validation
of diagnostic tests
• Development of two-tier items
• Design a specific grid
• Continuous improvement
3.6.1.1 Phase One: Define the Content Area of NSLOM
The first stage of MCQ development involves the design and development of a two- tier MCQ. These questions were specially developed for 11th- grade students at NSLOM in the UAE. NSLOM was chosen after an extensive literature review, reporting that both local and international students have learning difficulties and misunderstandings on these specific topics (Antwi, 2013). In this stage, first, the content boundaries and learning goals were determined according to the purpose of the study. The following educational objectives related to the field of study are recorded:
• Students can explain the types of Forces.
• Students can distinguish between Weight and Mass.
• Students can calculate net force.
• Students can determine and calculate the Normal Force.
• Students can explain Free-Body Diagrams.
• Students can explain Newton’s First Law.
• Students can explain Newton’s Second Law.
• Students can explain Newton’s Third Law.
• Students can interpret useful ropes and Pulleys.
• Students can determine and calculate problems related to Two Books Connected on a Table.
• Students can interpret effective Kinetic Friction.
• Students can interpret effective Static Friction.
• Students can determine and calculate problems related to Two Blocks Connected by a Rope—with Friction.
Then ten concept maps were drawn for each topic, and one for each concept included in this study. It also examines the relationship between theoretical knowledge and concept maps. All concept maps were validated by six physics teachers who also participated in teaching physics for the 11th grade. Test development is based on the final version of the proposed knowledge statement and concept map.
3.6.1.2 Phase Two: Identifying Students’ Conceptions when Describing and Explaining NSLOM
A significant feature of this stage involved defining student conceptions when describing and explaining NSLOM. In addition to a review of the research literature (Antwi, 2013), written answers (including semi-structured and freely answered questions) were used in 17 exercises to identify students’ ideas (see Appendix A). The researcher conducted these exercises with 60 students in grade 11 and designed responses for grade 11 physics students’ distracters factors based on literature reviews and the reasons presented in the interviews. Distracters factors include alternative conceptions, and they were created from this list. These alternative conceptions were then used to create a two-tier MCQs diagnostic test, where the questions included an introduction and four alternatives, one of which is correct answer. These questions were then evaluated by a group of physics teachers and supervisors of the physics department at a major district (zone) in Abu Dhabi.
The test was then prepared in its preliminary form, as it included 30 items. It was presented to a group of physics teachers and supervisors. The appropriate adjustments were made based on their opinions and observations. The following observations listed incorporate:
1. Reduce the number of test items to be less than 30.
2. Delete some items which required a long time to answer, as well as exclude items which their level exceeded the UAE 11th grade students.
3. Delete some items that require advanced mathematical skills.
4. Some educators also referred to the amendment of the wording of some paragraphs and alternatives to make them clearer.
Following this initial evaluation, questions were asked and considered appropriate for the study. The revised version of the test is shown in Appendix (B).
3.6.1.3 Phase 3: Development and Validation of NSLMAT
In this stage, and based on literature review and interviews, the two-tier diagnostic tests have been developed, where the first tier is a traditional multiple- choice step, and the second tier is a possible reason to give a specific answer to the first tier (Haladyna & Rodriguez, 2013). The second level contains four answers, one of which is the expected answer. Distractors included high frequencies of wrong reasons and scientifically unacceptable conceptions held by grade 11 physics students.
3.6.1.4 NSLMAT Validation
With regard to the validity of the content, the test was presented to a group of educators, consisting of 10 experts specialized in physics and physics education, consisting of inspectors and high school teachers, as well as to a group of university professors, (see Appendix C). The experts were asked to ensure that the test was content valid in terms of formulation of phrases and its language, scientific accuracy, comprehensiveness of the measuring tool, and its suitability for the goal for which it was developed. Additionally, the experts were asked to judge the clarity of the
paragraphs, and their suitability for the participants of the study, see Appendix (B).
The researcher obtained some opinions and suggestions from the judges, see Appendix (E).
In the light of these observation received, the researcher modified the initial form by rewriting some of the phrases, other phrases were excluded. Besides, the symbols rewritten in ways that are similar to one presented in the book published in the UAE for advanced 11th graders in 2019, and after all the changes made, the final image of the scale consisting of 16 questions can be found in Appendix (D). The questions include an introduction and four alternatives, one of which is correct.
Therefore, students must choose an answer from symbolic options A, B, C, and D.
While the justification for the answer choices from one of the options numbers 1, 2, 3, and 4. The focus of the test items was on: (1) conceptual understanding, (2) and procedural understanding. Table 4 shows the distributions of items on both conceptual and procedural domains.
Table 4: Distribution of Items in the Final Version of NSLMAT
Knowledge domains Items
Procedural knowledge Q2, Q4, Q6, Q7, Q9, Q10, Q11, and Q13 Conceptual Knowledge Q1, Q3, Q5, Q8, Q12, Q14, Q15, and Q16
3.6.1.5 NSLMAT Reliability
Split-half reliability was calculated to assess the internal consistency of this NSLMAT by comparing the results of the odd numbers with the results from the even numbers as illustrated in Tables 5 and 6, respectively.
Table 5: Reliability Statistics for NSLMAT Exam
Cronbach's Alpha
Part 1 Value 0.741
N of Items 16a
Part 2 Value 0.844
N of Items 16b Total N of Items 32
Correlation Between Forms 0.611
Spearman-Brown Coefficient Equal Length 0.759 Unequal Length 0.759 Guttman Split-Half Coefficient 0.758
Table 6: Scale Statistics of NALMAT Mean Variance Std.
Deviation N of Items
Part 1 13.73 3.857 1.964 15a
Part 2 12.03 4.309 2.076 13b
Both Parts 25.77 13.151 3.626 28
Because the reliability and the variance of the two parts were not equals, the researcher was adopted Guttman split-half reliability coefficient, and it was found 0.758. This means that the scores are acceptable in terms of reliability since it was between -1 and 1.
The internal consistency validity of the test for conceptual and procedural domains, as well as the test items, were also calculated as shown in the Tables 7, 8, and 9.
It is evident from Table 7 and Table 8 that most of the test items in its conceptual and procedural dimensions are associated with the overall score, with a statistically significant correlation at the level of significance 0.01, except for questions 4 and 12, and this indicates that the internal consistency characterizes the test.
Table 7: Correlation Coefficients Between Each Domain with Overall Score of Test Knowledge
domain Coefficient of correlation with the total
score Significance
level
Conceptual
Question 0.759** 0.01
Answer 0.561** 0.01
Total 0.796** 0.01
Procedural
Question 0.952** 0.01
Answer 0.896** 0.01
Total 0.944** 0.01
**. Correlation is significant at the 0.01 level (2-tailed).
Table 8: Correlation Coefficients Between Questions and Overall Score of the Conceptual Domain
Question number Coefficient of
correlation Significance level
2 Question 0.829**
0.01
Answer 0.788
4 Question 0.103
Not significant
Answer 0.463
6 Question 0.793**
0.01
Answer 0.703
7 Question 0.829**
0.01
Answer 0.834
9 Question 0.712**
0.01
Answer 0.827
10 Question 0.671**
0.01
Answer 0.745
11 Question 0.955**
0.01
Answer 0.885
13 Question 0.612**
0.01
Answer 0.755
Table 9: Correlation Coefficients Between Questions and Overall Score of the Procedural Domain
**. Correlation is significant at the 0.01 level (2-tailed).