CHAPTER III: RESEARCH METHOD
C. Data Collection Techniques and Instrument
The researcher used test to collect data namely vocabulary test and speaking test. Vocabulary test was used to determine the students’ vocabulary mastery. Meanwhile speaking test was used to determine the students' speaking ability. The blue print of instrument is available in appendix 2.
1. The instrument of collecting data a) Vocabulary test
The researcher conducted the written test to measure the students’ vocabulary mastery. The indicators of vocabulary test were noun, verb, adjective, and adverb. The topics were about procedure text for making food and dessert. The test was presented in 20 questions of multiple choices. The students were given the written test and doing it individually with time allotment was 50 minutes. For the criteria of assessment, each item got 5 points for the correct answer and 0 point for the wrong answer. So, to get students’ total score, the correct answer was multiplied by five. The vocabulary instrument can be seen in appendix 3.
b) Speaking test
In this study, the researcher used speaking test to measure the students' speaking ability. First, students had to choose one of topics provided. Then, the researcher gave students about 15 minutes to prepare some points that will be conveyed when speak in front of class. After that, the researcher called the students one by one to explain the procedure based on the topic they chose in front of the class at least 6 sentences in 3 minutes. The speaking instrument can be seen in appendix 4.
The researcher applied inter-rater to determine test scores: the researcher and the English teacher. Inter-rater reliability is the level of
consistency or accuracy when two or more raters are observing and measuring data on the same observed scenario60. In this study, the researcher used inter-rater to eliminate subjectivity in assessed students’ speaking ability. The researcher assessed the students' speaking ability based on three aspects which include: vocabulary, pronunciation, and fluency with scoring. For criteria of assessment, the highest score for each aspect was five. So, to get students’ total score was total students’ speaking score was multiplied by 100 then it was divided by 15.
Table 3.2
Speaking rubric adapted by H. Douglas Brown (2001)
Score Aspect
Vocabulary Fluency Pronunciation
1 2 3 4
1 Vocabulary used is so limited
Speak haltingly with long pauses
Error pronunciation frequently but still can be understood 2 Has enough vocabulary
but there are many convoluted words
Speak confidently but it pauses frequently and still halting
Pronunciation is clear although often quite
3 Has enough vocabulary but there are some convoluted words
Speak with a little halting and paused for a while
Error pronunciation is frequent but it does not disturb the meaning
4 Using variety vocabulary but there is inappropriate vocabulary
Speak fluently with little hesitation
Error pronunciation is quite rare
5 Using variety and appropriate vocabulary and sentences are easy to understand
Speak fluently without any pauses and hesitation
Pronunciation is very good and it is easy to understand
60 Lodico, Spaulding, and Voegtle, 114.
In arranging the vocabulary and speaking test, the researcher did several steps as follows:
a. Decided the appropriate topic based on curriculum basic competence, namely procedure text of making food and dessert.
b. Next, decided the questions which did not make students too difficult to answer and considered the language structure according to English grammar.
c. Then, decide the appropriate time allotment for the tests, namely 50 minutes for vocabulary test, and 3 minutes for each student’s performance in speaking test.
d. After that, asked the validation of the instrument to the expert judgment.
e. After the instrument was approved by expert, the instrument used to test the students.
2. Instrument Testing
Instruments that meet standards for validity, reliability, and practicality are required for testing in order to produce good and accurate results. As a result, it must be taken into consideration and guaranteed that the instrument is dependable and valid. Therefore, it is needed to test the validity, reliability, and practicality of the instruments.
a. Instrument Validity Test
In order to determine the validity of the variables, the researcher employed content validity and construct validity. According to
Brown, it could be claimed content validity if a test actually samples the subject matter that has been studied61. Meanwhile, construct validity is used to test the construction of the aspects to be measured based on certain theories62.
To obtain the content validity of the vocabulary test and speaking test, the researcher arranged the tests based on the material in the curriculum basic competence used for the eleventh grade students of SMKN 3 Jember. The basic competence namely by the end of Phase F, students use spoken and written texts and visuals in English to communicate accordingly with the situation, purpose, and the audience/reader. Various types of texts such as narratives, descriptives, expositions, procedures, arguments, discussions, and authentic texts are the main references in learn English in this phase.
Meanwhile to get construct validity, the instrument was arranged to measure students' vocabulary mastery and speaking ability based on certain theory. To test the content and construct validity, the researcher used expert judgment. In this case, after the instruments have been constructed based on certain theory and have been arranged based on the material in the syllabus, then those were consulted with experts. If there were some errors, the researcher would revise it.
61 Brown, Language Assessment Principles and Classroom Practices, 22.
62 Sugiyono, Metode Penelitian Kuantitatif, Kualitatif, dan R&D, (Bandung: ALFABETA), 125.
1) Vocabulary Test
The validators were two English lecturers with expertise in vocabulary. In this research, the validators were given the vocabulary test sheet to ensure its validity. From the validator 1 there was no revision for this instrument, meanwhile from the validator 2 there were some notes to be revised as follows:
The distractors should be added one more namely point E.
The distractors from questions number 2 and 18 should be changed because those were not verbs or phrases but those were sentences. It can be chosen between change the distractors or change the questions.
Then the researcher directly revised it. After revised, the instrument was ready to be tried out. The result of validity vocabulary test can be seen in appendix 5.
2) Speaking Test
To ensure the validity of the speaking test, the researcher used expert judgment. The validators were two English lecturers with expertise in speaking. They were given the instrument to measure the validity of the speaking test. There was revision from the validator 1 namely for the speaking rubric should be added some notes as the description of the score. Meanwhile from the validator 2, the note was the question of speaking test should be moved to the instruction. After that, the researcher
revised the instrument before used it for try out. The result of validity speaking test can be seen in appendix 6.
b. Instrument Reliability Test
Reliability test is the consistency of scores attained on different occasions. An instrument can be said to be reliable if the test conducted on two different occasion yield similar result. To test the reliability of the instrument, the researcher conducted a try out on the sample. The researcher chose 10 students randomly from each class as the try out sample. Before conduct the try out, the researcher asked permission to the English teacher and asked the available class. After found the available class, the try out sample was gathered. Then, the researcher gave the test instructions. After that the students did the try out based on the specified time allocation.
1) Vocabulary Test
In this research, the researcher used Cronbach’s Alpha by SPSS 26 version to determine the reliability of the instrument. In this case, to find out the results of benchmarks in interpreting the degree of instrument reliability, it can be done based on several criteria, namely:63
Table 3.3
Category of Reliability
Value Category
> 0.90 Very highly reliable
0.80-0.90 Highly reliable
63 Louis Cohen, Lawrence Manion, and Keith Morrison, Research Methods in Education 8th Edition, (New York: Routledge, 2018), 774.
0.70-0.79 Reliable
0.60-0.69 Minimally reliable
<0.60 Unacceptably low reliability On Wednesday, February 8 2023, the researcher conducted vocabulary try out test at 8.30 am. The scores can be seen in appendix 7. The result of SPSS output was:
Table 3.4 Reliability Statistics Cronbach's
Alpha N of Items
.713 20
Based on the SPSS output above, Cronbach’s Alpha value was 0.713. It can be concluded that the instrument was reliable.
b) Speaking Test
Inter-rater reliability used in this study by the researcher to provide an overview of the level of consensus agreement that the expert will provide in the form of score. In this study, the researcher conducted speaking test to students in order to obtain data about their speaking ability. The researcher used scoring rubric and it was helped by Cohen Kappa with SPSS 26. The categories between the levels of reliability between raters are:64
64 Mary L McHugh, ―Lessons in Biostatistics Interrater Reliability : The Kappa Statistic,‖
Biochemica Medica 22, no. 3 (2012): 276–282, https://hrcak.srce.hr/89395.
Table 3.5
Interpretation of Cohen’s Kappa
. On Wednesday, February 8, 2023 the researcher conducted speaking try out test at 9.30 am. The try out score from two raters can be seen in appendix 8. The result of SPSS output as follows:
Table 3.6
Kappa Reliability Test Symmetric Measures
Value
Asymptotic Standard
Errora
Approximate Tb
Approximate Significance Measure of
Agreement
Kappa .600 .180 3.530 .000
N of Valid Cases 10 a. Not assuming the null hypothesis.
b. Using the asymptotic standard error assuming the null hypothesis.
From the table above, it showed that Cohen’s Kappa value was 0.600. Based on the interpretation of Cohen’s Kappa it was categorized as moderate of agreement. The result considered adequate and further testing was not required.
c. Instrument Practicality Test
In this research, to decide whether or not the test is practical, the researcher observed the students when doing try out. The researcher used observation sheet which consists of students’ response toward the instructions, students’ response toward the questions, estimated time allocation for students, and students’ enthusiasm. Before the observation sheet was used, it was given to the expert judgment first.
From the first validator, there was no revision, meanwhile from the second validator, the aspect should be added one more namely add the aspect of students’ response to the questions. The result of validation can be seen in appendix 9.
1) Vocabulary Test
The vocabulary test can be said to be practical if students can do try out in a simple way. The instructions and questions do not make students confused and time allotment is appropriate for students.
The observation result when students were doing try out were none of students asked about the instructions. From 10 students, there were only two students asked about the questions number 4 and 12. It can be concluded that the instructions and questions did not make students confused and easily to be understood. For time allocation, all students finished the test on time and all students were enthusiastic doing the test. As the result,
the score of each aspect was 3 points. It indicated that the test was practical for students. The practicality result can be seen in appendix 10.
2) Speaking test
The speaking test can be said to be practical if students do the test easily without any difficulties, the instructions do not make students confused and time allotment is appropriate for students.
The observation result when students were doing try out were none of students asked about the question. From 10 students, there were three students asked about the instructions. It can be concluded that the instructions and questions did not make students confused and easily to be understood. For time allocation, there were two students who have not finished preparing some points that will be conveyed when speak and there were three students felt nervous doing the test. As the result, the score of each aspect was 3 points. It showed that the test was practical for students. The practicality result can be seen in appendix 11.