ISSN (print): 2579-8944; ISSN (online): 2579-9037
Avaliable online at: http://e-journal.unipma.ac.id/index.php/linguista 86
Communicative Language Ability and HOT or LOT of English National Final Examination Test Items
Anni Marhamah, dan Muhammad Ikbal
Sekolah Tinggi Agama Islam Negeri Mandailing Natal, Panyabungan, Indonesia e-mail: *[email protected]; [email protected]
Abstract
This study aims to observe and analyze English national final examination test items for senior high school. The study identifies and determines the variation of the test items’ level of questions, analyzes the number of higher order thinking and lower order thinking of the test items and analyzes the communicative language ability that exist in the tests items. The study was conducted through qualitative research focusing on the content analysis method. The data source used in this study was document of English National Examination including the transcript of the listening tests. The data taken from document of English National Examination were all English test items of UN for SMA 2014. The writer took documents of English UN for SMA including the cassette of its listening section from SMA Negeri 1 Rao, Pasaman Regency West Sumatera. The test items were observed, classified, analyzed, and then the results were interpreted and drew the conclusion. The result shows that analyzing the test items document is due to the levels of questions and communicative language ability, after all the types are identified and written down, they are used as the basis to make inferences and draw the conclusion.
Keywords: Qualitative Research; National Examination; Communicative Language Ability; HOT and LOT.
Introduction
The quality of education is still become an education problem in Indonesia. The quality of education is still in improving process. This existed in national education policy, to increase and improve the education quality and relevancy (Misi Menteri Pendidikan Nasional, 2011). Government creates many policies of education strategy in order to increase and improve the education quality and the student’s outcome. One of the government strategies is National Examination.
National examination, well known as Ujian Nasional (UN) is used to assess graduate competences nationally. National examination is a measurement at the end of learning process and as the key of student’s graduation from school that conducted in every year. It is used to measure and determine the education grade in Indonesia and act as one of government’s indicators in order to create clever and competence students. English is one of the subjects that are tested in national examination.
National Examination is a test to measure and evaluate the students' competence nationally by the central government after the process of teaching and learning (Peraturan Menteri Pendidikan Tahun 2005:1). Having national examination as the evaluation program means that national examination should be carefully designed so that those who pass the national examination can represent the improvement of national education which further will create intelligent and competitive graduates.
English national examination test items consist of fifty questions including fifteen items of listening section questions.
As an international language, English is considered as a very important language to be taught in Indonesia. Learning English in Indonesia is different from the other country since English is a foreign language in Indonesia. This condition affects the students’ ability to learn and comprehend the language itself. English as a subject taught at school and college deal with the students’ understanding and use of language. People in this world, today learning English as foreign language and it continues to spread, not only as the global language but also as the language of science, technology, and advanced research. It shows that English is very important now days, especially for students. They are expected to study English more.
The target forces Indonesian government to improve education quality. One of the strategies used to improve education quality by the government is making an evaluation program through national examination. Having national examination as the evaluation program means that national examination should be carefully designed so that those who pass the national examination can represent the improvement of national education which further will create intelligent and competitive graduates.
In relation to improve national education to create intelligent and competitive graduates, assessment technique which is appropriate to the aims of the curriculum and used to improve students thinking level should be designed and implemented. The high level questions should be designed in national examination in order that students who can pass national examination are those who can utilize their high order thinking level. Governments should consider the level of questions that should be exist in English national examination test items in order to get the best result from the students itself. Providing the test items that relates to their learning materials and based on the students’ Bloom Taxonomy will help the students to be able in answering the test in national examination.
Bloom (1956) promotes higher forms of thinking in education, such as analyzing and evaluating concepts, processes, procedures, and principles, rather than just remembering facts (rote learning). Bloom’s taxonomy is a multi-tiered model of classifying thinking according to six cognitive levels of complexity. Bloom's taxonomy is a set of three hierarchical models used to classify educational learning objectives into levels of complexity and specificity. The three lists cover the learning objectives in cognitive, affective and sensory domains. The cognitive domain list has been the primary focus of most traditional education and is frequently used to structure curriculum learning objectives, assessments and activities. Zaim (2016) explain three domains in order to master science knowledge relates to the taxonomy bloom. They are cognitive domain, affective domain and psychomotor domain. Zaim (2016) clarifies the revision of original Bloom’s Taxonomy including to six categories; remembering, understanding, applying, analysis, evaluation and creation.
To complete the analysis of the provided test items, researcher also will analyze communicative language ability of the test items. Communicative language ability (CLA) can be described as consisting of both knowledge, or competence, and the capacity for implementing, or executing that competence in appropriate, contextualized communicative language use. Hymes (1972b) describes language use as: The performance of a person is not identical with a behavioral record. It takes into account the interaction between competences (knowledge, ability for use), the competence of others, and the cybernetic and emergent properties of events themselves. (Hymes 1972b:283). Communication takes place in an infinite variety of situations, and success in a particular role depends on one’s understanding of the context and on prior experience of a similar kind.
Canale and Swain (1980), examining the theoretical bases of language teaching and language testing, distinguish ‘grammatical competence’, which includes lexis, morphology, sentence-grammar semantics, and phonology, from ‘sociolinguistic
competence’, which consists of socio-cultural rules and ‘discourse competence’, rules of discourse (cohesion and coherence).
Communicative language ability is consists of knowledge structures which is knowledge of the world (how we know everything in the world is a knowledge) and language competence which is knowledge of language (how we present and explain knowledge we have by using language as the communicating way), how we deliver knowledge by use language is supported by strategic competence and psycho- physiological mechanisms that based on the context of situation.
There are several studies toward English National Examination tests items including in the term of higher order Thinking and Lower order thinking skills. Widana (2017) conducted a research to provide knowledge and understanding to the teachers about the concept and characteristics of the HOTS (Higher Order Thinking Skills) assessment extensively and deep; and improve the skills of the teachers to develop HOTS assessment. Ahmad (2016) conducted a research to analyze which levels of the Barrett taxonomy were more reflected in ENE items of 2013/2014 academic year and whether the proportions of items among the twenty test packages in the ENE assessing students’ Lower Order Thinking Skills (LOTS) and Higher Order Thinking Skills (HOTS) are consistent. The results indicated that questions asking LOTS still prevailed in ENE items. Of all the twenty test packages, the items categorized into literal level represented around 68.6% of the total number of the questions. The results also showed the complete absence of Appreciation the highest level of thinking in the mentioned taxonomy. It is obvious that there is a shortage of items questioning students’ HOTS in the exam and they are not well-treated. Accordingly, this finding reveals that there is still much room for ENE to be the driving force in the effort to make learners critical thinkers. In the light of these data, this study recommends modifying the English National Exam by providing them with more question items that include HOTS.
Most of the studies only focus on one or some aspects of the taxonomy bloom.
There is no analysis of communicative language ability. Thus, study about analysis of HOTS and LOTS and Communicative Language Ability towards English national examination tests items is really needed in order to do evaluation and make sure that the provided test items are already good and relevant with the goals of education.
Referring to those explanations, this research aims to observe and analyze English national final examination test items for senior high school in terms of the level of questions (HOT or LOT) and communicative language ability. This research aims to see the variation of the test items’ level of questions, analyze the number of higher Order thinking and lower Order thinking of the test items and analyze the communicative language ability that exist in the tests items.
Methods
This study was conducted through qualitative research focusing on the content analysis method. A qualitative method referred to the research procedures which produce descriptive data such as people’s own written or spoken words and observable behavior (Bodgan and Taylor, 2002). It was done by analyzing documents (texts, images, film or music) or similar traces of experiences or interactions. This study was done by analyzing English National Examination at year 2014 document including the cassette for listening section that had been transcribed to investigate the level of questions and the communicative language ability of the English test items.
The data source used in this study was document of English National Examination including the cassettes of listening section that had been transcribed (the transcript of the listening tests). Document of English national examination 2014 is used as the data in this research. The data taken from document of English National Examination were all English test items of UN for SMA 2014.
Data Collection Procedure of all English test items of National examination for SMA year 2014, the writer took documents of English UN for SMA including the cassette of its listening section from SMA Negeri 1 Rao, Pasaman Regency West Sumatera. Then, the writer transcribed the listening section. The writer picked up fifty questions of English UN for SMA year 2014 then tabulated them. We collected the data by gathering test items that were found in the documents, then read and observed each test items carefully to find out whether the questions belong to HOT or LOT types, next classified and identified the data according to their features and finally identified the communicative language ability of the test items.
The data were analyzed through several steps. The test items was analyzed through the following procedures: analyzing the data by categorizing and counting the frequencies of occurrences of the level of questions types, writing down the frequency of occurrence of each type in the data sheets, identifying the communicative language ability of the test items, after all the types were identified and written down, they were used as the basis to make inferences and draw the conclusion.
Findings and Disscussion
1. Level of Questions (HOT or LOT) of Test Items
English National Examination of Senior High School documents year 2014 consist of fifty test items which is 15 test items belongs to Listening test. Based on the document analysis that has been done by We, it is found that there are eleven test items that belong to Remembering with percentage 22%, twenty two test items are belong to Understanding with percentage 44%, thirteen test items are belong to Applying with percentage 26%, three test items are belong to Analysis with percentage 6%, one test item is belong to Creating with percentage 2% and there is no test item that belong to Evaluating. These percentages are the result of the number of test items found divided the total number of the test items and times 100 %. It is explained in details below.
a. Remembering
It consisted of eleven test items with percentage 22%. Numbers of those test items in the document are no. 3,13,14,18,21,25,28,30,33,37 and 41.
b. Understanding
It consisted of twentytwo test items with percentage 44%. Numbers of those test items in the document are no. 1, 2, 7, 8, 9, 10, 11, 12, 16, 19, 20, 22, 23, 24, 26, 31, 34, 36, 39, 40, 43 and 45.
c. Applying
It consisted of thirteen test items with percentage 26%. Number of those test items in the document are no. 4,5,6,15,17,27,29,32,35,38,44,46 and 47.
d. Analysis
It consisted of three test items with percentage 6%. Numbers of those test items in the document are no. 48, 49 and 50.
e. Creating
It consisted of only one test item with percentage 2%. Number of the test item in the document is no. 45.
f. Evaluating
There is no test item belong to evaluating.
To make it clear, We provided the data into a table 1.
Table 1. Analysis of Cognitive Domain No.
test items
Remembering (C1)
Understanding (C2)
Applying (C3)
Analysis (C4)
Evaluating (C5)
Creating (C6)
1 V
2 V
3 V
4 V
5 V
6 V
7 V
8 V
9 V
10 V
11 V
12 V
13 V
14 V
15 V
16 V
17 V
18 V
19 V
20 V
21 V
22 V
23 V
24 V
25 V
26 V
27 V
28 V
29 V
30 V
31 V
32 V
33 V
34 V
35 V
36 V
37 V
38 V
39 V
40 V
41 V
42 V
43 V
44 V
45 V
46 V
47 V
48 V
49 V
50 V
Total 11 22 13 3 - 1
Percen tage
22% 4% 26% 6% - 2%
2. Analysis of Lower Order Thinking (LOT) and Higher Order Thinking (HOT) We analyzed the documents by identifying the verbs used in the questions of test items using the Revised Bloom’s Taxonomy Action Verbs (see Appendix C). The
first three levels (Remembering, Understanding and Applying) are called as Lower Order Thinking (LOT) and the others three levels (Analyzing, Evaluating and creating) are called as the Higher Order Thinking (HOT). From the content analysis, it is found that there are 46 test items that belong to lower level questions and showed the lower order thinking (LOT) with percentage 92%. Beside of it, there are only 4 test items that belong to higher level questions and showed the higher order thinking (HOT) with percentage 8%. Most of the test items are belong to LOT than HOT. This is related to the results of study by Putra and Abdullah (2019) entitled Higher-Order Thinking Skill (Hots) Questions in English National Examination in Indonesia. They examine one package of each English National Examination from 2013 until 2018. They analyze 210 multiple-choice items in which each examination contain 35 items of reading comprehension. The items are analyzed quantitatively through content analysis based on the aspects of HOTS in Revised Bloom’s Taxonomy. We found that there is insufficient amount of HOTS questions in English National Examination. 157 items classified into the LOTS and only 53 (25.23%) items are classified into HOTS. The second finding is that the level of HOTS included in English National Examination 2013-2018 is only the level of Analyze. The Differentiating and Organizing are the sub skills of the aspect of Analyze that are mostly included in all examinations. There is strong evidence for encouraging the test developers to provide adequate portions of HOTS-based items in English National Examination.
Lower Order Thinking (LOT)
There are 46 test items that belong to lower level questions and showed the lower order thinking (LOT) with percentage 92%. These 46 test items are divided into three types.
a. Remembering
11 test items are belong to remembering, since the test items asked the question that had the explicit answer in the text itself, it only asked the test takers to identifying, mentioning, showing, give names and provide definitions of the data based on the existing text. All of the answers are existed in the text. As Zaim (2016:33) said that all of these operational verb are used to determine the test based on the remembering level.
b. Understanding
22 test items belong to Understanding, since the test items asked the questions that had the implicit answer in the providing text. It asked the test takers to explaining, concluding, translating, getting conclusion, developing, summarizing and predicting what will happen based on the providing text. All of the answers are implicitly existed in the text, thus the test takers should understood the text first before answering the questions. As Zaim (2016:33) said that all of these operational verb are used to determine the test based on the understanding level.
c. Applying
13 test items belong to Applying, since the test items asked the question and asked the test takers to demonstrating, counting, relating, proving, resulting, showing, finding, completing and providing answer based on the providing text, such as providing synonym, antonym, what and who refers to whom, etc. As Zaim (2016:33) said that all of these operational verb are used to determine the test based on the applying level.
Higher Order Thinking (HOT)
There are only 4 test items that belong to higher level questions and show the higher order thinking (HOT) with percentage 8%. These four test items are divided into two types.
a. Analysis
Three questions are closed test which asked the test taker to fill the appropriate word in the blank word. The test taker would be able to fill the blank word when
they known and understood the text, so they are able to related the sentences and made it complete. As Zaim (2016) stated that make relation and choose the appropriate word in order to make a complete text as the part of Analysis. We did it after the process of understanding the text.
b. Creating
Another test item belongs to the part of creating since the question is asking the test takers to arrange a paragraph; automatically this is related to Creating. As Zaim (2016:33) said that combine, design, create, arrange and rearrange verbs are the operational verbs used to determine the test based on the understanding level.
To make it clear, we provided the data in form of table 2.
Table 2. Analysis of LOT and HOT
No. Levels Number Percentage Cognitive domain
1 Lower Order Thinking (LOT) 46 92% C1, C2, C3
2 Higher Order Thinking (HOT) 4 8% C4, C6
Thus, most of the test items are in the low level and act as the lower order thinking. This is related to the government policy to provide the national examination test based on the general knowledge that can be accessed, achieved and answered by all students in Indonesia since national examination is evaluated nationally.
Government considers that the test can be answered by the students not only in the urban areas but also in rural areas of Indonesia, since they are at the same level of knowledge to understand and answer the test in national examination.
3. Communicative Language Ability
Communicative language ability is divided into five aspects; knowledge structure, language competence, strategic competence, Psycho Physiological mechanism and context of situation, but not all of these aspects are existed in the test items of English National Examination of Senior High School year 2014. Some of these aspects are not existed in the test items since the aspects are close to speaking communication not in written language.
a. Knowledge Structure
Knowledge structure is involved in all of the test items in the English National examination document that had been analyzed by us. All of the test contain of variant knowledge that combined and put in the forms of text and written language by using English in order to access the students’ competence. There are many texts with various topics existed in the test, students or test takers answered the questions based on the providing text. This shows the involving of knowledge structure there. As Zaim (2016) stated that knowledge structure is all of the sub knowledge and science that have to understood by human for living and keep living in the world.
b. Language Competence
Language competence comprises essentially a set of specific knowledge components that are utilized in communication via language. This language competence related to the previous knowledge structure in providing the test items with various types of texts and questions.
Organizational competence
This is involved in some test items since it function is related to correct sentences, ordering and form or arrange sentences, paragraph or even test.
1) Grammatical competence
Grammatical competence is involved in four number of test items. They are number 27, 29, 32 and 46. Grammatical competence that is involved in these four test items to
determine and identify the test takers competence in language usage in terms of vocabulary since the questions are asked the test takers to provide the synonymy and antonym of word.
2) Textual competence
Textual competence is involved in seven number of test items. They are number 33, 35, 38, 42, 48, 49 and 50. Textual competence that is involved in these seven test items are to be asked and guided test takers to understand the providing text and to get the real meaning in order to answer the questions given, to understand the providing topics in order to be able in arranging, completing and re arranging the paragraph or text, and to accessed the test takers or student competence in creating a good paragraph structurally and systematically.
Pragmatic Competence
There is no pragmatic competence that found in the test items since this competence is related to speaking test that concerned with the relationships among signs and their referents and relationships between utterances and the acts or functions that speakers (or writers) intend to perform through these utterances. The most common questions in these test items are asking the implicit information from text and monologue. The dialogue or monolog of listening section also is provided with literal meaning, not pragmatic meaning.
1) Strategic Competence
There is no Strategic competence found in this written test items since this competence is also related to speaking language not written language. It is found in some of test items in the listening section since it is related to language competence or knowledge of language, the language user’s knowledge structures and features of the context in which communication takes place. The communication in the monologue or dialog is almost strategic to help students understand it.
2) Psycho Physiological mechanism
There is no test item containing of psycho physiological mechanism; it could not be analyzed since it is for speaking language or communication. As Zaim (2016) stated that it can distinguish the visual from the auditory channel and the productive from the receptive mode that are shared by our interlocutor following the communication attempt and evaluating the extent to which the communicative goal has been achieved.
3) Context of situation
Context of situation is found in dialogue that is existed in the listening section test items.
The dialogue is happen based on the context situation but we could not identify them well since it is only in audio. Context situation is not found in the written language test items (text). The texts’ entire topic of the test items are in general knowledge that can be understood by all students in every area of Indonesia since this national examination is conducted nationally. There is no specific topic that is related to one or more areas, program, location or even environment in the test items. Thus, it could not be identified the context of situation of the written language or text.
Conclusion
Based on the document analysis that has been done by us, it is found that the fifty test items consist of Remembering 22%, Understanding 44%, Applying 26%, Analysis 6%, Creating 2% and no Evaluating. The lower order thinking (LOT) of the test items is 92% and the higher order thinking (HOT) is 8%. LOT refers to the lowest level of questions and HOT refers to the higher level of questions.
It is concluded that most of the test items are in the low level and act as the lower order thinking. This is related to the government policy to provide the national examination test based on the general knowledge that can be accessed, achieved and answered by all students in every area of Indonesia since national examination is evaluated nationally.
The most common types of communicative language ability that found in the test items are the knowledge structure and language competence. The texts’ entire topic of the test items are provided in general knowledge that can be understood by all
students in every area of Indonesia. There is no specific topic that is related to one or some area, program, location or even environment in the test items. Thus, it could not be identified from the context of situation of the written language or text. This study has not discussed all aspects of bloom taxonomy; HOT and LOT and Communicative language ability theory in various kinds of tests. Due to this limitaton, we give the suggestions to other researchers to conduct another analysis and further research related to this topic of other data or kinds of tests.
References
Airasian, P. (1991). Assessment for Instructional Planning. In P. Airasian (Ed)., classroom assessment (pp.73-120). Chapter 3. New York: McGraw-Hill.
Alderson, J. C. (1981). Report on the discussion on Communicative Language Testing.
In J. C.
Alderson & A. Hughes (eds.), Issues in Language Testing (pp.55-65). London: The British Council.
Bachman, L.F. (1990). Fundamental Considerations in Language Testing. Oxford etc.:
OUP.
Bachman, L.F., & Palmer, A.S. (1996). Language Testing in Practice: Designing and Developing Useful Language Tests. Oxford etc.: OUP.
Canale, M., & Swain, M. (1980). Theoretical bases of communicative approaches to second language teaching and testing. Applied Linguistics,1, 1-47.
Canale, M., & Swain, M. (1981). A Theoretical Framework for Communicative Competence. In Palmer, A., Groot, P., & Trosper, G. (Eds.), The construct validation of test of communicative competence, 31-36.
Hymes, D. H. (1972). On Communicative Competence. In Pride, J. B., & Holmes, J.
(Eds.), Sociolinguistics,269-293. Baltimore, USA: Penguin Education, Penguin Books Ltd.
Zaim. (2016). Evaluasi Pembelajaran Bahasa Inggris. Jakarta, Indonesia:Kencana.
