An analysis of Junior High School English un test items administered in 2012, 2013 and 2014.

(1)

ABSTRACT

Shabrina, F. (2015). An analysis of Junior High School English UN test items administered in 2012, 2013 and 2014. Yogyakarta: English Language Education Study Program Sanata Dharma University.

UN, abbreviated from Ujian Nasional, is a high-stakes test which is administered in Indonesia. That is why the result of UN must be trustworthy. Since validity is one of testing principles which represent the trustworthiness (Brown, 2004), this research investigates the validity of UN test items.

To be more precise, this research analyzes the documents of Junior High School English UN administered in 2012, 2013 and 2014. It describes how the three consecutive years of Junior High School English UN looks like and obtains information about the levels of thinking skills which are employed in the three consecutive years of Junior High School English UN. Moreover, it uses Brown’s testing theories and Bloom’s taxonomy to analyze the data.

Based on the analysis, the three consecutive years of Junior High School English UN tests reading skills which integrate selective and interactive reading. Consequently, the three consecutive years of Junior High School English UN tests materials such as English vocabulary items, grammar and reading comprehension. Next, based on the Bloom’s taxonomy the three consecutive years of Junior High School English UN employs the lowest level of thinking skills. The three consecutive years of Junior High School English UN leads the test takers to think in the levels of remembering, understanding and applying.

(2)

ABSTRAK

UN, singkatan dari Ujian Nasional, adalah tes berkonsekuensi tinggi yang diselenggarakan di Indonesia. Itulah mengapa hasil dari UN harus terpercaya. Karena keabsahan merupakan salah satu prinsip test yang mewakili keterpercayaan (Brown, 2004), penelitian ini mempermasalahkan keabsahan dari soal UN.

Untuk lebih jelasnya, penelitian ini menganalisis dokumen UN Bahasa Inggris Sekolah Menengah Pertama (SMP) yang diselenggarakan pada tahun 2012, 2013 dan 2014. Selebihnya, penelitian ini mendeskripsikan UN Bahasa Inggris SMP selama tiga tahun berturut-turut dan menggali informasi mengenai tingkat kemampuan berpikir yang diterapkan dalam UN tersebut. Oleh karena itu, dalam data analisis, penelitian ini menggunakan teori tes bahasa oleh Brown dan taksonomi Bloom.

Berdasarkan data analisis, UN Bahasa Inggris SMP selama tiga tahun berturut-turut menguji kemampuan membaca yang mengintegrasikan membaca selective dan interpretive. Sebagai hasilnya, UN Bahasa Inggris SMP selama tiga tahun berturut-turut mengujikan materi seperti kosa kata bahasa Inggris, tatanan bahasa Inggris dan pemahaman membaca. Selanjutnya, berdasarkan taksonomi Bloom, UN Bahasa Inggis SMP selama tiga tahun berturut-turut menerapkan tingkat kemampuan berpikir tingkat rendah yang menuntun siswa berpikir dalam tataran mengingat, memahami dan mengimplementasi.

(3)

AN ANALYSIS OF JUNIOR HIGH SCHOOL ENGLISH UN

TEST ITEMS ADMINISTERED IN 2012, 2013 AND 2014

A SARJANA PENDIDIKAN THESIS

Presented as Partial Fulfillment of the Requirements to Obtain the Sarjana Pendidikan Degree

in English Language Education

By

Faradita Shabrina Student Number: 111214160

ENGLISH LANGUAGE EDUCATION STUDY PROGRAM DEPARTMENT OF LANGUAGE AND ARTS EDUCATION FACULTY OF TEACHERS TRAINING AND EDUCATION

SANATA DHARMA UNIVERSITY YOGYAKARTA

(4)

AN ANALYSIS OF JUNIOR HIGH SCHOOL ENGLISH UN

TEST ITEMS ADMINISTERED IN 2012, 2013 AND 2014

By

(5)

i

AN ANALYSIS OF JUNIOR HIGH SCHOOL ENGLISH UN

TEST ITEMS ADMINISTERED IN 2012, 2013 AND 2014

By

(6)

(7)

(8)

(9)

(10)

vi ABSTRACT

UN, abbreviated from Ujian Nasional, is a high-stakes test which is administered in Indonesia. That is why the result of UN must be trustworthy. Since validity is one of testing principles which represent the trustworthiness (Brown, 2004), this research investigates the validity of UN test items.

To be more precise, this research analyzes the documents of Junior High School English UN administered in 2012, 2013 and 2014. It describes how the three consecutive years of Junior High School English UN looks like and obtains information about the levels of thinking skills which are employed in the three consecutive years of Junior High School English UN. Moreover, it uses Brown’s testing theories and Bloom’s taxonomy to analyze the data.

Based on the analysis, the three consecutive years of Junior High School English UN tests reading skills which integrate selective and interactive reading. Consequently, the three consecutive years of Junior High School English UN tests materials such as English vocabulary items, grammar and reading comprehension. Next, based on the Bloom’s taxonomy the three consecutive years of Junior High School English UN employs the lowest level of thinking skills. The three consecutive years of Junior High School English UN leads the test takers to think in the levels of remembering, understanding and applying.

(11)

vii ABSTRAK

UN, singkatan dari Ujian Nasional, adalah tes berkonsekuensi tinggi yang diselenggarakan di Indonesia. Itulah mengapa hasil dari UN harus terpercaya. Karena keabsahan merupakan salah satu prinsip test yang mewakili keterpercayaan (Brown, 2004), penelitian ini mempermasalahkan keabsahan dari soal UN.

Untuk lebih jelasnya, penelitian ini menganalisis dokumen UN Bahasa Inggris Sekolah Menengah Pertama (SMP) yang diselenggarakan pada tahun 2012, 2013 dan 2014. Selebihnya, penelitian ini mendeskripsikan UN Bahasa Inggris SMP selama tiga tahun berturut-turut dan menggali informasi mengenai tingkat kemampuan berpikir yang diterapkan dalam UN tersebut. Oleh karena itu, dalam data analisis, penelitian ini menggunakan teori tes bahasa oleh Brown dan taksonomi Bloom.

Berdasarkan data analisis, UN Bahasa Inggris SMP selama tiga tahun berturut-turut menguji kemampuan membaca yang mengintegrasikan membaca selective dan interpretive. Sebagai hasilnya, UN Bahasa Inggris SMP selama tiga tahun berturut-turut mengujikan materi seperti kosa kata bahasa Inggris, tatanan bahasa Inggris dan pemahaman membaca. Selanjutnya, berdasarkan taksonomi Bloom, UN Bahasa Inggis SMP selama tiga tahun berturut-turut menerapkan tingkat kemampuan berpikir tingkat rendah yang menuntun siswa berpikir dalam tataran mengingat, memahami dan mengimplementasi.

(12)

viii

ACKNOWLEDGEMENTS

First of all, I would like to show my great gratitude for Allah SWT. He always raises me up when I am down. Besides, He always gives His endless love and countless blessings to me.

I would like to address my best gratitude to Bapak Markus Budiraharjo, S.Pd., M. Ed., Ed.D., for his greatest support, guidance, patience, suggestions, and time he gives to supervise my thesis. I thank him for teaching me many life lessons. Next, I would like to thank Ibu Henny Herawati, S.Pd., M.Hum., Bapak Dr. Emanuel Sunarto, M.Hum. and Bapak Barli Bram for their suggestions and guidance so I can finish my thesis. I would like to thank Ibu Yuseva Ariyani Iswandari, S.Pd., M.Ed. for being my mother in this home (PBI).

Moreover, I would like to thank Bapak Heri Sumanto, S.Pd., the vice headmaster of SMP Negeri 15 Yogyakarta, who provides the English UN items for me. It is not easy to get the English UN items because they will be used by schools to give practices to the next ninth graders. I was so lucky at that time.

(13)

ix

I would like to express my best gratitude to the best people around me who never stop encouraging me during my study. I would like to thank Mas Adityo Prawinanto for always supporting me. I would like to thank my best friends, BRE and Vivin, for believing me that I can finish my thesis. They always make me believe that a hero lays on me. Next, I would like to thank my partners in crime, Ria and Cemplis, for understanding me and giving me chances to improve my character. I am lucky to have them. I would also like to thank my friends Jum, Dheta, and BTR for giving me inspiration to always be happy no matter what.

Best regards,

(14)

x

PERNYATAAN PERSETUJUAN PUBLIKASI ... v

(15)

xi

D. Instruments and Data gathering Techniques ... 27

E. Data Analysis Techniques ... 28

F. Research Procedures ... 32

CHAPTER IV. RESEARCH RESULTS AND DISCUSSION A. The Description of the Three Consecutive Years of Junior High School English UN ... 37

1. Testing Selective Reading ... 39

2. Testing Interactive Reading ... 42

(16)

xii

LIST OF TABLES

Table Page

Table 3.1 The Analysis of Reading Types Tested in the Test Items ... 29 Table 3.2 The Analysis of the Materials which Were Tested in

the Filling-gap Texts... 30 Table 3.5 The Analysis of Questions which Represented

Reading Comprehension Tests ... 31

Table 3.6 The Analysis of Questions which Employed

Six Levels of Thinking Skills Based on Bloom’s Taxonomy ... 32 Table 4.1 The Data Presentation of Reading Test Items

Based on Brown’s Typology of Reading Tests ... 38

Table 4.2 The Materials which Were Tested in Filling-Gap Texts ... 39

Table 4.3 The Materials which Were Tested in Reading

Comprehension ... 42 Table 4.4 The Classification of the Levels of Thinking Skills

(17)

xiii

LIST OF FIGURES

Figure Page

Figure 2.1 The Changes of Bloom’s Taxonomy in Cognitive Domain ... 15

Figure 4.1 A Filling-gap Text taken from English UN 2012 ... 40

Figure 4.2 A Greeting Card taken from English 2012 ... 43

Figure 4.3 A Letter Taken from English UN 2013 ... 45

Figure 4.4 A Greeting Card taken from English UN 2014 ... 47

Figure 4.5 A Question Taken from the Letter in English UN 2014 ... 49

Figure 4.6 A Letter Taken from English UN 2014 ... 52

Figure 4.7 A Greeting Card Taken from English UN 2014 ... 53

Figure 4.8 A Greeting Card Taken from English UN 2012 ... 55

Figure 4.9 A Question Taken from Greeting Card in English UN 2014 ... 56

(18)

xiv

LIST OF APPENDICES

Appendix Page

Appendix 1 The Coding Process of the Levels of Thinking Skills

Employed in the Test Items ... 70 Appendix 2 The Coding Process to Describe How the Test Items

(19)

1 CHAPTER I

INTRODUCTION

This chapter presents information and rationale why the researcher conducts the research. There will be six sections in this chapter. First is the research background which provides information about the background of the research. Second is the research problem which the researcher formulates the problems based on the background of the research. Third is the problem limitation which provides the focus of the research. Fourth is the research objectives which explain the aims of the research related to the research problems. Fifth is the research benefits which contribute benefits of the research for education field. Last is the definition of terms which defines specific words used in the research to avoid misconception.

A. Research Background

(20)

made from assessment results are appropriate, meaningful, and useful in terms of the purpose of the assessment” (as cited in Brown, 2004). Moreover, the advances

in the study of cognition and measurement in the 1980s and 1990s require tests to be cognitively rich in order to gain appropriate and meaningful results (Lane, 2004). It means that tests should apply complex cognitive thinking (Lane, 2004). Briefly, tests have to prompt various levels of thinking skills in order to reach the validity.

Regarding the definition of high-stakes tests, Ujian Nasional (UN) constitutes as a high-stakes test. As stated in Permendiknas No. 5 tahun 2005, the results of UN would be used to these considerations: deciding students’ graduation, becoming one of the requirements for students to go to the next level of education, mapping the education quality of schools, determining schools’

accreditation and helping schools to improve the education quality. Since 2015, UN no longer decides the graduation (POS Penyelenggaraan UN 2015). However, the results of UN are used as the requirement to enroll at higher level schools. It means that UN still gives serious impacts to several parties such as students, schools and teachers.

(21)

administered in Indonesia. The researcher is going to analyze UN items particularly the English subject. The researcher chooses the English subject because she is now studying in English Language Education Study Program in Sanata Dharma University which obliges her to conduct research about English language or English language teaching.

Moreover, the researcher analyzes English UN items which were administered in 2012, 2013 and 2014. The researcher chose these three consecutive years of English UN because they are the most recent examinations which are held in Indonesia. Moreover, the researcher is going to analyze English UN items for Junior High School students. It is because she has her teacher candidate practice in SMP Negeri 15 Yogyakarta. She also has experiences in teaching ninth grade students as their private teacher.

B. Research Problems

Based on the research background, there are two questions appear.

1. How are Junior High School English UN test items administered in 2012, 2013 and 2014 best described?

(22)

C. Problem Limitation

There are two focuses of this research. First is the description of the three consecutive years of Junior High School English UN items. Second is the level of thinking skills which are employed in the three consecutive years of Junior High School English UN items. There will be an analysis of English UN questions. The analysis will be based on the Bloom’s taxonomy theory. According to Bloom

(1987), there are six levels of thinking skills namely remembering, understanding, applying, analyzing, evaluating and synthetizing.

D. Research Objectives

This research analyzes the three consecutive years of Junior High School

English UN items. Therefore, this research has two objectives as follows.

1. To describe how the three consecutive years of Junior High School English UN looks like.

2. To obtain information about the levels of thinking skills which are employed in the three consecutive years of Junior High School English UN.

E. Research Benefits

(23)

1. For Schools

This research describes how English UN items look like and obtains information about the level of thinking skills which are employed in the three consecutive years of Junior High School English UN. It helps schools to prepare materials for UN from the beginning of the new academic year. Schools can predict what the materials which will be tested in next UN are. Therefore, schools can design test items for the try-outs from the beginning of the new academic year. In addition, schools can give appropriate time for students to have extension classes so students will not get tired, stressed and confused because of the extension classes. It is because when students work under pressure, they cannot study the materials effectively (Goleman, 1997, p.117 as cited in Puspitasari, Abidin & Sawitri, 2010, p.4).

2. For English Teachers

(24)

3. For English UN Designers

The results and discussion of this research help English UN designers to evaluate English UN items. English UN designers can improve the quality of English UN items. They can update the topics of the reading texts implemented in English UN. Moreover, they can vary the questions by considering to the levels of thinking skills according to Bloom’s taxonomy in order to improve test takers’ thinking skills.

4. For Future Researchers

This research analyzes the levels of thinking skill employed in the three consecutive years of Junior High School English UN based on Bloom’s taxonomy. It is expected to inspire the next researchers to analyze the levels of thinking skills using other perspectives. For example, the next researchers may see the perspective of text genres which lead the test takers not only think about information written in the texts, but also think more critically.

F. Definition of Terms

In this section, the researcher is going to explain the keywords of this research. The explanation will help readers to avoid misconception.

1. Ujian Nasional (UN)

(25)

Indonesia, and English. Second is to be an achievement indicator of National Education Standard. Furthermore, UN is administered in the second semester of an academic year. It is usually administered on April up to May. Moreover, UN involves students who are in their last grade such as sixth, ninth and twelfth graders. In conclusion, UN is a test which evaluates either students’ competences or education system in Indonesia.

2. English UN Test Items

English UN consists of many test items. The test items of Junior High School English UN are texts and their following questions. For example, a descriptive text and its three questions constitute one test items. Moreover, the scenario of text constitutes one test item. It is because the scenario gives instruction for students about the tasks.

3. Bloom’s Taxonomy

The term taxonomy means a system for naming or organizing things into groups which share similar qualities (“Taxonomy”). Bloom’s taxonomy is an

(26)

8 CHAPTER II

REVIEW OF RELATED LITERATURE

This chapter presents the review of theoretical writings related to English language testing and levels of thinking skills. This chapter has two main points namely the theoretical description and the theoretical framework.

A. Theoretical Description

This point reviews the theories directly relevant to the topic. This point also reviews other similar research studies relevant to the topic. Regarding the topic of this research, this point consists of four sub-points as follows.

1. A High-stakes Testing

High-stakes tests describe tests which have serious consequences for

students, teachers, schools and school systems (Hargrove, Jones & Jones, 2003). Madaus (1988) argues that high-stakes tests are to make decisions which affects parties include students, teachers, schools, committees and districts (as cited in Au, 2007). That is why the results of high-stakes tests must be trustworthy. In addition, validity is one of testing principles which represent the trustworthiness (Brown, 2004). Gronlund (1998) notes,“ validity is the extent to which inferences

(27)

2004). It means that tests should elicit complex cognitive thinking (Lane, 2004). Briefly, tests have to prompt various levels of thinking skills in order to reach the validity.

2. Ujian Nasional (UN)

UN, abbreviated from Ujian Nasional, is a high-stakes test which is administered in Indonesia. Stated in Permendikbud No. 97 tahun 2013, UN is an instrument to measure and assess the standard competencies nationally in particular subjects. Moreover, UN is administered mainly by National Education Standards Board (BNSP). Based on POS Penyelenggaraan UN 2015, UN is annual program which is administered on the second semester every year; it is about April to May. Considering UN is administered nationwide, it is exactly directed to students in Indonesia including Elementary School, Junior High School, Senior High School, and Vocational School. The subjects which are tested in each level are different. For Junior High School students, there are four subjects which are tested. They are Indonesian, Mathematics, Science, and English (Bonawati, personal communication, October 21st, 2014). Thus, UN is an Education Department’s program which is administered nationally to test students’ proficiencies of particular subjects they have taken in schools.

(28)

After talking about UN, the next sub-point reviews the theories about English language testing. In reviewing the theories, this research uses language testing theories by Brown (2004). Besides, this research involves other experts in

order to support Brown’s theories. To be more precise, the review is in the next

sub-point.

3. English Language Testing

According to Brown (2004, pp. 3-4), a test is a method of measuring an individual’s ability, knowledge or performance in a given domain. It means that a test represents how good someone’s ability, knowledge or performance in a

particular subject. For example, English is one of the subjects which are tested. English language tests measure test takers’ abilities to perform English language skills, which are to read, listen, write or speak (Brown, 2004, p. 3). Besides, an English language tests measure test takers’ English language knowledge about vocabulary items, grammatical rules, and rhetorical features in written discourse (Brown, 2004, p. 3). Thus, English language tests measures test takers’ English language abilities, knowledge or performances.

According to Brown (2004, p. 185), reading is a process of conveying meanings from the written words in order to gain information. Brown states, “The written word continues in its function to convey information, to assume and

entertain us, to codify our social, economic and legal conventions…” (2004, p.

(29)

a. Perceptive Reading

According to Brown (2004, p. 189), perceptive reading is an assessment for students who are at the beginning level of reading a second language. The tasks of perceptive reading imply that the students are in the early stages of becoming “literate.” Therefore, perceptive reading is one of the reading test levels which focus on the components of larger stretches of discourse: letters, words, punctuation, and other graphemic symbols (Brown, 2004, p. 189). The assessment which represents perceptive reading are reading aloud, picture-cued items and written response (Brown, 2004, p. 189).

b. Selective Reading

According to Brown (2004, p. 189), selective reading is one of reading test levels which focuses on the form of a language. It ascertains the test takers’ recognition of lexicons and grammatical rules of language within a very short stretch of language (Brown, 2004, p. 189). For examples, the preposition during has two meanings. First, the preposition during represents the meaning of a period time as in the sentence They work during the night and sleep by day. Second, it represents the meaning of several times in a period as in the sentence I woke up several times during the night (“During”). Another example is the English vocabulary room. It means a part of the inside of a building that is separated from other parts by walls, floor and ceiling (“Room”).

(30)

want them to succeed (“Support”). Next, the word nice means enjoyable (“Nice”).

Last, the word careful means giving a lot of attention to what you are doing so you do not have an accident, make a mistake, or damage something (“Careful”)

The next examples are about the grammatical rules such as the use of simple past tenses, an adverb there, a pronoun it and simple present tenses. The simple past tense expresses events which began and end in the past (Azar, 1999, p. 27). According to Allen (1995, p. 86), “The great majority of English verbs form the past simple tense and past participle by adding the sound [d], [t] or [id] to the

stem…” Allen (1995, p. 86) states that there are irregular verbs used in the simple

past tense. Besides, the verb came is one of the examples of the irregular verb

(“Come”).

The other examples of the grammatical rules are the use of an adverb there and a pronoun it. The adverb there is an adverb which indicates places (“There”). Next, a pronoun is used in place of a noun (Azar, 1999, p. 132). For example, I read a book; it was good. The pronoun it refers to the noun book (Azar, 1999, p. 132). Next, the simple present is used to tell the truth (Azar, 1999, p. 13). Krohn (1986, p. 11) states the –s form of verbs is used in the simple present tense with third person singular objects; otherwise the simple form is used. For example, He works every morning.

Besides testing vocabulary items and grammatical rules, English language tests may test about English expression. One of English expression which may be tested is compliments. Compliments express appreciation and admiration at

(31)

compliment such as That’s nice, That’s neat, Pretty good and I really like your skirt (Bagus, 2012).

Moreover, there are two characteristics of selective reading. First, selective reading tasks typically are constructed by using stimuli such as sentences, brief paragraphs or simple charts and graphs (Brown, 2004, p.189). Second, brief responses are intended in selective reading (Brown, 2004, p.189). Briefly, one example of selective reading tasks is filling-gap task (Brown, 2004, p. 189). c. Interactive Reading

According to Brown (2004, p. 201), interactive reading is similar to selective reading. Interactive reading focuses on the form of a language and meanings, but it focuses more on the meanings (Brown, 2004, p. 201). It focuses on how the form of a language makes meanings. Therefore, there are two characteristics of interactive reading. First, it involves the test takers to interact with longer texts (Brown, 2004, p. 189). The test takers have to negotiate with the meanings of texts in order to get intake to the texts (Brown, 2004, p. 189). Second, interactive reading typically uses text genres such as narrative texts, descriptions, memos, recipes, announcements and so forth (Brown, 2004, p. 189). Moreover, Brown (2004, pp. 201-210) states that interactive reading uses tasks such as cloze tasks, a reading comprehension, short-answer-tasks, editing, scanning, ordering tasks, and information transfer (from charts, maps, diagrams and graphs).

(32)

According to Brown (2004, p. 3), a reading comprehension consists of a set of texts with the comprehension questions. There are eight features which cover the comprehension. They are main ideas (topic), expressions/idioms/phrases in context, inferences, grammatical features, detailed information, unstated details, supporting ideas and vocabulary in context (Brown, 2004, p. 206). Briefly, a reading comprehension usually involves reading strategies such as skimming, scanning and guessing word meaning from context (Brown, 2004, p. 206). To be more precise, skimming is a reading method of rapidly moving the eyes through the text with the purpose to gain main ideas and general overview of the content of the text (Campbell, 2005). Next, scanning is a reading method of rapidly covering a great breadth of the display in order to locate icons, symbols, or words in the text (Campbell, 2005).

d. Extensive Reading

Expensive reading is a type of reading which involves more than a page text including professional articles, essays, technical reports, short stories and books (Brown, 2004, p. 189). The tasks which represent extensive reading are note-taking and outlining (Brown, 2004, p. 215).

After talking about English testing theories according to Brown (2004), the next part talks about the levels of thinking skills derive from Bloom’s taxonomy. 4. The Levels of Thinking Skills

(33)

Create

taxonomy is a six-classification system which is to observe students’ behavior and infer students’ cognitive achievement. The classification consists of such categories: knowledge, comprehension, application, analysis, synthesis and evaluation (Bloom, 1987). As time goes by, Anderson and Krathwohl (2001) made a revision of Bloom’s taxonomy in cognitive domain. To be more precise,

the following graphs will present the revision of Bloom’s taxonomy in cognitive domain. Churches (2008) states these classifications present Bloom’s taxonomy in cognitive domain from the lowest level of thinking skills (remembering) to the highest level of thinking skills (creating).

The Original Bloom’s Taxonomy The Revision of Bloom’s Taxonomy

Figure 2.1 The Changes of Bloom’s Taxonomy in Cognitive Domain

These six levels of thinking skill are divided into two categories. The categories are “lowest level of thinking skill” and “highest level of thinking skill” (Anderson, 1990, p. 425). There are two levels which are considered as the lowest

(34)

levels of thinking skill. They are remembering, understanding and applying (Forehand, 2010). It is supported by Churches (2008) that students cannot understand the concept if they do not remember it first, similarly they cannot apply if they do not remember information and understand concepts. It means that the basic level of thinking start from remembering, understanding then applying. Moreover, there are three levels of thinking skill which are considered as the highest levels (Forehand, 2010). They are analyzing, evaluating and creating (Forehand, 2010). Mayer and Wittrock (1966) note these levels are considered as the higher levels because they leads students to transfer information they have before to solve new problems or new questions (as cited in Anderson & Krathwohl, 2001, p. 63). To be more precise, here is the explanation.

a. Remembering

According to Anderson and Krathwohl (2001, p. 63), remembering promotes retention which means that it leads students to memorize information at some later time and the same way they are given. Churches (2008) states remembering level is a process of retrieving relevant information. In assessing

students’ learning, remembering employs two behaviors such as recognizing and

(35)

the memorization in which it should concern on complex cognitive process (as cited in Haladyna, 2012, p. ix).

Second, recalling (retrieving) leads students to retrieve relevant knowledge from their long-term memory when given stimuli to do so (Anderson & Krathwohl, 2001, p. 69). The stimuli are usually questions (Anderson & Krathwohl, 2001, p.69). An example given by Haladyna (2012, p. 31) that students are given a task to mention one primer number. It needs students’ long term-memory in order to answer the question (Haladyna, 2012, p. 31). Thus, remembering level mostly employs memorization skill in assessing students’ competences.

b. Understanding

According to Anderson and Krathwohl (2001, p. 70), understanding occurs when students are able to construct meaning from instructional messages including oral, written or graphs. It means understanding leads students to build connection between the new knowledge and their prior knowledge (Anderson & Krathwohl, 2001, p. 70). The new knowledge here does not mean really “new”, but it still has connection to students’ existing knowledge (Anderson & Krathwohl, 2001, p.70).

(36)

meanings (Anderson & Krathwohl, 2001, p. 70). Second, summarizing occurs when students are able to suggest a single statement that represents presented information (Anderson & Krathwohl, 2001, p. 73). It means that summarizing level leads students to construct a representative meaning of the information (Anderson & Krathwohl, 2001, p. 73). An example of summarizing task is to determine a theme or main points of a paragraph or texts (Anderson & Krathwohl, 2001, p. 73). Thus, understanding level mostly focuses on the meaning construction.

c. Applying

(37)

d. Analyzing

According to Anderson and Krathwohl (2001, p. 79), analyzing occurs when students are able to categorize information into its constituent parts and determine how the parts are related to one another and to overall structure. Forehand (2010) note that analyzing involves students to be able to distinguish information by creating charts, diagrams, surveys or spreadsheets. Additionally, information here means the pieces of message which students have to organize and underlying the purposes (Anderson & Krathwohl, 2001, p. 79). Analyzing level has behaviors which are employed in leaning such as differentiating, organizing and attributing (Anderson & Krathwohl, 2001, pp. 80-84).

e. Evaluating

According to Anderson and Krathwohl (2001, p. 83), evaluating occurs when students are able to make judgments on quality, effectiveness, efficiency and consistency based on criteria they already perceive. Driscoll and Driscoll (2005, p. 348) state evaluating is when students make judgments about the extent to which methods or materials satisfy existing criteria. Briefly, there are the behaviors which are employed in evaluating skill such as checking and critiquing (Anderson & Krathwohl, 2001, pp. 83-84); Driscoll & Driscoll, 2005, p. 348; Forehand, 2010).

f. Creating

(38)

and Forehand (2010) argue that creating occurs when students are able to put together parts, elements or materials to form new structures or patterns. Creating does not mean writing which represents the remembering of ideas or the interpretation of materials (Anderson & Krathwohl, 2001, p. 85). Students must synthetize their own ideas or, in other words, take a position of particular cases (Anderson & Krathwohl, 2001, p.85). In brief, there are several behaviors which are employed in creating skill such as generating, planning, producing, inventing, designing and making (Anderson & Krathwohl, 2001, pp. 86-87; Churches, 2008).

B. Theoretical Framework

This research uses several theories such as high-stakes tests, Ujian Nasional (UN), Englisng language testing and Bloom’s taxonomy. First, the theories about high-stakes tests and UN are used to shape the researcher’s thinking framework in writing the research. To be more specific, high-stakes tests describe tests which have serious consequences for students, teachers, schools and school systems (Hargrove, Jones & Jones, 2003). Therefore, UN is considered as a high-stakes testing in Indonesia since it is to make decisions which affect testees include students, teachers, committees, schools and districts (Madaus, 1988 as cited in Au, 2007).

(39)

English language test theories. According to Brown (2004), language tests measure individual’s performance of a language like reading. There are four types of reading; two of them are selective and interactive reading (Brown, 2004). First, selective reading focuses on integrating the form of a language such as lexicons, grammar and discourse features (Brown, 2004). An example of selective reading tasks is fill-in-the-blank in form of multiple choices (Brown, 2004). Second, interactive reading also focuses on the form of a language, but it emphasizes more on the meaning of messages given (Brown, 2004). Therefore, it leads test takers to interact with longer texts in order to gain meanings (Brown, 2004). An example of interactive reading tasks is a reading comprehension (Brown, 2004). It is because a reading comprehension consists of numerous texts which lead test takers to gain meanings (Brown, 2004). Additionally, a reading comprehension consists of questions following the texts (Brown, 2004). These questions are main ideas (topic), expressions/idioms/phrases in context, inference (implied detail), grammatical features, detailed information, unstated details, supporting ideas and vocabulary in context (Brown, 2004, p. 206).

Third, the research uses Bloom’s taxonomy to answer the second research

question. Bloom’s taxonomy is used to obtain information about the levels of

(40)

Bloom’s taxonomy in cognitive domain are remembering, understanding,

applying, analyzing, evaluating and creating (Anderson & Krathwohl, 2001). These levels work from the lowest, remembering, to the highest, creating (Anderson & Krathwohl, 2001; Churches, 2008).

Accordingly, these six levels are divided into two categories (Anderson, 1990). First are the lowest levels of thinking: remembering, understanding and applying (Forehand, 2010). These levels promote retention in which leads students to work on their memorization (Anderson & Krathwohl, 2001). Second are the highest levels of thinking: analyzing, evaluating and creating (Forehand, 2010). Mayer and Wittrock (1966) note these levels promote transfers in which students are able to transfer their prior information or knowledge to solve new problems. Thus, each level of thinking skills has its complexities which go higher and higher (as cited in Anderson & Krathwohl, 2001).

(41)

(42)

24

CHAPTER III

RESEARCH METHODOLOGY

This chapter presents the methodology of the research and the analysis. There are six points in this chapter namely research method, research setting, research objects, research instruments and data gathering technique, data analysis technique, and research procedure.

A. Research Method

This research constituted qualitative research. It was because the researcher used her own experiences as the basic background of this research. Merriam (2009, p.5) shares that qualitative research focuses on how the researchers interpret their experiences, how they construct their knowledge and how they attribute meaning to their experiences. Since there were several methods in the qualitative research, this research used a content analysis method. There were two reasons why this research employed the content analysis method.

(43)

Second, this research had two objectives which were to describe the high-stakes test in Indonesia and to obtain the levels of thinking skills employed in that kind of test. Fraenkel and Wallen (2006, 485) notes the content analysis was to obtain descriptive information about the topic and information useful to education world. Thus, this research would investigate problems which related to education.

B. Research Setting

This research was conducted in ongoing process. The researcher started collecting the data from November 2014. The researcher, then, analyzed the data in January to May 2015. The researcher took times in conducting this research because there were several obstacles she faced. These obstacles actually help the researcher to learn deeply about the topic she investigated. Moreover, there were no particular places such as schools or classes for researcher to conduct this research. It was because this research did not use observation or interviews.

C. Research Objects

(44)

2014. The test items of the three consecutive years of Junior High School English UN constituted the population of this research. Fraenkel and Wallen (2006, p. 92) argue that population is the larger group to which the researcher hopes to apply the results. Fraenkel and Wallen (2006, p. 92) also argue that the smaller number of groups taken from the population was called as a sample. Hence, the sample of this research was 10 greeting cards, 19 letters, 32 filling-gap texts along with 183 following questions. Briefly, the researcher used a purposive sampling method in the instrumentation.

(45)

D. Instruments and Data Gathering Techniques

An instrument was a device which the researchers used to gather data, and it involved processes called instrumentation (Fraenkel & Wallen, 2006, pp. 112-113). The instrument of this research was personal construct which used the researcher herself as the major instrument of the research. It was supported by Cohen, Manion and Marrison (2011, p. 498) that the personal construct is a research instrument which uses the researchers’ constructs to interpret what they perceived in terms of their existing mind and the way they behave towards it. Merriam (2009, p. 150) convinces,” Since investigator is the primary instrument for gathering data, he or she relies on skills and intuition to find and interpret data from documents”. Thus, the researcher used her knowledge about Brown’s testing theories and Bloom’s taxonomy to collect and interpret the data which she found

from the three consecutive years of Junior High School English UN documents. The researcher used coding categories as the technique to gather the data since formulating coding categories was the technique applied in the content analysis (Fraenkel & Wallen, 2006, p. 487). Moreover, the researcher determined the categories before the analysis began. Fraenkel and Wallen (2006, p. 485) argue that the researcher can determine the categories before any analysis began.

(46)

used Bloom’s taxonomy to answer the second research question about the levels

of thinking skills which were employed in the three consecutive years of Junior High School English UN. Anderson and Krathwohl (2001) states there are six levels of thinking skills which were discovered by Bloom (1987) namely remembering, understanding, applying, evaluating and creating.

Accordingly, the six levels of thinking skills employed certain behaviors which have been the characteristics of each level. For example, remembering employed recognizing and recalling behaviors. Understanding employed interpreting and summarizing behaviors. Next, applying employed executing behavior. Analyzing employed differentiating, organizing and attributing behaviors. Evaluating employed checking and critiquing. Last, creating employed generating, planning, producing, inventing, designing and making behaviors (Anderson & Krathwohl, 2001, pp. 66-88; Churches, 2008).

E. Data Analysis Techniques

(47)

1. Counting the Frequency of Reading Types

One of the objectives of this research was to describe the three consecutive years of Junior High School English UN test items. In describing the test items, the researcher made a cross tabulation of reading types which were tested in the test items as the continuation of the coding process. The cross tabulation was presented as follows.

Table 3.1 The Analysis of Reading Types Tested in the Test Items Brown’s typology of reading tests. According to Brown (2004, p. 189), there are four reading types namely perceptive, selective, interactive and extensive reading. These reading types were used to classify the data which has been analyzed. Moreover, the researcher counted the frequency of the test items which included in each reading type and changed it to the percentage. It was supported by Fraenkel and Wallen (2006, p. 490) that the end product of the coding process must be numbers.

2. Counting the Materials which Were Tested in the Filling-gap Texts

(48)

Junior High School English UN looked like. The researcher found an aspect which could be the results of the topic she investigated. The aspect was about the materials which were tested in the filling-gap texts. Additionally, the researcher used the following table to help her analyze the data.

Table 3.2 The Analysis of the Materials which Were Tested in the Filling-gap Texts

Materials

Material 1 Material 2

Frequency Percentage (%)

Based on the coding process, the researcher listed the materials which were tested in the filling-gap texts. The researcher, then, counted the frequency of how many times certain materials appeared. The researcher also counted the frequency in form of percentage. Finally, the researcher used the counting result as the findings which would be discussed in the nest chapter. It was supported by Fraenkel and Wallen (2006, p. 490) that the end product of the coding process must be numbers.

3. Counting the Materials which Were Assessed in Reading Comprehension

(49)

Table 3.3 The Analysis of Questions which Represented

Reading Comprehension Tests

The materials

Material 1 Material 2 Material 3 Material 4 Material 5 Material 6

Frequency Percentage

(%)

The researcher counted the frequency of how many times the material appeared. The researcher then counted the frequency in form of a percentage. Briefly, the researcher used the counting result as the data to be discussed in the next chapter. It was supported by Fraenkel and Wallen (2006, p. 490) that the end product of the coding process must be numbers.

4. Counting the Questions which Employed Six Levels of Thinking Skills

Based on Bloom’s Taxonomy

(50)

Table 3.4 The Analysis of Questions which Employed Six Levels of Thinking Skills Based on Bloom’s Taxonomy

Six Levels

of Thinking Skills The Behaviors 2012 2013 2014 Frequency

Percentage

The researcher counted the frequency of the questions which represented each behavior of the levels of thinking skills. The researcher then changed the frequency to the percentage. Finally, the counting result would be the finding which discussed in the next chapter. It was supported by Fraenkel and Wallen (2006, p. 490) that the end product of the coding process must be numbers.

F. Research Procedure

There were eight procedures which the researcher passed when conducting this research. The procedures were presented as follows.

1. Determining the Objectives

(51)

looked like. Second, was to obtain information about the levels of thinking skill employed in the three consecutive years of Junior High School English UN. 2. Defining the Terms

The researcher defined the important terms of this research to avoid misconception. As Fraenkel and Wallen (2006, pp. 485 – 486) state that researchers as in all research might incur misunderstanding, the important terms must be defined. Consequently, there are two important terms in this research to define. First is UN abbreviated from Ujian Nasional. The term UN was important to be defined because it was one of many examinations which were held in Indonesia. The definition of the term UN helped the readers avoid misunderstanding about what kind of test UN was. Second, “English test items” was another important term in this research. It was important to define because it would frequently appear in each chapter of this research. It helped the reader also not to assume that the term “items” was the test questions. Briefly, the researcher defined the term “English test items” as the English reading passages along with

their following questions since Junior High School English UN consisted of reading passages and questions.

3. Locating the Relevant Data

(52)

located data. The researcher used that data since the objective of this research was to describe Junior High School English UN.

4. Developing a Sampling

The researcher developed a purposive sampling in this research. The researcher then chose 10 greeting card, 19 letter and 31 filling-gap texts along with 183 following questions. The researcher intentionally determined the sample because it had specific reasons which supported the topic she investigated. As Fraenkel and Wallen (2006, p. 100) state, researchers could use a purposive sampling to determine the sample which specifically supported the purposes of the research.

5. Formulating Coding Categories

The research conducted two coding processes in this research. It was because the researcher applied two techniques when determining the coding categories. The researcher determined the coding categories to gather the data about the levels of thinking employed in the three consecutive years of Junior High School English UN before any analysis began. Fraenkel and Wallen (2006, p. 485) argue, researchers could determine the categories before any analysis began.

6. Reliability and Validity

(53)

(1959) argue, “Triangulation is a powerful way of demonstrating concurrent validity, particularly in qualitative research” (as cited in Cohen, Manion, and

Morrison, 2011, p. 195). Moreover, this research applied three types of triangulation in order to measure the reliability and validity of the collected data.

First is time triangulation. Cohen, Manion, and Morrison (2011, p. 196) state, “This type attempts to take into consideration the factors of change and process by utilizing cross-sectional and longitudinal designs.” Therefore, the process which allowed the researcher to make revisions ensured the researcher’s ways of thinking so that it made the collected data valid. Second is theoretical triangulation. Cohen, Manion, and Morrison (2011, p. 196) state, “This type draws upon alternative or competing theories in preference to utilizing one viewpoint only.” The researcher used theories about testing by Brown (2004) and

about levels of thinking skill by Anderson and Krathwohl (2001) to shape her point of view. That was why these theories made the consistency and correctness of the data collecting process.

Last is investigator triangulation. Silverman (1993) states that researchers can measure the research reliability and validity by engaging more than one observer (as cited in Cohen, Manion, and Morrison, 2011, p. 196). Therefore, the researcher engaged experts’ judgments to measure the reliability and validity of

(54)

7. Analyzing the Data

The researcher used counting technique in analyzing the data. It was supported by Fraenkel and Wallen (2006, p. 490) that counting is a characteristic of some content analysis methods. Moreover, the researcher counted the frequency of several aspects which would support the topic she investigated. The researcher counted the frequency in the form of a percentage as well.

8. Reporting the Research Results

(55)

37 CHAPTER IV

RESEARCH RESULTS AND DISCUSSION

This chapter presents research results which had been analyzed in ways justified in the research methodology. In addition, it discussed the results in light of the research questions. The first research question is about how the three consecutive years of Junior High School English UN is best described. Therefore, the first point of this chapter fully describes how the three consecutive years of Junior High School English UN looks like. The second research question is about the levels of thinking skills employed in the three consecutive years of Junior High School English UN for Junior High School are. The second point of this research obtains information about how high the three consecutive years of Junior High School English UN employs levels of thinking skills. Thus, the research results and discussion are presented in the two main points as follows.

A. The Description of the Three Consecutive Years of Junior High School

English UN

(56)

that reason, this research used Brown’s theory of reading test to analyze the data. Here is the data presentation.

Based on the data analysis, the three consecutive years of Junior High School English UN tested reading skills. Brown (2004, p. 185) argues that reading is a process of conveying meanings from the written words in order to gain information. The data showed that the three consecutive years of Junior High School English UN consisted of reading passages which allowed the test takers to convey meanings in order to get information. Brown (2004, p. 3) also argues that English language tests measure test takers’ abilities to perform English language

skills, which are to read, listen, write or speak. To be more precise, the types of reading which are tested in the three consecutive years of Junior High School English UN presented as follows.

Table 4.1 The Data Presentation of Reading Test Items Based on Brown’s Typology of Reading Tests

(57)

there were no questions which tested perceptive and extensive reading. To be more precise, the discussion is presented as follows.

1. Testing Selective Reading

The three consecutive years of Junior High School English UN tested reading skill through selective reading. It was proved by filling-gap texts which delivered in the three consecutive years of Junior High School English UN (Brown, p. 189). These texts constituted a selective reading because they assessed the test takers’ knowledge about English grammar and vocabulary items. It was supported by Brown (2004, p. 189) that selective reading focused on the recognition of lexicons and grammar (Brown, 2004, p. 189). Moreover, here is a table which represented the materials which were tested in the three consecutive years of Junior High School English UN through selective reading.

Table 4.2 The Materials which Were Tested in Filling-Gap Texts

Materials

English Vocabulary items English tenses

Frequency 88 5

Percentage (%) 94.6% 5.4%

(58)

43. …

I went to Germany with my family … (43) a school holiday a few years ago. It was

getting dark, so we were looking for a hotel. Then we … (44) upon two hotels next

each other with … (45) opening. We just randomly chose one and settled of the night.

Figure 4.1 A Filling-Gap Text Taken from English UN 2012

(59)

consisted of three words or even sentences. Thus, the filling-gap text represented two characteristics of selective reading as Brown (2004) states.

Moreover, Figure 4.1 represented the materials which were assessed in filling-gap texts. First, there were two questions which assessed English vocabulary items. They were questions number 43 and 45. Question number 43 assessed the use of preposition during since it could be used to show period of time (“During”). Furthermore, the preposition during in the reading passage was to show the period of time when the author stared her/his holiday until the end of her/his holiday. Next, question number 45 assessed the vocabulary item in the context. It assessed the meaning of an English word room since the correct answer was a noun which represented something opened in the hotels. Room means a part of the inside of a building that is separated from other parts by walls, floor and ceiling (“Room”).

Second, question number 44 assessed the test takers’ knowledge about English tenses. It assessed the use of the simple past tense since the reading passage indicated the author’s experience which happened in the past. As Azar

states (1999, p. 27), the simple past tense expresses events which began and end in the past. Because it assessed the simple past tense, the test takers were expected to know the past form of the verb come which the answer was came (“Come”). The verb came is one of the examples of the irregular verb used in the simple past tense (“Come”). As a result, filling-gap texts integrated the test takers’ knowledge

(60)

2. Testing Interactive Reading

Based on the analysis, the three consecutive years of Junior High School English UN assessed the test takers’ reading performances by integrating interactive reading. It was supported by Brown (2004, p. 189) that integrative reading involved test takers to deal with text genres. Similar to selective reading, interactive reading focuses on the form of a language and meanings, but it focuses more on the meanings (Brown, 2004, p. 201). Furthermore, two of text genres which were delivered in the three consecutive years of Junior High School English UN for Junior High School were greeting cards and letters. Based on the analysis, these text genres promoted reading comprehension tasks since reading comprehension was one example of interactive reading tasks (Brown, 2004, pp. 201-210). To illustrate the analysis, here is the table.

Table 4.3 The Materials which Were Tested in Reading Comprehension

The Materials Frequency Percentage (%)

(61)

The following text is for questions 3 and 4.

3. From the text we know that Fachri … A. gets a medal for the best student B. is the best student at school C. graduates from school D. got five for his test

4. “… please step forward …”

The underlined word means …

A. _{come to the front}

B. _{move back}

C. _{one step to the left}

D. _{stand in line}

Dear Fachri,

When your name is called as the best student please step forward ─ for high five!

You are the best. Congratulations!

Uncle Farhan

precise, here are the examples of reading passages which assessed reading comprehension.

Figure 4.2 A Greeting Card Taken from English UN 2012

(62)

rapidly move their eyes to find the main point of the passage in order to gain the overview of the content (Campbell, 2005).

Next, question number four assessed the test takers’ vocabulary knowledge. It assessed the word forward which meant coming to the front (“Forward”). Moreover, the test takers might guess the word meaning from the

(63)

Figure 4.3 A Letter Taken from English UN 2013

(64)

skimming strategy since Brown (2004, p. 206) tones that skimming strategy is necessary in reading comprehension. The test takers might move their eyes rapidly through the passage to get the general overview of the content (Campbell, 2005). As a result, the main topic of the passage was spending holiday in three places. It was because the passage generally told about the author’s holiday

experiences in Huatulco, Iceland and New York.

Second, question number 12 asked the detailed information stated in the passage. It asked where the author’s friend spent his/her holiday. To find the answer, the test takers might use scanning strategy as it is suggested by Brown (2004, p. 206). The test takers might move their eyes rapidly to find the keywords spent and holiday. The test takers could find the keywords in the first and the second paragraph. Furthermore, the answer was in the second sentence of the first paragraph. The word spent in the first paragraph was directed to the author’s friend. As a result, the author’s friend spent her/his holiday in Spain.

(65)

The following text is for questions 2 and 3.

Dear Susi,

Congratulations on your success in “The National English Storytelling Competition

for Junior High School Students 2013.”

You have beaten other finalists from all over Indonesia. You are best of the best. I knew you would win. You are so talented. Your English is excellent. You can act,

sing and dance, too. I’m so proud of you.

I’m sure you’ll be more successful in the future.

Hug & kisses

Ria

2. What is the purpose of the text?

A. To inform other finalists of the competition result. B. To show that Ria is very proud of herself.

C. To praise Susi on her success in winning the storytelling competition. D. To encourage Susi to join the National English Storytelling Competition.

3. _{Which sentence in the text express a compliment?}

A. You are so talented. B. I know you would win.

C. You have beaten other finalists.

D. You will be more successful in the future.

author went to these places because the climate and the tourist spots were different from others. From the test items discussed so far, three-year English UN for Junior High School focused on assessing both the form of English language and reading comprehension.

Figure 4.4 A Greeting Card Taken from English UN 2014

(66)

English expression. Brown (2004, p. 206) states that inferences and expressions are two features which cover reading comprehension. Question number two asked about the purpose of the passage. It led the test takers to infer information in the passage. As Brown (2004, p. 206) notes, inferencing strategy is necessary in reading comprehension. Therefore, the test takers had to see the context of the passage and summarize the purpose of the passage that it was to praise Susi on her success in winning the storytelling competition.

Next, question number three asked about an English expression which was written in the passage. The expression which was tested was a compliment. Considering question number three had four choices which consisted of various expressions, the one which constituted a compliment was the choice A, You are so talented. As a result, the choice A constituted a compliment because it expressed appreciation and admiration at someone’s appearance, belongings or works (Bagus, 2012).

(67)

Figure 4.5 A Question Taken from the Letter in English UN 2014

The question in Figure 4.5 asked about what the pronoun it referred to. Therefore, it led the test takers to find a noun in the second paragraph which referred to the pronoun it. Azar (1999, p. 132) states a pronoun is used in place of a noun. In addition, the test takers had to use their scanning skill to find information about the pronoun it referred to. Brown (2004, p. 206) says that scanning is one of reading skills which is needed in reading comprehension. The test takers might move their eyes rapidly to find a noun which referred to the pronoun it. In a nutshell, the noun that referred the pronoun it in the second paragraph was Mount Bromo.

The discussions above were reporting how three consecutive years English UN looked like. They answered the first research question. Moreover, the researcher was going to discuss another topic related to the levels of thinking skills in order to answer the second research question. Here is the discussion.

7. “It was so beautiful” (paragraph 2).

What does it refer to?

A. The peak.

B. The dessert.

C. The sunrise.

(68)

B. The Levels of Thinking Skill Employed in the Three Consecutive Years

of Junior High School English UN

Answering the second research question, the researcher analyzed 173 questions in order to obtain information about the levels of thinking skills which were employed in the three consecutive years of Junior High School English UN. The researcher used Bloom’s taxonomy to investigate the levels of thinking skills

which were employed in the three consecutive years of Junior High School English UN test items. To be more precise, here is the data presentation.

Table 4.4 The Classification of the Levels of Thinking Skills in Reading Test Items Based on Bloom’s Taxonomy

Six Levels

of Thinking Skills The Behaviors 2012 2013 2014 Frequency

Percentage thinking skills as Anderson (1990, p.425) categorized. They employed three levels of thinking such as remembering, understanding and applying (Anderson, 1990, p.425). It was supported by Forehand (2010) that remembering, understanding and applying were considered as the lowest levels of thinking skills.