View of Hots Manifestation in English Summative Assessment

(1)

│ Volume 8 Number 2 October 2023 E-ISSN: 2503-4405│ │ │ │ P-ISSN: 2580-3441│

RachmawatiAchadiyah, Muhammad Ulul Azmi HOTS Manifestation on English Summative Assessment

HOTS Manifestation on English Summative Assessment

Rachmawati Achadiyah¹, Muhammad Ulul Azmi² Universitas Kyai Abdullah Faqih, Indonesia

Abstract

Higher-order thinking gains a positive role in language assessment since it is compulsory to be implemented in the teaching-learning and evaluation system of the latest curriculum. The implementation of HOTS in English tests assists students in practicing their analytical, critical, and problem-solving skills in real life. It is essential to know how the HOTS’ question is narrated in English summative assessment. This study specifically employed a case study as a research design with content analysis as the technique to analyze the data. Three papers of the English summative test of the first, second, and third grades of junior high school became the source of the data. The findings revealed that the ninth-grade English summative test used higher-order thinking skills in 72% of the questions, including both multiple choice and essay formats. Meanwhile, the seventh and eighth grades prioritize the acquisition of language patterns and the mastering of vocabulary. The higher-order thinking skills (HOTS) encompassed various cognitive levels, including problem-solving, providing recommendations, offering solutions, conducting analysis, making judgments, anticipating outcomes, and fostering creativity. This study examined the process by which English teachers, in their role as test designers, create questions that require higher-order thinking skills (HOTS) for English summative assessments. The study focused on the use of different operational verbs within the cognitive domains of C4, C5, and C6.

Keywords:High order thinking skills; English summative; English assessment

INTRODUCTION

The teaching-learning involves some crucial aspects. Assessment is one of them. It is the teacher's responsibility to analyze and evaluate students' competence and abilities gained during the teaching-learning process. The assessment has several benefits for the teacher. It can assist the teacher in examining students’ strengths and weaknesses in a certain domain, evaluating their global knowledge during the course, and determining their proficiency level.

Assessment does not only assist the teacher but also the students to know how they have understood lessons that have been taught to them. They need to know their progress on a particular lesson or course. (Kurniawati & Sundawa, 2019) Besides these, the curriculum designer also took the benefits of the assessment. The curriculum designer uses the result of the assessment as the main source to decide whether the topic learning will be utilized again, revised, or even omitted.

(2)

In language-learning, teaching and assessing are intertwined relationships. In teaching, teachers transfer knowledge, competence, and training skills to students. While assessing, teachers check and verify whether knowledge, competence, and skills are achieved well or not.

By conducting the appropriate assessment, the teacher can place students based on their proficiency, identify students’ strengths and weaknesses, and give students feedback.(Tosuncuoglu, 2018)Specifically, in foreign language learning, the assessment has many varieties of purposes. They are formal and informal tests, formative and summative, norm- referenced and criterion-referenced. This study focuses on summative assessment. These two types of assessment have their characteristics in assessing students. The main characteristics of a formative assessment are that it provides students’ improvements through teachers’

corrective feedback and it is undergone throughout the teaching-learning process. Moreover, summative assessment is to measure students’ achievement of the lessons that they are learning at specific times, and it was undergone at the end of the language course.(Frank, 2012)

In summative assessment, the teacher and students have obtained the cumulative snapshot to evaluate the overall progress and report on students' achievement at periodic times for certain language courses. Students are compulsory to take a summative assessment at the end of the course. Considering how prior work affected the outcomes is one strategy to give summative assessments more significance and beneficial for students. Teachers can ask students questions which as "How did your manner of working help or not assist?" and "What did we do in class that helped or did not help you in this test?"(Indriati, 2011)

Conducting summative assessments in language learning gives some benefits for foreign language learning. Summative assessment support curriculum improvement since it links students’ knowledge and learning objective. It depicts what program has to be changed, kept on going, or revised.(Hatipoğlu, 2010)For many students, summative assessment is an alarm to boost their motivation to study to achieve certain goals. The regular periodic times assist them in studying based on the timeline decided. For teachers, a summary of the class’s performance is provided through the summative evaluation. The teacher can assess their students' faults and strengths using their results.(States et al., 2018)The varicosity of summative assessment on language learning provides teachers with in-depth information.It depicts what was successful and what was not during the academic year. Teachers may change their curricula based on this information to improve learning requirements for the future school year.(Kebangsaan Malaysia

& Majid Khan Rana, 2020)Common summative assessments used in language learning are in- depth reports, projects, and personal evaluation papers.(Keus et al., 2019) Meanwhile, in the Indonesian educational context, summative assessment is close to the mid-term examination or Penilaian Tengah Semester (PTS) and the end-semester test or a final test also known as Penilaian Akhir Semester (PAS). Midterm and final exams are frequently used as assessment tools to gauge how well students have mastered the curriculum. However, midterm can serve as a dual-purpose tool, assessing students’ achievements while also helping students learn. In addition to a final exam, courses typically have one or more midterm exams distributed during the semester; these midterm exams may or may not be cumulative.(Clemmensen et al., 2013)

Moreover, the latest curriculum requires the integration of higher-order thinking skills into the teaching-learning process and assessment. Critical thinking, problem-solving, creativity and innovation, and collaborative and communicative ability are considered as Higher Order Thinking Skills (HOTS).(Abkary & Purnawarman, 2020) asserted that HOTS is crucial to promote the 21^stteaching-learning style. Teaching the 21^st thinking style assists students to be critical thinkers and problem solvers and survive in nowadays demands. The integration of HOTS in language assessment is a substantial way to actualize students as critical thinkers. The demands of education trends nowadays render educational stakeholders to oblige the integration of HOTS in assessment. Due to this, teachers must have the ability to integrate and apply HOTS thinking level in the form of a test.

(3)

RachmawatiAchadiyah, Muhammad Ulul Azmi HOTS Manifestation on English Summative Assessment In English assessment, the instruction of questions or stems has an essential role in determining what thinking level the assessment designer or teacher has utilized. As has been known that the taxonomy blooms have widely utilized the assessment designer or teacher to create and develop the question.(Santy et al., 2020)The questions that are asked to students have the greatest influence on their responses. To facilitate analysis between lower and higher- order thinking level in English assessment, the table is presented as follow:

Table 1. The LOTS and HOTS Question Example Operational

Verb LOTS Question Operational

Verb HOTS Question

Tell What does the text tell about?

What happened after?

Judge Is there a better solution to overcome the first problem?

What are criteria do you have in classifying peoples’. . . .

Who was the

protagonist character in the story?

Justify If you were . . . . what would you do?

Describe Describe personalities

of character . . . . In what position you are?

State your reason!

Describe the features of

……. Do you think is it a good

or bad thing?

Find Which paragraph tells the solution of the story?

How would you feel if?

Who was the main

character of the story? Argue Do you believe in the story. . . .?

Why and why not?

Distinguish Can you find the difference between fact 1 and fact 2 from the text?

How would you handle if.

. . .

What makes difference

between text 1 and text Please, provide your arguments on . . . .

(4)

Adapted from (Sulistyo, 2011) These concepts have thus far represented one aspect of human potential in terms of categorized thinking ability. In English assessment, LOTS requiresthe student to think in a simple way that seeks the answer from the text or what it calls the explicit meaning. In contrast, HOTS question requires students to analyze ata higher thinking level. They are asked to find the implicit meaning through a complex thinking process. They are not only seeking the implicit meaning but also justifying themselves toward the text.(Syahdanis et al., 2021) Hence, the implementation of HOTS in English summative assessment is highly recommended.

Evaluation of HOTS integration in English summative assessment has been explored in prior studies by (Ariffin et al., 2015).Arifin attempted to examine the percentage of LOTS and HOTS question use at the Ujian Khas Bahasa Inggris as an entrance exam for individuals seeking a diploma in English at one of Malaysia's regional colleges. According to the findings, several of the entrance exam items required to be altered in order to assure the accuracy of the tools used to assess candidates' language skills.A similar study was carried out by (Yuliati & Lestari, 2018), which aims to explain how students used their critical thinking abilities to solve HOTS-focused problems in instructional evaluation courses. The findings demonstrated that students still needed to develop their ability to think critically when responding to HOTS questions.

Importantly, the findings also discovered that students who were in high groups able to respond more critical than students who were in low and middle groups. In a similar vein, (Afriadi, 2021) uncovered the thinking level used in the national exam of SMK academic year 2017. LOTS and HOTS were integrated into the assessment. Nevertheless, the number of HOTS is less and needs to be increased. Reviewing these three and other prior studies, a study in portraying the hots manifestation in English summative assessment has rarely been studied directly. These prior studies did not depict the detailed instruction of how HOTS is narrated in the test. Due to this,

2?

Re-state Who do you think the trouble maker in this story?

Create Re-tell or Re-write the text with your own language!

What factors do you think give the most influence to the pollution?

Predict What might happen if it was happened in . . . . .

Name What is the best title for

this text? Summarize What can be concluded from the text?

Translate What was the phrase

meant ? Write the rest of the

video’s explanation!

Define What is the definition of

. . . based on the text? Prove Find the evidence of the complexity of . . . .

List List the main characters

of the story! Interpret What might the phrase mean for . . . .

(5)

RachmawatiAchadiyah, Muhammad Ulul Azmi HOTS Manifestation on English Summative Assessment this present study aims to propose a new approach to how HOTS question items are formulated in English summative assessment.

METHOD

This study was specifically designed witha qualitative approach which a case study as a design. The consideration of it is built upon a holistic and real-world perspective(Yin, 2018). The narration of how the teacher formulated the HOTS question on English summative assessment was depicted in this study.The source data of this study is the English summative test of seventh, eighth, and ninth in even final exams of 2022-203 academic year. All of the English tests were designed by LP Ma’arifSidoarjo. All schools under the supervision of LP Ma’arifSidoarjo must use this English test. Since, this study is portraying the hots question narration, the research instrument is documents. Bowen (2009) supports the notion that document analysis involves the methodical review and assessment of many types of documents, such as printed materials and electronic content supplied through computers and the Internet. Document analysis, similar to other qualitative research analytical methodologies, necessitates the examination and interpretation of data to extract significance, cultivate understanding, and produce empirical knowledge.

Chanda (2022) categorizes the materials utilized in the case study into seven sorts, with one of them being content analysis. Content analysis is used by researchers to identify particular words or concepts in texts or collections of materials. Researchers draw conclusions about the content, author(s), intended recipients, and even the cultural and historical context of texts by measuring and evaluating the existence, meanings, and connections of specific terms and concepts. Texts such as book chapters, essays, talks, interviews, newspaper headlines and articles, speeches, and others are utilized for content analysis. The utilization of this method will depend on the type of research and the specific research topics. As a result of these factors, content analysis is commonly used as a qualitative research technique. The process for data or content analysis in this study involves reviewing the final examination test, assessing the operative verbs and question narrative, and classifying them into higher order thinking skills (HOTS).

RESULT AND DISCUSSION

Given that the main focus of this study is the HOTS question narration, we will solely analyze the HOTS question narration by assessing the operational verbs, thinking level employed, and question narration. The specifics are illustrated as follows:

Table 2. The Seventh Grade

Question’s Number Operational Verb Thinking level Question narration

19 choose C5 - Evaluation Choose the answer

below which matches the picture

20 Choose C5 – Evaluation Choose the answer

below which

(6)

matches the picture

21 Choose C5 – Evaluation Choose the answer

22 choose C5 – Evaluation Choose the answer

The English summative test of seventh grade consisted of two types of questions. They were multiple-choice and short essays. There were twenty-five questions of multiple-choice and five short essays. Nonetheless, only four questions were indicated as HOTS questions or 16% of the multiple-choice questions. These four questions have the same style which uses choose as an operational verb. The test designer provided different pictures to inquire students into deciding what best expressions related to the picture. The essay question also did not provide students with HOTS questions. The five essay questions inquire students to supply the missing vocabulary, arrange the jumbled phrase into a good sentence and translate the dialogues from English to Indonesian. This test did not comply with the test rules of Indonesian requirements, which demanded the test designer to integrate HOTS in the assessment. According to (Kurniawan: 2020) HOTs questions must at least be assessed 50% of the number of questions.

This phenomenon must be investigated and overcome. Nurhatmanti et al. (2021) emphasized the need of cultivating and implementing critical thinking as a higher-order thinking skill in both teaching and evaluation processes at the elementary level. In their study, Pratiwi et al. (2019) discovered that three English teachers at a high school in Bali incorporated 5.4% of questions that required higher-order thinking skills (HOTS) in the summative test. The inability of English teachers to create questions that require higher-order thinking skills (HOTS) relates to the

scarcity of HOTS questions.

(7)

(8)

220 |

ENGLISH EDUCATION

Journal of English Teaching and Research

Table 3. The Eighth Grade

Question’s Number Operational Verb Thinking Level Question Narration

5 Mean C-5 What does the

picture mean?

6 Recommend C-5 What must we do

when we find the warning?

7 Mean C-5 What does the

picture mean?

9 Recommend C-5 From the text above

What should we do when we can’t attend the party?

11 analyze C-4 Why does the writer

send the card?

19 analyze C-4 What is the purpose

of the text above?

Essay

6 analyze C-4 What is the writer’s

purpose send the card?

10 Recommend C-5 What should we do

if we can’t attend the party?

As well as the seventh-grade paper test, the eighth grade has two types of questions they were multiple choice and short essay. It has twenty-five questions for multiple choice and ten questions for short essays. Nevertheless, only 32% of questions are designated as HOTS questions or eight questions. Giving recommendations, interpreting the picture, and analyzing the writer’s purpose was a higher thinking level applied in eighth grade. Other questions are categorized as lower and middle-order thinking levels. Students’ vocabulary and grammar accuracy became the majority of question types. These have been against by (Svendsen, 2015), who encapsulated that language is not merely learning vocabulary and grammar; EFL students must be able to use language as a tool for communicating meaningfully as human beings in the 21^st century. In communicating meaningfully, an individual must comprehend and be able to

(9)

Rachmawati Achadiyah, Muhammad Ulul Azmi

| 221

HOTS Manifestation On English Summative Assessment process the information critically(Saptuti Susiani et al., 2018). Thus, it should be contemplated by the English teacher as the test designer to give more amount HOTS questions. Concerning that the integration of HOTS is not only in the teaching-learning process but also in the assessment aspect.

Table 4. The Ninth Grade Question’s

number Operational Verb Thinking level Question Narration

1 Analyze C-4 Dinar’s intention of writing

the letter is ….

2 Interpret C-5 According to the text, we

know that Diana is ….

3 Analyze C-4 From the underlined

sentence, we know that ….

4 Analyze C-4 According to the above

dialogue, we know that ….

5 Analyze C-4 From the drug label above,

we know that ….

6 Interpret C-5 The statement “cook

thoroughly” in the labels means ….

7 Giving suggestion C-5 Why do we use pepermint

to repel mice?

9 Analyze C-4 From the above text we

know that ….

12 Summarize C-5 From the text, we can

conclude that the product

….

15 Giving suggestion C-5 Why should the user

discard the product after ten days?

16 Analyze C-4 From dialog above, we

know that ….

(10)

222 |

ENGLISH EDUCATION

17 Compare C-5 Two different

statements in the dialogue

above about Tom

Cruise which have the same meaning are ….

18 Interpret C-5 The product above is used

for ….

19 Analyze C-4 From the label we know

that ….

20 Giving suggestion C-5 We use a fork when making

Anzac Biscuits for …

21 Giving Suggestion C-5 Why is it suggested that we add the lemon?

22 Giving Suggestion C-5 Why should we do the

instruction above?

24 Interpret C-5 “No Preservatives.” The

word “preservatives”

means ….

Essay

1 Compare C-5 Compare and find out the

similarities and differences between about both of the dialogue above!

2 Interpret C-5 The statement “You’re

absolutely right” in the dialogue 2 means ….

3 Justify Position C-5 “English is the most

important tool in globalization.”

Read the thoughts, then express your agreement or disagreement and give the reason why you agree or disagree!

(11)

| 223

HOTS Manifestation On English Summative Assessment

5 Create C-6 Write a short paragraph

using simple past tense!

6 Create C-6 Study the pictures and

make a good recipe based on the pictures!

From Table 4,It was found that the question of the English final test paper of ninth grade is dominantly HOTS question. Out of twenty-five multiple-choice choices, eighteen questions belong to HOTS questions or 72%. As well as the essay, five questions indicate HOTS from seven questions.

Question items 1,2,4,5,9,16, and 19 are types of analysis questions. The question narration is dominantly “ from the text, from the dialogue above, we know that”. This type of question requires students to analyze after reading and comprehending the text or dialogue.

They are asked to find the main idea of the text or the dialogue. As (Fitriani & Kirana, 2022) have conducted in their study analyzing belongs to a higher thinking level. Analysis questions require the student to comprehend the whole of the content and ask them to re-inform in another way. A personal letter was provided for answering questions items 1 and 2. Asking the writer’s intention in writing the letter and interpreting the receiver of the letter’s personality. The third question was about analyzing the phrase “ I hope she recovers quickly”, students are called to analyze what the sentence means and what the speaker intends to say when they are saying that sentence. The fourth question also utilized the daily dialogue and called for students to analyze what the dialogue was about.

As well as the fourth question the fifth question demanded students to comprehend, analyze, and select the appropriate statements relevant to the presented medication label. There were several pieces of information contained on the label and some chemistry terms. Students should have sufficient background knowledge and vocabulary knowledge to answer the fifth question correctly.

The analysis leads to the conclusion that The ninth-grade English test requires students to express and develop their higher thinking skills since the majority of questions cognitive domain apply higher thinking skills level. This result ties well with previous studies wherein (Santy et al., 2020), revealed that C5 and C6 are the most cognitive level which was utilized by English test designers to assess students’ English achievement for university students when COVID-19 pandemic. A similar result was also found by(Pratiwi et al., 2020), who concluded that when the tasks of the test were based on critical thinking skills, the students’ responses reflected the same manner. Pratiwi, et.al (2020) analyzed the English summative assessment of nine senior high schools in East Java and invited the English test designers to be interviewed. The findings were the school institutions assigned the English teachers to design and develop English assessments based on the HOTS.

CONCLUSION

This study was conducted to portray how high-order thinking skills manifested in summative assessment. It found that the ninth grade has the largest amount in providing students higher order thinking skills. Meanwhile, the seventh and eighth grades were indicated to assess students in low-order thinking skills. The ninth-grade English test employed C4, C5, and C6 as the thinking levels which various thinking styles such as making analysis, giving

(12)

224 |

ENGLISH EDUCATION

recommendations, giving suggestions, interpreting, making predictions, and creating. Therefore, students’ answers can be used as a benchmark to determine their ability to think critically. On the other hand, the seventh and eighth grades provided low-order thinking skills which more emphasized language patterns and vocabulary mastery.

Ultimately, it is crucial for EFL teachers, who are responsible for designing the test, to incorporate questions that require higher-order thinking skills (HOTS) in both the teaching and assessment processes for each high school grade. At the primary level, the incorporation of Higher Order Thinking Skills (HOTS) can be introduced to enhance students' abilities in analysis, creative thinking, and problem-solving. The integration of linguistic components, language proficiency, and higher-order thinking skills (HOTS) represents an enhancement in the evaluation of English language proficiency. To enhance future study, it is strongly recommended that future researchers undertake a more comprehensive examination of EFL teachers' expertise and proficiency in formulating higher-order thinking skills (HOTS) questions for English summative assessments.

REFERENCES

Abkary, N. S., & Purnawarman, P. (2020).Indonesian EFL Teachers’ Challenges in Assessing Students’ Higher-Order Thinking Skills (HOTS). 509(Icollite), 482–489.

https://doi.org/10.2991/assehr.k.201215.076

Afriadi, R. (2021). Implementation of Levels of Thinking Skills and Communicative Language.

5(2019), 57–68.

Ariffin, K., Muhamad, A., Zamri, L., Liza, N., & Kadir, A. (2015). An investigation of the frequency of HOT and LOT of Bloom taxonomy in the diploma English entrance exam. The Asian Journal of English Language & Pedagogy ISSN, 3, 228–241.

Clemmensen, L. H., Sliusarenko, T., Christiansen, B. L., & Ersbøll, B. K. (2013). Effects of mid-term student evaluations of teaching as measured by end-of-term evaluations: An emperical study of course evaluations. CSEDU 2013 - Proceedings of the 5th International Conference on

Computer Supported Education, September 2016, 303–310.

https://doi.org/10.5220/0004353503030310

Fitriani, D. H., & Kirana, D. P. (2022). Higher Order Thinking Skill (Hots) in English Language Textbook in Senior High School. International Journal of Research on English Teaching and Applied Linguistics, 2(2), 55–66. https://doi.org/10.30863/ijretal.v2i2.2455

Frank, J. (2012). The roles of assessment in language teaching. English Teaching Forum, 32.

https://files.eric.ed.gov/fulltext/EJ997527.pdf

Hatipoğlu, Ç. (2010). Summative evaluation of an English language testing and evaluation course for future English language teachers in Turkey. Elted, 13(December 2010), 40–51.

Indriati, T. (2011). EDUCATIVE ASSESSMENT IN ENGLISH LANGUAGE TEACHING by THE GRADUATE PROGRAM IN ENGLISH LANGUAGE STUDIES.

Kebangsaan Malaysia, U., & Majid Khan Rana, A. (2020). Exploring Summative Assessment and Effects: Primary to Higher Education Serious game engagement’s model View project Virtual Museum View project Kashif Ishaq. March 2021.

https://www.researchgate.net/publication/350353443

(13)

| 225

HOTS Manifestation On English Summative Assessment Keus, K., Grunwald, J., & Haave, N. (2019). A method to the midterms: The impact of a second

midterm on students’ learning outcomes.Bioscene, 45(1), 3–8.

Kurniawan, A. (2020). The Barriers in Integrating Higher Order Thinking Skills in Indonesia Context.

Jurnal Bahasa & Sastra, 20(2), 139–144.

Kurniawati, S., & Sundawa, D. (2019). An Analysis of National Standard School Examination Items Based on the Characteristics of Hots (Higher Order Thinking Skills) Questions for the Main Items of K13-071 Academic Year 2016/2017 in Karawang Regency. International Journal Pedagogy of Social Studies, 3(2), 100–112.

Nurhatmanti, R., Lian, B., & Dedy, A. (2021). Analysis of Students’ Higher Order Thinking Skills (HOTS) in Natural Science Subjects Class V Elementary School. Indonesian Journal of Primary Education, 5(2), 208–213. https://doi.org/10.17509/ijpe.v5i2.37107

Pratiwi, N. P. W., Dewi, N. L. P. E. S., & Paramartha, A. A. G. Y. (2019). The Reflection of HOTS in EFL Teachers’ Summative Assessment.Journal of Education Research and Evaluation, 3(3), 127. https://doi.org/10.23887/jere.v3i3.21853

Pratiwi, N. P. W., Dewi, N. L. P. E. S., Paramartha, A. A. G. Y., Mustika, N., Nurkamto, J., Suparno, S., Anggita, N. L., Utami, P., Eka, N. L. P., Dewi, S., Gede, A. A., Paramartha, Y., Akib, E., &

Muhsin, M. A. (2020). Influence of questioning techniques in EFL classes on developing students’ critical thinking skills.Journal of Education Research and Evaluation, 5(2), 127.

https://doi.org/10.21070/jees.v5i2.752

Santy, N. P. L., Dewi, N. L. P. E. S., & Paramartha, A. A. G. Y. (2020). the Quality of Teacher-Made Multiple-Choice Test Used As Summative Assessment for English Subject. Prasi, 15(02), 57.

https://doi.org/10.23887/prasi.v15i02.25560

Saptuti Susiani, T., Salimi, M., & Hidayah, R. (2018). Research Based Learning (RBL): How to Improve Critical Thinking Skills? SHS Web of Conferences, 42, 00042.

https://doi.org/10.1051/shsconf/20184200042

States, J., Detrich, R., & Keyworth, R. (2018). Summative Assessment (Wing Institute Original Paper). Undefined, March. https://doi.org/10.13140/RG.2.2.16788.19844

Svendsen, B. A. (2015). Language, youth and identity in the 21st century: Content and continuations.

Language, Youth and Identity in the 21st Century: Linguistic Practices Across Urban Spaces, Appadurai 1990, 3–23. https://doi.org/10.1017/CBO9781139061896.002

Syahdanis, J. D., Sofyan, D., & Yunita, W. (2021). Analysis of Hots in English Teacher-Made Test.

Jurnal Basis, 8(2), 203–216. https://doi.org/10.33884/basisupb.v8i2.4479

Tosuncuoglu, I. (2018). Importance of Assessment in ELT. Journal of Education and Training Studies, 6(9), 163. https://doi.org/10.11114/jets.v6i9.3443

Yin, R. K. (2018). Case study research and applications: Design and methods. In Journal of Hospitality & Tourism Research (Vol. 53, Issue 5).

https://doi.org/10.1177/109634809702100108

Yuliati, S. R., & Lestari, I. (2018). Higher-Order Thinking Skills (Hots) Analysis of Students in Solving Hots Question in Higher Education. Perspektif Ilmu Pendidikan, 32(2), 181–188.

https://doi.org/10.21009/pip.322.10