ENGLISH TESTING AND LEARNING
AT THE VOCATIONAL SCHOOL:
STUDENT’S LIVED EXPERIENCE
A THESIS
Presented in Partial Fulfillment of the Requirements for the Master’s Degree
in English Language Studies
by Neny Mariani
Student Number:
056332027
THE GRADUATE PROGRAM IN ENGLISH STUDIES SANATA DHARMA UNIVERSITY YOGYAKARTA
STATEMENT OF ORIGINALITY
This is to certify the originality of the written study. No single part of this thesis but materials cited as references, either explicitly or implicitly, shows similarities or resemblance with any other work, and therefore to the best of my knowledge, no copyright violation is indicated. Hereby I declare that this thesis represents my own work. I assume full responsibility for the content of the work and bear all consequences it may result in.
Yogyakarta, 21st December 2009
Neny Mariani
LEMBAR PERNYATAAN
PERSETUJUAN PUBLIKASI KARYA ILMIAH
UNTUK KEPENTINGAN AKADEMIS
Yang bertanda tangan di bawah ini, saya mahasiswa Universitas Sanata Dharma Yogyakarta dengan
Nama : Neny Mariani Nomor mahasiswa: 056332027
menyatakan bahwa demi pengembangan ilmu pengetahuan memberikan kepada Perpustakaan Universitas Sanata Dharma Yogyakarta karya ilmiah yang berjudul:
“English Testing and Learning at the Vocational School: Student’s Lived Experience” yang saya tulis beserta perangkat terkait yang tersedia.
Dengan demikian kepada Perpustakaan Universitas Sanata Dharma Yogyakarta saya berikan hak dan wewenang untuk menyimpan, mengalihkan dalam bentuk media lain, mengolahnya dalam bentuk pangkalan data, mendistribusikan secara terbatas, dan mempublikasikannya di internet atau media lain untuk kepentingan akademis tanpa perlu meminta ijin dari saya maupun memberikan royalti kepada saya selama tetap mencantumkan nama saya sebagai penulis.
Demikian pernyataan ini saya buat dengan sesungguhnya.
Dibuat di : Yogyakarta
Pada tanggal: 21 Desember 2009
Yang menyatakan,
Neny Mariani
ACKNOWLEDGMENTS
First of all, I wish to offer my biggest adoration to the Almighty, who has
granted me an opportunity to pursue further education and kept sending so many
wonderful people I learn from and work with. I am so grateful for all the blessings,
inspiration and never-ending guide from nothing to something, prior to, during, and
upon completion of this master’s thesis. I guess there will never be bigger celebration to
mark the accomplishment of this project than expressing the words of glory, "Praise be
to Allah, the Cherisher, Sustainer of the worlds!". Also, I do not want to miss this
opportunity to express a word of honor to my parents, from whom I received my first
education in my earliest years, for raising me up and introducing the Most Gracious and
Merciful.
Although the content of this thesis is essentially and eventually my own
responsibility, this work by all means results from a great number of contributors.
Therefore I wish to take this opportunity to acknowledge and appreciate all kinds of
contribution which I have received directly and indirectly from those whose names I will
not be able to print out here one by one.
First of all, I would extend my particular gratitude to my adviser, Pak Bis (Dr J
Bismoko), for his great dedication and allowing me to get an easy access for correction
and consultation. It would almost be impossible to do this research project without the
existence of such a prominent figure behind the preparation, implementation and
presentation of the work.
I have been largely indebted to all the lecturers and everyone in the graduate
program of this university. I would like to thank the Graduate Program Director, Prof.
Dr. Augustinus Supratiknya for legalizing this report. I thank the Board of Examiners,
Pak Muk (FX. Mukarto, Ph.D), Pak Dwi (Dr. B.B. Dwijatmoko, M.A), Pak Alip (Dr. Fr.
B. Alip, M.Pd, M.A.) for helping me finalize this work with the thorough examination. I
also thank the rest of the lecturers, Pak Barli (Drs. Barli Bram M.S.), Bu Novi (Dr.
Novita Dewi, M.S., M.A. (Hons)), and Pak Pomo (Prof. Dr. Soepomo Poedjosoedarmo)
for their "shower of sparks" and "drops of dews".
For the financial and non financial support as well as attention, my gratitude goes
to LPMP Yogyakarta, including Bu Anna (Dra. Anna Maria Hendraswari, M.A.), Pemda
Kabupaten Kulon Progo (including Pak Ridwan) and Dinas Pendidikan Kabupaten
Kulon Progo (including Pak Yatiman ).
Next, I would express my sincere gratitude to my research participants, my
fellow English teachers and students, for every word they have contributed, their
honesty, and the time and effort they have spent to reply to my enquiries. For me, they
have been sunshine in the rainy day and water in the desert. I would like to express my
thankfulness to everybody in the school where I spend wonderful years, SMK Negeri 1
Pengasih, which includes the school principal Drs. H. Rumawal, my colleagues and
students, for the encouragement, challenge, spiritual, technical and financial support.
I thank all authors and researchers whose work is very valuable and contributive
to my project. I have made every effort to trace all the resources the best I could, and I
hope I do not miss a thing.
To all my fellow students and my acquaintances, I would say a big thank-you for
the share of knowledge and feelings. All the years when we were involved in such tough
and fierce discussions, arguing for and against, sharing laughter and tears have been so
precious to me. I will always miss the time when the feeling of friendship was that
strong.
Upon completion of the work, I would like to dedicate this thesis to the
Motherland and to the world. I realize this work will be more meaningful it gains
attention from wider audience. I look forward to and welcome any response coming
from all directions.
Yogyakarta, 21st December 2009
Neny Mariani
TABLE OF CONTENTS
TITLE PAGE
APPROVAL PAGE i
DEFENSE APPROVAL PAGE ii STATEMENT OF ORIGINALITY iii STATEMENT OF LICENCE AGREEMENT iv ACKNOWLEDGEMENTS v TABLE OF CONTENTS viii
LIST OF FIGURES xi
LIST OF TABLES xii
LIST OF ABBREVIATIONS xiii
ABSTRACT xiv
INTISARI xvi
CHAPTER I INTRODUCTION 1
A. BACKGROUND 1 B. PROBLEM IDENTIFICATION 5
C. PROBLEM LIMITATION 10
D. PROBLEM STATEMENT 11
E. RESEARCH GOALS 12 F. RESEARCH BENEFIT 12 CHAPTER II LITERATURE REVIEW 14
A. THEORETICAL REVIEW 14 1. Assessment 14 2. Language testing, English testing and backwash effects 18 3. Learning and learning autonomy 29 4. Language learning and language learning autonomy 34 5. Vocational school 37 6. Experience and attitude 39 7. Qualitative research and methodologies 40 B. THEORETICAL FRAMEWORK 46 CHAPTER III METHODOLOGY 55
A. RESEARCH DESIGN 55
B. DATA SOURCES 60
C. TYPES OF DATA 62
D. DATA COLLECTION TECHNIQUES 63
E. DATA PROCESSING PROCEDURE 66
1. DATA COLLECTING 66
2. DATA ANALYZING 67
F. TRUSTWORTHINESS OF RESEARCH RESULTS 68
CHAPTER IV ANALYSIS RESULTS 73
A. DATA AND SOURCE SELECTION 73
B. INSTRUMENT VALIDATION AND ACCOUNTABILITY 78 C. DATA GATHERING, RECORDING AND PRESENTATION 80 D. DATA ANALYSIS ACTUALIZATION 85 . E. DISCUSSION: DESCRIPTION AND INTERPRETATION 106
1. Student participant portraits in English testing and learning 106
2. Thematic description and illustrative quotations 115
3. Participants’ meaning of testing 129 CHAPTER V CONCLUSIONS, IMPLICATIONS, AND RECOMMENDATIONS 144
A. CONCLUSIONS 144
B. IMPLICATIONS 148
C. RECOMMENDATIONS 149
BIBLIOGRAPHY 151
APPENDICES 158
Appendix 1 Interview probes 158
1.1 Questions to students 158
1.2 Questions to teachers 158
Appendix 2 Coding blue print 159
2.1 Participant: students 159
2.2 Participant: teachers 160
2.3 Observation on test implementation 161
2.4 Inductive codes 161
Appendix 3 Observation data of events and behavior from official meetings and environs 162
3.1 Official meeting 1 ( external) 162
3.2 Official meeting 2 ( external) 163
3.3 Official meeting 3 ( external) 165
3.4 Official meeting 4 (internal) 165
Appendix 4 Observation data of events and behavior from test sites 166
4.1 Classroom test 1 166
4.2 Classroom test 2 166
4.3 Classroom test 3 167
4.4 Classroom test 4 167
4.5 Classroom test 5 167
4.6 High-stakes test 1 168
4.8 High-stakes test 3 170
Appendix 5 Observation data of physical and non physical environs 171
5.1 Situation prior to classroom test 1 171
5.2 Situation prior to classroom test 2 172
5.3 Situation prior to high-stakes test 1 172
5.4 Situation prior to high-stakes test 2 173
5.5 Situation after high-stakes test 1 174
5.6 Situation after high-stakes test 2 174
5.7 Situation after high-stakes test 3 174
Appendix 6 Researcher’s diary 175
Appendix 7 Interview data 183
7.1 Student Atik 1 183
7.2 Student Atik 2 191
7.3 Student Siti 1 201
7.4 Student Siti 2 207
7.5 Student Endang 214
7.6 Teacher Iin 1 221
7.7 Teacher Iin 2 226
7.8 Teacher Nanik 1 238
7.9 Teacher Nanik 2 245
Appendix 8. News-stories and information 249
8.1 Radio news 249
8.2 Television news 250
8.3 Printed news 257
8.3.1 Printed news 1 257
8.3.2 Printed news 2 258
Appendix 9 Documents 259
9.1 Official document 1 259
9.2 Official document 2 260
9.3 Student Endang’s answer sheet 261
9.4 Student Atik’s work 262
9.5 Student Atik’s school report 263
9.6 Classroom test sheet 266
9.7 High-stakes test booklet 267
9.8 Test-result report 269
9.8.1 Score report from high-stakes test 1 269
9.8.2 Score report from high-stakes test 2 270
LIST OF FIGURES
1. Figure 1.1 Tripartite partnership model 6
2. Figure 1.2 First, essential stage in learning enhancement partnership 7
3. Figure 1.3 Learning and testing relationship 8
4. Figure 1.4 Testing and stakeholders 9
5. Figure 2.1 Communication in testing environment stage 1, 2, 3 21 6. Figure 2.2 Bachman’s categorization of the use of tests 25
7. Figure 2.3 Brown’s language test type categorization 27
8. Figure 2.4 McNamara’s language test type categorization 27
9. Figure 2.5 Test type categorization 29
10. Figure 2.6 Testing in the language program toward independent learning 47 11. Figure 2.7 Positioning English testing at the Vocational School 50 12. Figure 2.8 English testing aspects categorization 52 13. Figure 3.1 From research questions to research results 59
14. Figure 3.2 Types of data 62
15. Figure 3.3 Data processing procedure 66
16. Figure 4.1 Student’s lived experience in English testing and learning
at the vocational school seen from various angles 79
17. Figure 4.2 Data processing stages 87
18. Figure 4.3 High-stakes test and thematic description, gleaned from
secondary data sources 118
19. Figure 4.4 Different interests within testing 140
LIST OF TABLES
Table 2.1 Theoretical concept map 54
Table 3.1 Research specification 58
Table 4.1 Student participant data 76
Table 4.2 Teacher participant data 76
Table 4.3 Observation blueprint 81
Table 4.4 Sample data observation 1 82
Table 4.5 Sample data observation 2 83
Table 4.6 Probing interview questions 84
Table 4.7 A priori coding inventory for student participants 85
Table 4.8 Sample corpus of data 88
Table 4.9 Sample tabulating table 96
Table 4.10 Semi-ordered list of responses to the open-ended question
“Bagaimana perasaanmu sebelum, pada saat, dan setelah
mengikuti ulangan harian?” 101
Table 4.11 Category of responses to the open-ended question
“Bagaimana perasaanmu sebelum, pada saat, dan
setelah mengikuti ulangan harian?” 103 Table 4.12 Emerging themes and categories from student participants’
narratives and behavior 104
Table 4.13 Components, categories and indicators constituting
environment around English testing implementation 116 Table 4.14 Emergent categories from primary and secondary data 117
LIST OF ABBREVIATIONS
CARLA: Center for Advanced Research on Language Acquisition
Depdiknas: Departemen Pendidikan Nasional
EFL: English as a Foreign Language ESL: English as a Second Language ESP: English for Specific Purpose FHI: Family Health International LOTE: Language Other Than English MTs: Madrasah Tsanawiyah
RRI: Radio Republik Indonesia
SLA: Second Language Acquisition SMK: Sekolah Menengah Kejuruan
SMP: Sekolah Menengah Pertama
TOEIC: Test of English for International Communication TVRI: Televisi Republik Indonesia
UWE: University of the West England
ABSTRACT
Neny Mariani, 2009. English Testing and Learning at the Vocational School: Student’s Lived Experience. Yogyakarta: Sanata Dharma University, the Graduate Program in English Studies
In the world of English education, in both theory and practice, testing which is known as a means to measure learners’ abilities, continually poses big problems. In the national level, testing has also become the main issue of a continual debate and never-ending polemic. Despite all the shortcomings it may bring with it, testing is widely used since it serves enormous practical functions. In this situation, there is a need to promote learner-centered learning in that learning and learner’s factors should be taken into consideration in the use and design of English tests. This enhancement will accelerate learning autonomy; improve quality and sustainability of learning process and achievement.
This research aimed at describing, interpreting and explaining how the vocational school students experience English testing, revealing what testing means to them and what in their perspective constitutes an ideal English test. As progressive qualitative research, it was not meant to explain, predict, and control the student and teacher properties and behavior. Instead, it was to reveal, describe and understand better how the vocational school students experience English testing.
In order not to marginalize the participants as a result of the inquiry, this research adopted the advocacy/participatory approach by starting with the suppression and oppression issues and assumed that as the researcher I would proceed collaboratively. Data were taken from an Indonesian vocational school with three students (primary) and two English teachers (secondary) as data acquisition sources. Data collection instruments included observation, interview and document reviews. Verbal data, the participants’ narratives, were analyzed with content analysis comprising descriptive and interpretative. To improve trustworthiness of the research, data and methodological triangulation and thick description were implemented over the course of the study. As a researcher, I took an active participation and intervened by designing an on-going evaluation in order to advocate becoming more empowered, autonomous, and self-fulfilling.
The research reveals that despite the similar concerns among the student participants in testing, learning and achievement, there were unique emotional experience and learning experience. It also reveals emergent themes around the time of English testing which, in addition to the prefigured themes test backwash, qualities, test content, learning and autonomy, includes status of test, fairness, time allocation, achievement, competitiveness, preparedness. In addition, my interpretation of what testing meant to the students as well as teachers shows that there was a shift of concerns among teacher participants in different types within
different contexts of testing. In the context of classroom testing, the teacher participants had different concerns from the student participants, validity and
reliability, while the students were concerned about learning and achievement. In the context of high-stakes testing, however, teachers and students shared similar concerns, learning and achievement. That classroom testing potentially created a point of clash and hurt student learning as its implication was due to the different concerns in testing between the teachers and students. That high-stakes testing created a gentleman’s agreement was due to the similar concerns. These research results will lead to better understanding for the participants, students and teachers, researcher as well as the audience. The participant improved awareness of testing and learning and the researcher and audience improved emphatic understanding of the participant experience, would consequently help facilitate learner’s learning need and in turn lead to healthy learning process and better results such as in particular to design a more progressive test and make a wiser decision on why, when, how to use an educative test as assessment.
INTISARI
Neny Mariani, 2009. English Testing and Learning at the Vocational School: Student’s Lived Experience. Yogyakarta: Universitas Sanata Dharma, Kajian Bahasa Inggris
Di dalam pengajaran bahasa Inggris, baik dalam teori maupun praktek, tes yang dikenal sebagai alat ukur kemampuan siswa masih menghadapi berbagai masalah. Di kancah nasional, tes juga menjadi perdebatan dan polemik yang tak kunjung selesai. Dengan segala kekurangannya, tes masih digunakan secara luas karena masih banyak fungsinya. Dalam situasi seperti ini, diperlukan upaya peningkatan pembelajaran yang berpusat pada pemelajar dalam hal faktor-faktor kegiatan belajar dan siswa perlu diperhitungkan dalam penyusunan dan pemakaian tes. Penekanan ini akan membawa dampak percepatan pembentukan kemandirian belajar, meningkatkan kualitas dan keberlangsungan proses serta keberhasilan proses dan prestasi belajar.
Penelitian ini mencoba menggambarkan, memahami, dan menjelaskan bagaimana pengalaman siswa SMK dalam tes Bahasa Inggris dan mengungkap apa makna tes bagi mereka serta apa pandangannya yang merupakan tes ideal itu. Karena bersifat kualitatif, penelitian ini tidak dimaksudkan untuk menerangkan, memprakirakan dan mengendalikan unsur-unsur yang ada pada siswa dan guru, termasuk perilakunya. Namun, penelitian ini akan membuka tabir, menggambarkan dan memahami secara lebih baik bagaimana pengalaman siswa SMK terhadap tes Bahasa Inggris.
Untuk tidak mengesampingkan peserta sebagai akibat dari penelitian ini, pendekatan advocacy/participatory digunakan dengan dimulai adanya isu penekanan dan penindasan, dan anggapan bahwa peneliti akan melakukan tugasnya dengan kompromi. Data diperoleh dari sebuah SMK di Indonesia dengan sumber data dari tiga siswa (data primer) dan dua guru Bahasa Inggris (data sekunder). Alat pengumpulan data berupa observasi, wawancara, dan telaah dokumen. Data verbal yang berupa ucapan peserta dianalisis dengan analisis isi yang meliputi analisis deskriptif and interpretatif. Untuk meningkatkan keterpercayaan hasil penelitian, triangulasi data dan metodologi serta thick description dilakukan selama proses dan pelaporan hasil penelitian. Sebagai peneliti, saya terlibat aktif mengambil bagian dan melakukan intervensi dengan cara melakukan refleksi kegiatan yang terus menerus dalam membantu menyuarakan agar lebih berdaya, mandiri dan dapat memenuhi kebutuhan belajarnya sendiri.
Penelitian menunjukkan bahwa meskipun di antara sesama siswa terdapat persamaan dalam kepentingan yaitu belajar dan prestasi, mereka memiliki perbedaan dalam pengalaman emosional dan pengalaman belajar. Penelitian ini juga menunjukkan munculnya tema-tema baru seputar pelaksanaan tes yaitu
selain dampak, kualitas, isi tes, belajar dan kemandirian, juga mencakup status tes, keadilan, alokasi waktu, prestasi, kebersaingan, dan kesiapan. Sebagai tambahan, interpretasi saya mengenai apa makna tes bagi siswa dan guru SMK menunjukkan adanya pergeseran kepentingan guru bila terdapat perbedaan jenis dan konteks pengujian. Dalam konteks tes berbasis kelas, guru mempunyai perbedaan kepentingan dengan siswa, yaitu validitas dan reliabilitas; sementara itu siswa mempunyai kepentingan belajar dan prestasi. Namun dalam konteks tes skala besar, guru dan siswa mempunyai kepentingan yang serupa, yaitu
belajar dan prestasi. Bahwa tes berbasis kelas berpotensi menimbulkan konflik dan menghambat proses belajar siswa sebagai implikasinya itu terkait adanya perbedaan kepentingan dalam tes antara guru dan siswa tersebut. Tes berskala besar menimbulkan adanya saling pengertian itu karena kesamaan kepentingan. Hasil penelitian ini akan memberikan pemahaman yang lebih baik bagi peserta, siswa dan guru, peneliti dan khalayak. Meningkatnya kesadaran diri peserta terhadap pengujian dan pemelajaran, dan rasa empati dari peneliti dan khalayak akan membawa ke arah proses belajar yang sehat dan hasil yang lebih baik, dan membantu mengakomodasi kepentingan dan kebutuhan belajar siswa, misalnya dengan merancang tes yang bersifat progresif serta melakukan kebijakan tentang mengapa, kapan, bagaimana menggunakan tes sebagai alat penilaian yang mendidik.
CHAPTER I
INTRODUCTION
To identify and expose the main research problems elaborately, in this
introductory chapter I will examine concepts and issues in English education,
specifically English testing in the global as well as local context. I will begin with the
background of the study and consecutively discuss the problem identification, problem
limitation, problem statement, research goals, and research benefits.
A. BACKGROUND
Over the last few decades, as writers say, language test theory has put a great
deal of emphasis on three major areas. First is the nature of language; that is when we
talk about skills and language components. Next is the nature of language use; that is
when we talk about authenticity. Third is the nature of measurement; that is when we
talk about validity and reliability. This can be seen from the work of, take for example,
Bachman (1995) and McNamara (1996). It is not surprising then that in practice test
developers and specialists are reportedly deeply concerned with anything related to
defining language competence, understanding the factors which influence performance
on language tests and finding valid and reliable ways of measuring language competence
and development, and recently greatly worried about whether test materials and tasks
represent actual use of language for communication. In conclusion, we are excessively
concerned with the overt function of the test, as a means of measurement and
assessment, ignoring the covert but influential function, that is, as a learning experience.
It is clear for us that most test theories as proposed by Bachman (1995),
McNamara (1996) and Brown (2005) imply that they place test-takers as the objects
of the test. Test-takers are considered as “something we can do measurement upon”.
It is not clear whether it will give any benefits or harms to them. The definitions
imply that test is designed for the sake of other parties, while the primary
stakeholders, which are in fact the test-takers themselves, are totally forgotten. This
is what I call oppression, suppression and colonization. Up to this point, it is
obvious that so far we have been busy dealing with constructing a test for its own
sake and for its own right, rather than catering for student learning needs or
facilitating learning in the stage of evaluation within the teaching and learning
process. Whether the test construction has harmful or positive effects on learning
remains out of discussion and little or even almost nothing about learning has been
taken into consideration in the design and use of language tests in most EFL/ESL
courses. The lack of discussion and the absence of such consideration in the test
design and use is greatly regrettable. In relation to this, Hughes’ (2003:1) assertion,
showing the inadequacy in the test design and use, is worth quoting, “[…] a great
deal of language testing is of very poor quality. Too often language tests have
harmful effects on teaching and learning […]”.
In this problematic situation, we then need a review on the most current
educational and learning issues. Life-long education has been widely noted since
early times and Delors et al. (1998:20) reaffirm it by iterating that the concept of
education pursued throughout life recently emerges as one of the keys to the twenty
first century. Education, an ongoing process of improving knowledge, skills and
based, learning to know, learning to do, learning to be and learning to live together,
is said to have a fundamental role to play in personal and social development in that
it is an exceptional means of bringing about personal development and building
relationship among individuals, groups and nations.
Next, let us refer to the postmodernism theory, with its progressivism
philosophy, which says education should be directed to make learners empowered,
autonomous, and eventually become self-fulfilling individually and socially in their
own world. Finney (2002:73) puts it this way,
The purpose of education from the point of view of the process model is to enable the individual to progress toward self-fulfillment. It is concerned with the development of understanding, not just the passive reception of ‘knowledge’ or the acquisition of specific skills. The goals of education are not defined in terms of particular ends or products, but in terms of the
processes and procedures by which the individual develops understanding and awareness and creates possibilities for future learning. (Italicized words are my emphasis)
Here we should note that the key word to education is “self-fulfillment”, and
consequently we can say we need learner-centeredness, an approach to education
that aims at developing in each student a sense of responsibility for his or her own
learning (The University’s Strategic Plan, 1995 as quoted in Hall, 2000:2). In other
words it means the development of learner autonomy should be the heart of any
educational program, which certainly includes language teaching and learning.
To ensure the enhancement of learner-centered learning and the development
of learner autonomy as stated above, therefore, I felt it urgent to do research on a
particular topic so that learning, which is the most essential of all in education,
would be appropriately ensured, adequately addressed and consistently become the
main consideration in the whole process of designing and implementing educational
evaluation, including testing.
Test, which is in a general sense the main instrument in learner assessment,
is indispensable part of the ESL/EFL/ESP teaching and learning process, and it has
been my interest to do a research on. The choice of the topic was due to the
following reasons. First, as you observe, test as the main instrument of assessment
remains controversial and problematic in both theory and practice in that students’
learning needs and viewpoints in testing has been paid little attention. In addition, we
can refer to that of two prominent advocates of ESP Hutchinson and Waters
(1994:146), who realize and notice there is a shortage of sound theoretical or
empirical basis for ESP testing and lack of discussion or guidance on ESP testing:
“Far more research is therefore needed before we will really know what the
requirements of a good ESP test are”. Second, my personal and professional
background as an EFL teacher had driven me conduct numerous kinds of tests for
vocational school students in the local, regional as well as national level. Thus, in
Trochim’s words (2006:1), I had been living and grown with the phenomenon, or in
other words, it is my “direct experience of the phenomenon”. Therefore in relation to
the learning enhancement and the learner autonomy development as I have already
described, then I concentrated on the topic, English testing.
In relation to the topic, I believed, the focus of further research on testing
should be on learning factors and an understanding of the impacts of language
testing to learning, instead of language or test property analysis. The statement of
Chalhoub-Deville (2003:380) is worth quoting to confirm the idea,
Given current indications in language testing, the nature of the components underlying the L2 construct as well as the manner in which these components interact are issues that researchers need to address in the twenty-first century.
Indonesian vocational schools, whose mission is to provide a middle-class
workforce and to improve their competitiveness, are greatly concerned with their
EFL or more specifically ESP teaching and learning. A policy shift from a
supply-driven system to a demand-supply-driven system guided by labor market signal, curriculum
reform, developing syllabus design, improvement on learning resources, teacher
training, the adoption of the international standardized test, periodical debate contest
are clear illustrations of the struggle. Nevertheless, the major common problem
remains. Having spent their years of schooling, a number of students are still
considered ‘unsuccessful’ English learners, not only emerging with a very low
proficiency but also indicating low motivation and having a high dependence on
teachers in their learning.
Thus, despite tremendous progress in curriculum reforms and policies, two
significant things, which are implications of the curriculum reforms itself, remain
untouched: understanding of student’s learning needs and experience in testing. In
such a situation, to do research on English testing and learning is highly relevant
since the results will surely be contributive to learning enhancement and
sustainability and eventually help overcome the persistent problems in education.
B. PROBLEM IDENTIFICATION
The problematic situation, which takes place in both theory and practice,
implies a kind of oppression or colonization upon learners, and then it raises
questions as “If a test is a measurement, why measure?, “Is measuring the ultimate
goal in all education?”, ”Where is in language testing the room for learning
test totally inauthentic in language testing?”, “Does something inauthentic have no
meaning and bring bad impacts to learning?”, “If a test is an instrument for us to
use, why is it not designed as user-friendly?”. With all these problematic questions,
it is clear that an educational reform is urgent to do, and further research on English
testing is the right way to implement the reform.
To get out of the problematic situation in the world of education, it is a
good idea to refer to what Delors et al. (1998: 27) state,
The main parties contributing to the success of educational reforms are, first of all, the local community, including parents, school heads and teachers; secondly, the public authorities; and thirdly, the international community. Many past failures have been due to insufficient involvement of one or more of these partners. (bold-faced is my emphasis)
From here, it is obvious that the involvement of all parties in the reform is
obligatory; local community’s contribution takes precedence over the others, and
that the absence of any party in the involvement will lead to another failure. Figure
1.1 presents a clear illustration of this.
First Party
Local Community
Educational Reforms
Second Party
Public Authority Third Party
International Community
Figure 1.1 Tripartite partnership model in education reform (visualizing the idea of Delors et al., 1998 )
Furthermore, for the success of educational reforms, Delors et al. (1998:27)
also necessitate a dialog among the parties as they point out,
Local community participation in assessing needs by means a dialogue with the public authorities and group concerned in society is a first, essential stage in broadening access to education and improving its quality. (my emphasis)
This is in line with learner-centered teaching orientation which begins with the
premise that the learner is the central reference point in decision making (Tudor,
1997). Thus, if we want to enhance learner-centered learning and to know the
requirements of a good test, learning-needs assessment is significant to do, and
having a dialog with test stakeholders to ensure sufficient involvement of pu
blic authority and local community is highly relevant. This can be illustrated in
Figure 1.2.
Local Community:
Students,parents, teachers, school heads
Educational Reform:
Enhancing learner-centered learning
Public Authority:
Government, researchers
Figure 1.2 First, essential stage in learning enhancement partnership (developed from the idea of Delors et al., 1998)
One of the logical consequential applications of this theory is that it is urgent
to do research which cares about the learners’ concern by having a dialog with
students and teachers as the local community party. Again, Tudor (1997) confirms
students inform them with their knowledge and experience; and learning is more
successful if methodology and study mode take into account student preferences and
characteristics. In this research, therefore I as the researcher representing the public
authority party will facilitate what they want to say in order to implement the reform
effectively.
To begin with, let us study how learning and testing is interrelated.
Here, Figure 1.3 shows that in the context of educational program, testing can be
seen as a learning environment, which constitutes one of factors enhancing learning.
Factors enhancing learning Learning environment Testing
Figure 1.3 Learning and testing relationship
There must be a great number of factors enhancing learning, one of which is learning
environment. There must be a lot of things which constitute learning environment.
And testing is just a part of it. By all this I mean that if language testing is highly
problematic, it is in itself a bad learning environment which in turn may hurt
learning.
Further, as we are all aware of, language testing as an important phenomenon
in language learning has a number of stakes-holders or parties who are concerned in
it, ranging from local community such as the student test takers, public authority
such as the government to international community, such as authors and test
specialists. This is illustrated in Figure 1.4.
Government
Test specialists
Schools
LANGUAGE TESTING
Society
Educators, Teachers
Parents Students
Figure 1.4 Language testing and stakeholders
Since students and teachers, the most concerned stakeholders, deserve
benefits from testing, instead of colonization and oppression, time has come to
include their participation in the reform. It is necessary to do a research project
which enables to amplify and hear their voices as the starting point to shift the focus
from dealing with the nature of language, the nature of language use, the nature of
measurement, to understanding of the nature of learning.
In order to investigate the problems thoroughly, it is necessary to employ a
research paradigm which puts its interests in the participants’ narratives and enables
to go deeper by digging out their experience to the research topic; a research with
critical theory as the theoretical perspectives, which are concerned with empowering
human beings to transcend the constraints placed on them by race, class and gender
approach is proved to be the right answer. Unlike quantitative research, which has
interests in making generalization, qualitative research, as literature indicates, does
not mean to make any generalization, for human beings cannot be generalized. A
number of methods are available to choose, ranging from, observation, open-ended
interviews, semi-structured interviews, structured interviews and documentary
analysis. More detailed description on the progressive qualitative research methods
will be available in the next chapter.
C. PROBLEM LIMITATION
To anticipate any probable problem which most likely arose from the
research due to various constraints such as the limited time, cost and my own
personal capacity, it was necessary to set up boundaries by using delimitation and
limitation as parameters for my study (cf. Creswell, 2003:147). I had to narrow down
the coverage (scope) and methodology of this study and identify any potential
weakness of the study.
From the numerous problems in language testing and learning already
identified, I wanted to focus my study more specifically on English testing and
learning. I limited the scope of English test here only to that conducted at the
vocational school consisting of classroom test and high-stakes tests with limited
aspects of goal, process and material. The stakeholders were limited to students and
teachers, with the reason that they represented local community to include in the
dialog and while they were the key factor in the reform, so far their involvement was
very low and inadequate as already indicated. To put it in another way, the
regretfully ignored in both research and practice. The scope for teachers was limited
to those who teach English at the vocational school, while the scope for students
here is only to those who major in business studies at the vocational school. As to
methodology, I limited to those which are applied in the qualitative research. I chose
phenomenology as the strategy of enquiry with narrative designs. Data collecting
instruments such as interview, observation and document review would be
applicable.
The implication of this is that the research results would be interpretable
within the limit and the findings could be subject to other interpretations.
D. PROBLEM STATEMENT
Testing has been one of the most problematic areas in English education.
English testing has been used and designed for its own right and its own sake.
Learner’s learning factors have received less attention than they should. It is obvious
up to this point that in order to achieve the purpose of education as suggested by the
process model of curriculum in the progressivism philosophy, learning enhancement,
as already discussed, is urgent to implement. To ensure the learning enhancement,
one of the best ways is doing research in the problematic areas.
In order to be aware of and understand the students’ lived experience in
English testing at the vocational school, this research, therefore, aimed to answer
theoretically and empirically the following question, what is the vocational school
students’ lived experience in English testing and learning?. Therefore, throughout
the research we discussed and tried to seek the answer to how vocational school
E. RESEARCH GOALS
This research aimed to describe, and interpret, how vocational school
students experience English testing and what in their perspective constitutes a good
English test and to reveal the answer to the research question theoretically and
experientially. As qualitative research, it was not meant to explain, predict, and
control the participant properties and behavior (cf. Creswell, 2003:31). Instead, it
was to reveal, describe and understand better how the vocational school students
experience English testing and to help them understand their own perspective, in
order to advocate becoming more empowered, self-fulfilling, and autonomous,
which means they can do what they are supposed to do and they can do it on their
own when they learn (cf. Finney, 2002:73). The goal of this research was therefore
to describe and interpret the students’ lived experience in English testing around the
time they learn English.
F. RESEARCH BENEFITS
This research, in general, will surely be beneficial to the evolving body of
knowledge in English education, science and technology. First, it will contribute
further insights for responding to problems regarding the use and development of
English test not for its own sake but in relation to what ESL is concerned with:
learning; and second, it will help educational policy makers and test specialists in
relation to course design and more specifically to the use and design of English tests.
The research, which comprises the process and the report, is expected to
bring benefits to the participants, researcher, and audience. The research process
of what English tests mean to the students. In another way of saying then, a positive
change is expected to occur in that the results will enlighten marginalized students.
For the teacher and student participants, the results help them to understand their
own perspectives. Meanwhile the research report which consists of the description
and interpretation of what is the participants’ lived experience in English testing will
bring benefits to the audience. Specifically for teachers of English, it will help tackle
persistent problems in English teaching delivery and more particularly in testing. For
broader audience, it will lead to their empathic understanding, emancipation,
empowerment, autonomy and self actualization so that learning in general will be
more significant and sustainable.
CHAPTER II
LITERATURE REVIEW
The main purposes of this chapter are; first, to ensure the status of the
research topic, and second, to identify and clarify the research question theoretically,
how I will use it to account for the research problems and intended solutions. It
consists of two parts; theoretical review, in which I will try to show the complexity
of the research problem theoretically, and theoretical framework, which will be a
review of the answer to the research question and why so.
A. THEORETICAL REVIEW
Here I will examine central issues related with research subject and research
methodologies comprising constructs, concepts and their sub-categorizations.
Theories upon assessment, language testing, English testing, backwash effects,
learning, learning autonomy, language learning, language learning autonomy,
experience, attitude, vocational school and qualitative research methodologies will
therefore be reviewed. Each will include the what, why and how.
1. Assessment
To present the clearer definition of testing, let me begin with the context
where it takes place, i.e. assessment. The existence of assessment in an educational
program is beyond question. Rayment (2006:1) presents a number of definitions of
assessment, and I would bring three of them. According to him, the first definition
of assessment is
The process whereby evidence is obtained through the outcome of specific questioning such as tests and surveys, and used to determine results based on the findings of such methods. These may also define the status or value of an event, thing or person’s abilities based on performance or importance.
Within an educational context, it also means “a method of inquiry to determine
the extent of learning” and “the process of evaluating students.”
In comparison, I will refer to Indonesian Directorate for Vocational
School affairs, Direktorat Pembinaan SMK, and Department of National
Education, Depdiknas. This is so as the two institutions deal with all vocational
and educational matters in Indonesia, where this research takes place.
Assessment, which is translated into Indonesian as “penilaian”, according to
Direktorat Pembinaan SMK (2008:49), is
serangkaian kegiatan untuk memperoleh, menganalisis, dan menafsirkan data tentang proses dan hasil belajar peserta didik yang dilakukan secara sistematis dan berkesinambungan, sehingga menjadi informasi yang bermakna dalam pengambilan keputusan
which means a series of activities to obtain, analyze and interpret data on the
process and achievement of student learning which is systematically and
continually implemented so as to become meaningful information in decision
making. Similarly, Depdiknas (2003:12) defines it as:
[...] penerapan berbagai cara dan penggunaan beragam alat penilaian untuk memperoleh informasi tentang sejauh mana hasil belajar siswa atau ketercapaian kompetensi (rangkaian kemampuan) siswa. Penilaian menjawab pertanyaan tentang sebaik apa hasil atau prestasi belajar seorang siswa
which is equivalent to:
the application of a variety of methods and employment of various means to obtain information on to what extent students get their learning results and achieve their competencies or a series of abilities. Assessment answers how well the student achievement is.
By these definitions, I can tell that assessment is actually a process or activity
employing various means with the purpose of obtaining information on student’s
performance. This is the fundamental meaning and purpose of assessment.
Hughes (2003) and Harmer (2007:379) make a difference between
summative and formative assessment. Summative assessment is defined as the
kind of measurement that takes place to round things off or make a one-off
measurement such as the end-of-year tests or the big public exams which many
students enter for. Formative assessment, on the other hand, is said to relate to
the kind of feedback teachers give students as a course is progressing and which,
as a result, may help them to improve their performance.
Indonesian government through PP Tahun 2005 (2005 Government
Regulation) No. 19 Article 63-70, pp 19-22 presents the national standard
educational assessment and includes three types of assessment to be conducted
by three different parties: (individual) educator/teacher, educational institution,
and the government (official). Each serves different functions. That which is
conducted by the individual educator aims to monitor the process, progress of
and improve learning achievement. The second type of assessment is to evaluate
the extent to which the graduate competency standard can be achieved. The third,
which is conducted by the government aims to nationally evaluate the extent to
which the graduate competency standard for particular subjects – one of which is
English - can be achieved.
Added to the basic purpose of monitoring school performance as such, as
Maughan (2009:1) notices, there are now aims of raising standards by improving
higher order skills. With the new aims, as she explains, assessment can now be
designed to impact on the actual learning that takes place in schools rather than
purely to measure the outcomes of learning. Similarly, McNamara (1996)
recommends that assessment needs to be integrated with the goals of the
curriculum and have a constructive relationship with teaching and learning.
Wiggins (1998:1) makes a similar point that the ideal of assessment is “an
educative assessment system that is designed to improve, not just audit, student
performance” and that “elements of educative assessment are first, authentic
tasks and second, feedback, which is more than blaming and praising”.
From here, we notice that the change in emphasis is going on. Assessment
may be developed to produce a positive backwash on the teaching and learning
and in the long run to promote learning autonomy. The National Forum on
Assessment (1995) as quoted by North Central Regional Educational Laboratory
(2009:1) strengthens this idea by providing the more detailed description of
quality assessment,
High-quality assessment must rest on strong educational foundations. These foundations include organizing schools to meet the learning needs of their students, understanding how students learn, establishing high standards for student learning, and providing equitable and adequate opportunity to learn.
I fully support this and wish to emphasize the main key points “meet the learning
needs”, “understanding how students learn”. This is also in line with the idea
proposed by Hutchinson and Waters (1994) available in the next discussion about
the starting point of language learning and teaching.
2. Language testing, English testing and backwash effects
As already discussed, one instrument of assessment most widely
recognized is test. It is then necessary to search for what it means. Bachman
(1995:20), basing on Carroll (1968), states that “a test is a measurement
instrument designed to elicit a specific sample of an individual’s behavior”. He
continues to explain, “What distinguishes a test from other types of measurement
is that it is designed to obtain a specific sample behavior” (1995:20 – 21).
Meanwhile, language test is defined by McNamara (1996) as a procedure for
gathering evidence of general or specific language abilities from performance on
tasks designed to provide a basis for predictions about an individual’s use of
those abilities in real world contexts. So based on what those scholars state, I can
say that English test is a measurement instrument for gathering evidence of
general or specific English abilities of an individual.
The nature of testing, as McNamara (1996:4) further explains, has
changed quite radically over the years to become less impositional, more
humanistic, conceived not so much to catch people out on what they do not
know, but as a more neutral assessment of what they do. More than this, above
all, I wish to amplify that tests must be part of learning experiences for all
involved (Coombe and Hubley, 2008:2) and testing should always be seen as
part of the learning process (Riddell, 2003). Similarly, Brown (2005:252)
stresses the importance of looking at tests within the context of real, living
language programs. He argues that testing is not an isolated entity floating free of
any language teaching reality and tests “should never be treated as though they
“and should be done in as pleasant an atmosphere as is possible in the
circumstance”.
As one part of learning process and experiences, then – let me stop for
one-moment reflection – testing, then should first support learning facilitation
with a deep understanding of how people learn, and second, lead to learning
autonomy. This is most compatible with the recent philosophy of learning and
matches the ideal assessment criteria previously discussed. Wiggins (1998:21)
confirms, “[t]esting that is deliberately designed to teach and improve, not just
measure, is the essence of educative assessment.” Hughes (2003:2) adds that we
cannot expect testing only to follow teaching and rather, we should demand of it
that is supportive of good teaching and where necessary, exerts a corrective
influence on bad teaching.
I intend to show that in any testing event, the test must have and bring a
certain degree of authenticity in language use. Here, then I shall lead the
discussion on the position of tests and testing in English language learning,
referring to what Egbert et al. (1999:3-7) write for optimal language learning
environment.
First, as they say, learners have opportunities to interact and negotiate
meaning. Here I will argue that in English testing, students and educators are
interacting and negotiating meaning. Students try to understand and do the
instructions, decipher and answer the questions.
Second and third, learners interact in the target language with an
authentic audience and are involved in authentic tasks. Language testing is in
communication, from which they can learn something. Bachman (1995:2)
reminds us:
Unlike tests of other abilities or areas of knowledge, where we frequently use language in the process of measuring something else, in language tests, we use language to measure language ability.
In line with the statement, three possibilities will take place in the case of English
testing:
A. In an English test, we use English in communicating the instruction or
rubrics and the content
B. In an English test, we use languages other than English (LOTE) in
communicating the instruction or rubrics and use English in the content
C. In an English test, we use English and languages other than English in
communicating the instruction or rubrics and the content.
In the context of English education, A and C, I will argue, are basically
authentic, no matter what English ability or knowledge we want to measure. This
is so because both give opportunity to experience using English to communicate.
In the context of English education, where learners learn to communicate, testing
is by all means authentic practice where learners communicate with their
educators, teachers or test makers, and test administrator whether or not it totally
employs English as the medium of communication. Therefore, in my point of
view, regardless of how low the quality of an English test is said, in this context I
would say that all English tests have a particular degree of authenticity.
The fact that there is a communication process in English testing is
beyond question. Biagi (1992:149) describes communication as the act of
communication: sender, message, channel/medium and receiver, and
acknowledging scholars who describe the process of mass communication, she
explains: “A sender puts a message on a channel, which is the medium that
delivers the message to the receivers”. Referring to Biagi’s then, the phenomenon
of communication in English testing environment can be visualized as Figure 2.1.
Communicator Medium Communicant
Tester Testing Testee
Communicant Medium Communicator
Tester Test Work Testee
Communicator Medium Communicant
Tester Test Result Testee
Figure 2.1 Communication in testing environment, stage 1, 2, 3
(Adapted from Biagi’s Elements of Mass Communication, 1992)
In English testing, there may be sorts of communication such as
interpersonal communication, when it involves two people (e.g. teacher and
student). There is also a possibility what she terms mass communication to take
place. This is when, as she says, communication runs from one person or group
of persons through a transmitting device (a medium) to large audience (e.g.
between teacher and students). Figure 2.1 illustrates communication process
I would like to strengthen my argument that English testing in English
education is by all means authentic in itself, by adopting the ethnographic
framework of communication proposed by Hymes (1974) as quoted in
Wardhaugh (1992:245-246). The context of interaction in English testing is
characterized by the following variables:
Setting and Scene : taking a test/ an examination,
formal situation, at school
Participant : teacher/test maker and
student/test-taker
End : evaluation, sense of accomplish-
ment
Act of Sequence/Topic : language form, meaning and use
Key : serious
Instrumentalities/Modes : written and spoken
Norm of interaction and interpretation: rule-bounded
Genre : various
Fourth, learners are exposed to and encouraged to produce varied and
creative language. In English testing, students are exposed to varied and creative
language, in written as well as spoken forms. In a particular test, students are also
encouraged to produce varied and creative language.
Fifth, learners have enough time and feedback. In English testing, students
are given time allocation to do the test, and later they will get the feedback in the
form of scores as well as other information.
Sixth, learners are guided to attend mindfully to the learning process. This is
still questionable whether in any English testing, students are guided to attend
mindfully to the learning process so that learning becomes and remains sustainable.
Seventh, learners work in an atmosphere with an ideal stress/anxiety level.
English testing never takes place in isolation. It takes place in the context of
teaching, and like another test, it is an influential factor, since it brings psychological
situation, such as anxiety, distress, enjoyment, feeling of success and failure. Tests,
for example, are identified by Bailey in Ellis (2004:480) as one of the sources which
may cause anxiety. Ellis (2004:522) believes that anxiety arising out of tests and a
fear of negative evaluation is likely to have a debilitating effect on L2 learning,
nevertheless, he believes that it can also have a facilitating effect. Further, he says
“[h]ow anxiety affects learning will depend on its strength and the situational
context.”
Eighth, learner autonomy is supported. It is still a question to answer
whether every English testing supports learner autonomy. It is one major task of this
research to find out through the narrative stories of the students as one main
stakes-holders how the actual use of English test supports learner autonomy. In the previous
discussion, however, we come to a conclusion that English testing should.
Up to here, we see then that English testing has opportunities to bring
conditions for optimal language learning. If we are more concerned with learning,
therefore we should make good use of the opportunities and be aware of its
backwash effects since the use and the design of tests is said to bring certain
implication to learning, whether positive (beneficial) or negative (harmful).
teaching and learning” (Hughes & Ur, 2003:1), “the impact of tests on the teaching
programme […]” (McNamara, 1996:23), “the effect that testing practices have on
teaching practices” (Wharton, 2005:4). The existence of test backwash is
acknowledged by researchers and writers. McNamara (1996), speaking about test
backwash, concludes that the use of multiple choice standardized test for
achievement purpose has a negative effect on classrooms as teachers teach to the test
and there is also often a mismatch between the test and the curriculum. Standardized
tests, he goes on to say, are seen too often having a negative, restricting influence on
progressive teaching.
Caine (2005) investigated the backwash effects of the existing English tests
and the proposed new speaking test on teaching and learning in Japan. He noticed
the harmful backwash effects of the existing test under study since the
examination-driven “hidden syllabus” prevents the teachers from implementing, in practice,
communicative methodology. And, confirming earlier findings by other researchers,
he observed the backwash effects fall only at the level of content (what is actually
taught) instead of the level of methodology (how teachers teach).
Wharton (2005:4), referring to Prodomou 1995, furthermore states backwash
is determined not merely by test writers but also teachers and learners:
An important part of the description of a test, then, is the description of any backwash effect that it hopes to have or is believed to have. That is not to say that backwash is entirely determined by test writers. It also depends on the ways in which teachers and learners interpret test requirements, and work with them in the classroom.(my emphasis)
What I have to underline from this statement is that backwash also depends on the
ways in which teachers and learners interpret test requirements and work with them
toward the test and their experience in it plays a vital role in deciding whether a test
will bring either positive or negative impacts. In relation to this, Taylor (2000:2) in
Caine (2005:17) points out,
Some of the stakeholders listed above (e.g. examiners and materials writers) are likely to have more interest in the ‘front end’ of a test, i.e. the test assessment criteria or test format. Others may see their stake as being primarily concerned with the test score. Some stakeholders, such as learners and teachers, will naturally have an interest in all aspects of the test. (my emphasis)
In an educational language program, the fundamental use of testing, is said to
provide information for making decisions, that is, for evaluation (Bachman, 1995:54,
78). In Bachman’s term, tests with the fundamental use means they are used for
“purely descriptive purposes”, that is only when the results of tests are used as a
basis for making a decision that evaluation is involved (1995:22). In addition to the
fundamental use, Bachman (1995:21) states that, in line with Wiggins’ (1998) view,
tests are often used for pedagogical purposes, either as a means of motivating
students to study or as a means of review material taught, in which case no
evaluative decision is made on the basis of the test result. For pedagogical purpose, it
is clearly the positive which is expected to come from testing.
The use of test
Figure 2.2 Bachman’s categorization of the use of tests
According to Brown (2005:1), the first and most basic distinction in language
testing involves two very different functions. In his dichotomy, he uses a specific
