ANALYTIC SCORING PROCEDURE IN SPEAKING ASSESSMENT AT HOSPITALITY PROGRAM OF STATE VOCATIONAL HIGH SCHOOL 1 BUDURAN SIDOARJO.

(1)

ANALYTIC SCORING PROCEDURE IN

SPEAKING ASSESSMENT AT HOSPITALITY

PROGRAM OF STATE VOCATIONAL HIGH

SCHOOL 1 BUDURAN SIDOARJO

THESIS

Submitted in Partial Fulfillment of the Requirement for the degree of Sarjana

Pendidikan (S.Pd) in Teaching English

By:

RIZKY AMALIA INDAYANI

NIM D35212054

ENGLISH TEACHER EDUCATION DEPARTMENT

FACULTY OF TARBIYAH AND TEACHER TRAINING

SUNAN AMPEL STATE ISLAMIC UNIVERSITY

SURABAYA

(2)

(3)

APPROVAL SHEET

(4)

(5)

LEMBAR PERNYATAAN PERSETUJUAN PUBLIKASI KARYA ILMIAH UNTUK KEPENTINGAN AKADEMIS

Sebagai sivitas akademika UIN Sunan Ampel Surabaya, yang bertanda tangan di bawah ini, saya:

Nama : RIZKY AMALIA INDAYANI

NIM : D35212054

Fakultas/Jurusan : FTK/PENDIDIKAN BAHASA INGGRIS E-mail address : rizkyamaliaindayani@yahoo.com

Demi pengembangan ilmu pengetahuan, menyetujui untuk memberikan kepada Perpustakaan UIN Sunan Ampel Surabaya, Hak Bebas Royalti Non-Eksklusif atas karya ilmiah :

Skripsi Tesis Disertasi Lain-lain (………) yang berjudul :

ANALYTIC SCORING PROCEDURE IN SPEAKING ASSESSMENT OF

HOSPITALITY PROGRAM AT STATE VOCATIONAL HIGH SCHOOL 1

BUDURAN SIDOARJO.

beserta perangkat yang diperlukan (bila ada). Dengan Hak Bebas Royalti Non-Ekslusif ini Perpustakaan UIN Sunan Ampel Surabaya berhak menyimpan, mengalih-media/format-kan, mengelolanya dalam bentuk pangkalan data (database), mendistribusikannya, dan menampilkan/mempublikasikannya di Internet atau media lain secara fulltext _{untuk kepentingan} akademis tanpa perlu meminta ijin dari saya selama tetap mencantumkan nama saya sebagai penulis/pencipta dan atau penerbit yang bersangkutan.

Saya bersedia untuk menanggung secara pribadi, tanpa melibatkan pihak Perpustakaan UIN Sunan Ampel Surabaya, segala bentuk tuntutan hukum yang timbul atas pelanggaran Hak Cipta dalam karya ilmiah saya ini.

Demikian pernyataan ini yang saya buat dengan sebenarnya.

Surabaya, 20 Februari 2017

Penulis

(Rizky Amalia Indayani) KEMENTERIAN AGAMA

UNIVERSITAS ISLAM NEGERI SUNAN AMPEL SURABAYA

PERPUSTAKAAN

(6)

ABSTRACT

Indayani, Rizky Amalia (2017). Analytic Scoring Procedure in Speaking Assessment At Hospitality Program of State Vocational High School 1 Buduran Sidoarjo.

A Thesis. English Education Department, Faculty of Tarbiyah and Teacher Training, Sunan Ampel State Islamic University, Surabaya. Advisor: Dra.

Arba’iyah Yusuf, M.A.

Key Words: Analytic Scoring Procedure, Analytic Rubric, Speaking

Students are categorized as a good or bad speakers depend on someone who assesses their ability while speaking. It is common that speaking assessment for classroom purposes done by the teachers has been done without any theoretical basis. So, the

teachers just use their feel judgment to score the students’ performance. By doing

that, the teachers determine the score based on their impression without making an analysis of the aspects of speaking ability. Thus, the research is aimed to figure out whether the teacher’s rubrics for speaking assessment of hospitality students at State Vocational High School 1 Buduran comply with the characteristic of Analytic Scoring Procedure or not. The subjects of this research are the teacher and the students of tenth grade students of Hospitality Program at State Vocational High School 1 Buduran. There are 2 rubrics used by the teacher. The researcher used descriptive qualitative method to present the data. Here, the researcher collected the data used three techniques, those are; observation checklist, interview guideline, and documents. The data was obtained from document and interview. After collecting the documents, the researcher analyzed it by using checklist rubric about the valid analytic scoring rubric for oral assessment adapted from two theories. First, the theory from Mary J. Allen in her book Developing and Applying Rubric,1 and the Theory from Susan M. Brookhartin his book by the title “How to Create and Use Rubrics for Formative Assessment and Grading”. In conclusion, these rubrics are

compliant enough with the characteristics of analytical rubric based on Allen’s theory

in both created process and the natures. In particular, rubric for presentation is compliant by having 70% compliant to the characteristic of analytic rubric based on the created process and 70% compliant based on the natures. Additionally, rubric for conversation based on the created process characteristic is compliant by having 70% compliance and 100% compliant based on the natures.

1

(7)

ABSTRAK

Indayani, Rizky Amalia (2017). Prosedur Penilaian Analitik di Penilaian Kemampuan Berbicara Pada Program Perhotelan Negeri SMK 1 Buduran Sidoarjo. Sebuah tesis. Pendidikan Bahasa Inggris, Fakultas Tarbiyah dan Keguruan, Sunan Ampel Universitas Islam Negeri, Surabaya. Pembimbing: Dra. Arba'iyah Yusuf, M.A.

Kata kunci: Prosedur Penilaian Analitik, Analitik Rubrik, Berbicara

Siswa dikategorikan sebagai pembicara yang baik atau buruk tergantung pada seseorang yang menilai kemampuan mereka saat berbicara. Pada umumnya, Penilaian kemampuan berbicara untuk tujuan kelas yang dilakukan oleh guru telah dilakukan tanpa dasar teoritis. Jadi, para guru hanya menggunakan pertimbangan terhadap penampilan siswa pada saat berbicara. Dengan melakukan itu, para guru menentukan skor berdasarkan kesan mereka tanpa membuat analisis aspek kemampuan berbicara. Dengan demikian, penelitian ini bertujuan untuk mengetahui apakah rubrik guru untuk berbicara penilaian siswa perhotelan di SMK Negeri Tinggi 1 Buduran sesuai dengan karakteristik dari Prosedur Penilaian Analitik atau tidak. Subyek penelitian ini adalah guru dan siswa dari siswa kelas X Program Perhotelan di SMK Negeri Tinggi 1 Buduran. Ada 2 rubrik digunakan oleh guru. peneliti menggunakan metode deskriptif kualitatif untuk menyajikan data. Di sini, peneliti mengumpulkan data menggunakan tiga teknik, yaitu; ceklis observasi, wawancara pedoman, dan dokumen. Data tersebut diperoleh dari dokumen dan wawancara. Setelah mengumpulkan dokumen, peneliti menganalisis dengan menggunakan rubrik tentang rubrik analitik scoring berlaku untuk penilaian lisan diadaptasi dari dua teori.

Pertama, teori dari Mary J. Allen dalam bukunya ‘Developing and Applying Rubric’2 ,

dan Teori dari Susan M. Brookhartin bukunya dengan judul ’How to Create and Use Rubrics for Formative Assessment and Grading’. Kesimpulannya, rubrik ini cukup sesuai dengan karakteristik rubrik analitis berdasarkan teori Allen di kedua proses pembuatan dan sifat. Secara khusus, rubrik untuk presentasi sesuai dengan memiliki tingkat kesesuaian 70% dengan karakteristik dari rubrik analitik berdasarkan proses pembuatannya dan 70% sesuai berdasarkan cirinya. Selain itu, rubrik untuk percakapan berdasarkan karakteristik proses pembuatannya sesuai dengan memilikikesesuaian 70% terhadap proses pembuatan dan 100% sesuai berdasarkan sifat.

2

(8)

LIST OF TABLES

4.1. Characteristic of Analytic Rubric based on Created Process ... 41 4.2. Characteristic of Analytic Rubric ”based on the standard” ... 42 4.3. Characteristic of Analytic Rubric “Design to assess created authentic task” ... 43 4.4. Characteristic of Analytic Rubric “The number of criteria is the essential

elements of the task” ... 45 4.5 Characteristic of Analytic Rubric Everything on every task is not always assessed ... 46 4.6 Characteristic of Analytic Rubric Numbers of criteria are based on the kinds of assignment ... 47 4.7 Characteristic of Analytic Rubric The same number in levels of performance for each criterion within a rubric ... 48 4.8 Characteristic of Analytic Rubric Additional descriptors to each level of

performance ... 50 4.9 Characteristic of Analytic Rubric Assigning point values to performance on each criterion ... 52 4.10 Characteristic of Analytic Rubric Equal intervals between the point values in a rubric ... 53 4.11 Characteristic of Analytic Rubric Checking the Rubric ... 54

4.12 Characteristic of Analytic Rubric”based on the natures” ... 56 4.13. Characteristic of Analytic Rubric “Helps to examine efficiently complex

products or behaviors” ... 57 4.14. Characteristic of Analytic Rubric “Helps to precisely define targets or

expectations”

(11)

(12)

LIST OF APENDICES

Appendix 1 Teacher’s rubrics

Appendix 2 Form of observation checklist

Appendix 3 Form of interview guideline

Appendix 4 Result ofobservation Checklist

(13)

CHAPTER I INTRODUCTION

This study is about analytic scoring procedure in speaking assessment at

hospitality program of state vocation high school 1 buduran sidoarjo. This

chapter discusses the area of the study that be covered in some headings;

background of study, research questions of the study, objectives of the study,

significance, scope and limitation, then definition of the key terms.

A.Background of The Study

Speaking seems intuitively the most important of all the four language

skills (listening, writing, reading, and speaking) since people who know a

language are referred to as a ‘speaker’ of that language, as if speaking

included all other skill knowing that language1

. Through speaking, somebody

shows their idea or their feeling. Moreover in English class, the teacher will

know students idea through speaking. As another skill, the teacher conduct

speaking test to measure that students understand and have a progress during

study in the class.

A speaking test is a procedure to measure speaking ability that

requires the learner to speak or to produce utterances and he/she is assessed

1

(14)

2

on the basis of his/her utterances2

. The learners will know their progress,

strengths and weakness during study the lesson in the class. Furthermore,

through speaking test the teacher knows their students about their progress.

In an elementary level, a speaking test may be focused on assessing

one or two aspects of speaking skills such as the pronunciation, intonation,

and stress. In advanced level, the assessment is focused on the language

function. Unfortunately, when speaking assessment is concerned, not all

teacher when evaluates students using a detail description about how to

develop a good scoring guide or a scoring sheet. Experts in language testing

have discussed types and techniques of language testing without giving

information on how to score the test3

.

Presently, learners are demanded to speak English during their English

Lesson or in full case for English for Specific Purposes Students such students

of Vocational High School in a hospitality program. They are forced to speak

English fluently because the purpose of their study is getting in touch with

many foreigners in their working world.

Students are categorized as a good or bad speakers depend on

someone who assesses their ability while speaking. It is common that

speaking assessment for classroom purposes done by the teachers has been

done without any theoretical basis. So, the teachers just use their feel

2

Underhiil, N. Testing spoken language. Cambririge: (Cambridge University Press:1987) p.98

3

(15)

3

judgment to score the students’ performance. By doing that, the teachers

determine the score based on their impression without making an analysis of

the aspects of speaking ability. Thus, the researcher directly shows the use of

analytic scoring procedure in assessing speaking ability of students used

English for Specific Purposes. Analytic schemes have also been found to be

particularly useful for scorers who are relatively inexperienced4

.

Most of researchers find out the effectiveness of those scoring

procedure by comparing both of analytic and impressionistic. In this research,

the researcher wants to find out resemblance of the teacher’s rubric for

speaking assessment with the characteristic of Analytic Scoring rubric and

make it related with English for Specific Purposes (ESP). It should be

analytic because this scoring procedure is more effective than the holistic

one5

. There are several studies which is similar to this study. The first study

is by Luu Trong Tuan, 2012, University for Natural Resources and

Environment, Ho Chi Minh City, Vietnam, Teaching and Assessing Speaking Performance trough Analytic Scoring Approach. The aim of this research was to examine if the analytic scoring approach in teaching and assessment

would be more effective for improving students’ speaking performance6

. The

4

Underhiil, N. Testing spoken language. Cambririge: (Cambridge University Press:1987) p.138

5

Mukminatien, Nur. 2000. The Advantages of Using an Analytic Scoring Procedure in Speaking Assessment, TEFLIN Journal, Vol. XI, No. 1 Agustus 2000. Universitas Negeri Malang.

6

(16)

4

research also explored the students’ attitude towards the implementation of

analytic scoring approach in speaking teaching and assessment.

The second study is Ying-Yung Chuang, Cheng Shiu University,

China Taiwan. Foreign Language Speaking Assessment: Chinese Taiwanese

College English Teachers’ Scoring Performance in the Holistic and Analytic

Rating Methods. The purpose of this study was to investigate college English

teachers’ scoring performance of the Holistic and Analytic rating methods,

their views and concerns with the components of oral skills, and whether

teachers’ background variables influenced their scoring performance.

This research only focuses on the teachers’ rubric in assessing

speaking of hospitality students at Vocational High School 1 Buduran

Sidoarjo. This school was chosen as the research place because the quality of

the school. Some of big institutions whether domestic or foreign have

cooperation with this Vocational High School. Relating the vision of SMKN 1

Buduran that is become an education and training institution which is able to

produce professional, independent and competing graduates in national and

International social work7

. In particular, Hospitality Program there is student

exchange program which some of students has a chance to go overseas to job

training. The program collaborates with Malaysia. thus, the students of

hospitality program must speak English very well.

7

(17)

5

B. Statement of the Research Problem

Based on the background of the study, the researcher analyzed the compliance

of the teacher’s rubric for speaking assessment with the characteristic of

Analytic Scoring rubric. In line with the statements described above, the

researcher formulated the problems as follow:

Do the teacher’s rubrics for speaking assessment of hospitality students at

State Vocational High School 1 Buduran comply with the characteristic of

Analytic scoring procedure?

C. Objectives of the Study

Related to the problems above, the researcher has formulated the major

objective of this study to describe the resemblance of teacher’s rubric for

speaking assessment for English for Specific Program in terms of;

To figure out whether the teacher’s rubrics for speaking assessment of

hospitality students at State Vocational High School 1 Buduran comply with

(18)

6

D. Significance of Study

By completing this study, the researcher expected which brought much

benefit to many people. The analytic procedure of speaking assessment

provided a double function: as a valid and reliable instrument and as a

diagnostic procedure or remedial teaching. Therefore, it serves as feedback

both for the students and teacher.

1. For the Teacher

By doing an analysis in teachers’ scoring rubric used in assessing

speaking, we can give more suggestion whether the rubric has already

complied with the characteristic of analytic procedure or not. So, the

teacher’s assessment for hospitality students will be more effective and

the teacher can further make a decision on which speaking skill needs

more practice in the remedial teaching.

2. For Business Travel Students

By doing this research and knowing that the procedure of scoring is based

on the analysis of features, this scoring procedure can be used as feedback

for the learner to improve his/her performance. So, after knowing the

results of each student, they will know which language component that

(19)

7

E. Scope and Limit of the Study

The scope of this study is English teaching. Specifically, the main data

is taken from the rubrics for assessing speaking in Individual presentation

with expression of intention as the material and short conversation with

expression congratulating others as the material. The elements and the

characteristics of the rubrics are analyzed in both created process and the

natures of an analytic rubric.

Furthermore, the study is limited to the teacher who has created and

used the rubric for assessing students’ speaking skill. He is an English teacher

for seventh grade of Hospitality students at State Vocational High School 1

Buduran.

F. Key Terms 1. Hospitality

Based on Cambridge Advance Learner Dictionary, hospitality is defined as

a term when people are friendly and welcoming to guests and visitors. So,

hospitality students are only a group of people at State Vocational High

School 1 Buduran Sidoarjo who learn particular topics for hospitality

where the English is needed to be used in certain situation especially for

(20)

8

2. Speaking

Speaking is an interactive process of constructing meaning which is

comprised of producing and receiving information (Brown, 1994; Burns &

Joyce, 1997). Besides, speaking also classified to monologue and

dialogue8

. In education world, speaking practice is not just producing some

words in specific purposes. But, it must be assessed so both of the teacher

and learner will know someone’s ability in speaking.

3. Analytic Scoring Procedure

The procedure of scoring based on analysis of features is called analytic

(Lyoyd-Jones, 1997; White, 1985; Spandel and Stiggins, 1990). This

procedure provides two advantages especially for classroom purposes: as

an instrument to measure the students’ speaking proficiency and as a

diagnostic procedure for remedial teaching.

8

(21)

CHAPTER II

REVIEW OF RELATED LITERATURE

In this chapter, the researcher attempts to provide two sub-sections. The first is

the review of related literature which describes the basic theories used in this

research. The second is review of previous study; explains the differences of this

study from others which are conducted by other researchers.

A. Scoring Procedure of Speaking Test

The scoring procedure of a functional speaking test on the contrary is more

complicated as it requires the rater to pay attention to the overall components of

speaking skill at one time.1

The functional speaking test can be done in an

interactive communication or transactive one (Brown and Yule in Mukminatien,

2000: 39). Interactive communication is referring to the function of language to

maintain social interaction like interview and role play. The transitive interaction

is referring to a type of communication that is focused on conveying the

massage, and not on the interaction, like storytelling, giving speech, reading an

announcement, presenting a report, and many others. 2

According to Underhill in Nur Mukminatien (2000: 39) there are two

approaches in the scoring procedures of a speaking test, the analytic approach

1

2

(22)

10

and the impressionistic approach3. First is scoring the learners’ speaking ability

by separating the components of speaking skill into sub skills, and the rater

scores each component, and then sums the sub scores into final score. The latter

is scoring/judging the learners’ speaking ability on the basis of the rater’s general

impression on the learners’ performance without necessarily separating the

speaking components.

B. Analytic Scoring Procedures

Jones et.al, defines that the procedure of scoring based on analysis of features is called analytic. Trong Tuan also states that analytic scoring is

assessment which the objectives of the final product is broken down into criteria

parts, and each part is scored independently.4

The procedures of this method

involve the separation of the various features of a discourse into categories for

scoring purpose, meanwhile the final score from this analytic is the rating which

is calculated and summarized from the total score for all of the parts that are

being evaluated. Therefore, to avoid bias towards the whole product, it is

necessary for the teacher to treat each criterion or part as separate. Related to this

research conducting in analyzing an analytic scoring procedure that the teacher

used in assessing speaking in Hospitality at State Vocational High School 1

3

Ibid..

4

(23)

11

Buduran, there are the advantages and the disadvantages of scoring procedures.

It is described as follow:

1. Advantages and Disadvantages of Analytic Scoring procedures

a. The Advantages

Analytic scoring is preferred over holistic schemes by many speaking

specialists for a number of reasons.5 It means that analytic scoring

provides more useful diagnostic information about students’ speaking

abilities in term of where their weaknesses are and where their strengths

are. Downing and Haladyna (2006) argues that analytic scoring has been

considered as more interpretable scoring approach because it can access

the examinee’s specific strengths and weaknesses and can identify the

particular components of speaking discourse that an examinee needs

developing.

Park (2004) identified that the explicitness of the analytic scoring

guides offers teachers a potentially valuable tool for providing speakers

with consistent and direct feedback6. He also adds that analytic scoring are

particularly useful for second language learners. Moreover, another author

examines that some learners who acquire English as their second language

may probably have excellent speaking skill in terms of its content and

5

Tuan, Luu Trong. 2012. Teaching and Assessing Speaking Performance through Analytic Scoring Approach . University for Natural Resources and Environment, Ho Chi Minh City, Vietnam. Theory and Practice in Language Studies, Vol. 2, No. 4, pp. 673-679, April 2012

6

(24)

12

organization, otherwise they may have much lower grammatical control;

others may have an excellent control of sentence structure, but may not

know how to organize their speech in a logical way. Thus, the analytic

scoring scales take a role when the same rubric categories repeatedly are

used to show students’ knowledge if they have made progress in some or

all aspects that being assessed7.

b. The Disadvantages

Another major of analytic scoring is the disadvantages of it. In this

case, rating speaking performance takes a lot of time since examiners are

required to make more than one decision for every speaking performance8.

When scoring analytically, an examiner has to check, consider, and score

each criterion of the speaking ability and then gives a total score

depending on the coefficient put forward. Hughes (2003) warned that in

scoring analytically, the criterion scored first may effect on subsequent

criteria which are scored later, making the overall effect of a speech

diverted to an individual criterion.

An additional problem with some analytic scoring schemes is that

even experienced essay judges sometimes find it difficult to assign

7

Moskal, B.M. (2000). "Scoring rubrics: What, when and how?" Practical Assessment, Research & Evaluation, 7 (3). Accessed on http://pareonline.net/getvn.asp?v=7&n=3

8

(25)

13

numerical scores based on certain descriptors9. In this aspect, there are

possibilities for scorers to disagree with one another. It is more difficult to

achieve intra- and inter-rater reliability on all of the dimensions in an

analytic scoring scheme than on a single score yielded by a holistic scale.

This disadvantage is inevitable, especially with trained or

un-experienced scorers. Grammar-related categories are somewhat wrong –

right categories whereas other categories are judgments. Focusing on

wrong – right categories will always be easier than judgments.

2. Analytic Oral Language Scoring Rubric

A rubric is a coherent set of criteria for students’ work that includes

descriptions of levels of performance quality on some particular criteria. The

main purpose of rubrics is to assess performances. Some performance

performances that might assess with rubrics are Oral communications

including Making a speech to the class, reading aloud, conversing in a foreign

language.

The genius of rubrics is that they are descriptive and not evaluative. Of

course, rubrics can be used to evaluate, but the operating principle is you

match the performance to the description rather than “judge” it. Thus good

rubrics have to State standards, curriculum goals, and instructional goals and

objectives are the sources for what types of performances your students

9

(26)

14

should be able to do. Rubrics are usually categorized by two different aspects

of their composition. One is whether the rubric treats the criteria one at a time

or together. The other is whether the rubric is general or is task-specific and

only applicable to one assessment.

There are two approaches in the scoring procedures of a speaking test, the

analytic approach and the impressionistic approach10. Analytic rubrics

describe work on each criterion separately. Holistic rubrics describe the work

by applying all the criteria at the same time and enabling an overall judgment

about the quality of the work.

Indonesian teachers mostly tend to use numeric scores in interpreting

students’ score. Even though particular measurements and the details of

scoring method are included, the levels of the students’ achievement during

the learning process and the explanation of their result are significantly

needed. In connecting to this, some specific scales in different criteria and

levels are considered. Therefore, most classroom purposes, analytic rubrics

are best11. Focusing on the criteria because students can see what aspects of

their work need what kind of attention. It will also be used to make decisions

about the future. For example, it is a diagnostic procedure for remedial

teaching and as a procedure to teach something next year.

10

11

(27)

15

3. Characteristics of Analytical Scoring Procedure

a. Created Process

Rubrics provide the criteria for assessing students' work. They can be

used to classify virtually any product or behavior, such as essays, research

reports, portfolios, and works of art, recitals, oral presentations,

performances, and group activities. Judgments can be self-assessments by

students; or judgments can be made by others, such as faculty, other

students, fieldwork supervisors, and external reviewers. Rubrics can be

used to provide formative feedback to students, to grade students, and to

assess courses or programs.

As mentioned above, rubric is divided into two kinds: holistic rubric

(one global, holistic score for a product or behavior) and analytic rubric

(separate, holistic scoring of specified characteristics of a product or

behavior). Specifically, as this research is focused on analytic rubric, there

are some particular characteristics explained by Allen in these following

details12:

1) Based on the standards

Standards are typically one-sentence statements of what students

should know and be able to do at a certain point. There are many

sources that teachers can find examples of goals and standards that

12

(28)

16

might be appropriate for students. There are national and state

standards as well as numerous websites with many good choices.

Regarding to this, there are three steps in deciding the standards:

reflect, review and write. Basically, it is unnecessary to start from

scratch, which can confine teacher’s thinking. However, it is necessary

to look at the work of others; therefore, considering what do teacher

really wants the students to know and be able to do when they leave

the grade or school. This is called as reflection. As a result of this

reflection, teacher might reach consensus on a few things you most

value and agree should be included in the standards. Teacher might

actually write a few standards or, teacher might produce a long list of

possible candidates for standards. Furthermore, reviewing is the next

step. Then, the last one is writing. After writing the standards, it is

considerable to make guiding questions to the standards makers or

teachers’ partners.

2) Designed to assess created authentic task

Authentic assessment does not compete with traditional assessments

like tests. Rather, they complement each other. Each typically serves

different assessment needs, so a combination of the two is often

appropriate. As teacher wants to find a way students can demonstrate

(29)

17

consider the authentic task is also designed based on the standards and

the real world. Therefore, as the task for students is good, the analytic

rubric should be created and scaled based on the task given as well.

3) Consists of the characteristics of good criterion

Criteria here mean indicators of good performance on a task. In order

to concept an analytic rubric, questions such as "What does good

performance on this task look like?" or "How will I know they have

done a good job on this task?" frequently need to be appeared. In

answering those questions teacher will be identifying the criteria for

good performance on that task. Teacher will use those criteria to

evaluate how well students completed the task and, thus, how well

they have met the standard or standards. Regarding to this, the first

thing to be considered is the number of criteria is the essential

elements of the task; the next one is everything on every task is not

assessed; and the last is numbers of criteria are based on the kinds of

assignment.

4) The same number of levels of performance for each criterion within a rubric

As mentioned before, keep the number of criteria manageable.

Teacher does not have to look for everything on every assessment.

(30)

18

rubric, three levels for two other criteria, and four levels for another

criterion, all within the same rubric. Rubrics are very flexible. There is

no need to force an unnatural judgment of performance just to

maintain standardization within the rubric. If one criterion is a simple

either/or judgment and another criterion requires finer distinctions,

then the rubric can reflect that variation.

5) Additional descriptors to each level of performance

Descriptors are recommended but not required in a rubric. As

described in rubrics, descriptors are the characteristics of behavior

associated with specific levels of performance for specific criteria. For

example, in the following portion of an elementary science rubric, the

criteria are observations are thorough, predictions are reasonable, and

conclusions are based on observations. Labels (limited, acceptable,

proficient) for the different levels of performance are also included.

Students will be more certain what are expected to reach each level of

performance on the rubric if descriptors are provided. Furthermore, the

more detail a teacher provides about what good performance looks like

on a task the better a student can approach the task. Teachers benefit

as well when descriptors are included. A teacher is likely to be more

objective and consistent when applying a descriptor such as "most

(31)

19

such as "acceptable." Similarly, if more than one teacher is using the

same rubric, the specificity of the descriptors increases the chances

that multiple teachers will apply the rubric in a similar manner. When

a rubric is applied more consistently and objectively it will lead to

greater reliability and validity in the results.

6) Assigning point values to performance on each criterion

As mentioned above, rubrics are very flexible tools. Just as the number

of levels of performance can vary from criterion to criterion in an

analytic rubric, points or value can be assigned to the rubric in a

myriad of ways. For example, a teacher who creates a rubric might

decide that certain criteria are more important to the overall

performance on the task than other criteria. So, one or more criteria

can be weighted more heavily when scoring the performance. For

example, in a rubric for solo auditions, a teacher might consider five

criteria: (how well students demonstrate) vocal tone, vocal technique,

rhythm, diction and musicality. For this teacher, musicality might be

the most important quality that she has stressed and is looking for in

the audition. She might consider vocal technique to be less important

than musicality but more important than the other criteria. So, she

might give musicality and vocal technique more weight in her rubric.

(32)

20

7) Equal intervals between the point values in a rubric

Analytic rubrics are flexible tools. Shape them to fit teacher’s needs,

not the other way around. In other words, points should be distributed

across the levels of a rubric to best capture the value you assign to

each level of performance. So, feeling bound by any format constraints

when constructing a rubric is not necessary. The rubric should best

capture what you value in performance on the authentic task. The

more accurately your rubric captures what you want your students to

know and be able to do the more valid the scores will be.

8) Checking the rubric

As a final check on a rubric, teacher can do any or all of the following

before applying it: let a colleague review it, let students review it -- is

it clear to them, check if it aligns or matches up with standards, check

if it is manageable, consider imaginary student performance on the

rubric. The last suggestion, imagine that a student had met specific

levels of performance on each criterion (for an analytic rubric). Then

make sure if that performance translates into the score that is

seemingly appropriate. Of course, teacher will never know if he/she

really has a good rubric until it is applied. So, working to perfect the

(33)

21

and then try it. Find out what needs to be modified and make the

appropriate changes.

b. Natures

Scoring rubrics are explicit schemes for classifying products or

behaviors into categories that vary along a continuum. Some particular

natures are identified based on the advantages and the function of analytic

rubric; it can be identified as these following points:

1) Helps to examine efficiently complex products or behaviors

Specific criteria in analytic rubric help teacher a lot in examining

complex products created by students or behaviors that are going to be

measured. The levels also give specific and clearer explanation for

characteristics that have good or bad scores.

2) Helps to precisely define targets or expectations

The specific criteria mentioned in analytic rubric can facilitate teacher

in indicating the definition of targeted or expected achievement for

students’ products, performances, and behaviors that are going to be

assessed.

3) Criterion-referenced

Each criterion of the criteria mentioned in the rubric has to have

(34)

22

or expected. In addition, the description should be referenced based on

the criterion.

4) Ratings can be done by students to assess their own work or others

In analytic rubric, as there are specific definition of the target and

expectation, the scales are also clear enough for people who read it, so

the assessment is possibly applicable for the students who has their

works or even other students to assess their friends.

5) Gives diagnostic information to teacher

As this is talking about analytic rubric, the result of the assessment

process using the rubric should be analytic as well. It means the

information should be clear and detail based on the criteria and the

explanation about the target that students have to achieve.

6) Gives formative feedback to students

As explained in the previous point, analytic rubric is accessible for

everyone, both students and teacher to assess something. This means,

the feedback that is reflected as a result of measurement using the

rubric is clearer for everyone as well; moreover for students whose

(35)

23

7) Easier to link to instruction

When teacher gives task to students, there must be instructions as well.

Therefore, every criterion in the rubric has to be connected to the

instruction from the teacher as well.

8) Accessible for formative assessment

According to Susan, analytic rubric is designed to assess formative

assessment. The goal of formative assessment is to monitor student

learning to provide on-going feedback that can be used by teacher to

improve their teaching and by students to improve their learning.

Specifically, it helps students to identify their strengths and

weaknesses and target areas that need works.

9) Adaptable for summative assessment

As explained above, analytic rubric is created for formative

assessment; as consequences, summative assessment can be measured

by this rubric when it is adapted. The goal of summative assessment is

to evaluate students learning at the end of an instructional unit by

comparing it against some standards. It often has a high point value.

10)Can be combined when overall score for grading is needed

It is easier for teacher to wrap up the result of students’ achievement

during the learning process or the result of the task by combining the

(36)

24

countable and it can be rated based on the grade, the analytic rubric is

considerable.

C. English for Specific Purposes (ESP)

Over the last three decades, English for specific purposes (ESP) has become

a successful field13

. English was born in the 190s as a technical texts-analysis

area14

. English for Specific Purposes (ESP) has boomed in regular education

such in Vocational High School. Some of vocational High School inserts ESP in

learning English. In addition, the approach to language description for English

Academic Purposes developed by Swales (1990) has been embraced by ESP15.

Based on Swales, 1992: 300 English for specific purposes (ESP) is a

subdivision of a wider field, language for specific purposes which is defined as

the area of inquiry and practice in the development of language programs for

people who need a language to meet a predictable range of communicative

needs16

.in addition, English for Specific Purposes (ESP) is a learner-centered

approach to teaching English as an additional language which focuses on

developing English communication skills in a specific discipline, such as

accounting, management, education, law, engineering, IT technology, health and

13

Gonzales, Carolina Ramirez, 2015 English for Specific Purposes: Brief History and Definitions. Costa Rica University. P 380

14

Ibid 381

15

Basturkmen, Helen. 2010. Developing Courses in English for Specific Purposes. University of Auckland: New Zealand) P. 10

16

(37)

25

medicine. It involves teaching and learning the specific skill and language

needed by particular learners for a particular purpose.

In other hand, English for Specific Purposes (ESP) as a learner-centered

approach, its main purpose has been that of fulfilling the specific needs of target

learners to satisfy either their professional or vocational demands17

. Typically,

ESP has functioned to help language learners cope with the features of language

or to develop the competencies needed to function in a discipline, profession, or

workplace18

.

D. Previous Studies

It has been prove that Analytic scoring procedure more advantageous

compared to the impressionistic one. But, some researchers have studied some

issues dealing with Analytic Scoring procedure in different focus and different

sub. The findings of studies almost show same result.

The first study is by Luu Trong Tuan, 2012, University for Natural

Resources and Environment, Ho Chi Minh City, Vietnam, Teaching and Assessing Speaking Performance trough Analytic Scoring Approach. The aim of this research was to examine if the analytic scoring approach in teaching and

assessment would be more effective for improving students’ speaking

17

Gonzales, Carolina Ramirez, 2015 English for Specific Purposes: Brief History and Definitions. Costa Rica University. P 382

1818

(38)

26

performance19

. The research also explored the students’ attitude towards the

implementation of analytic scoring approach in speaking teaching and

assessment.

Then the second study is by Ying-Yung Chuang, Cheng Shiu University,

China Taiwan. Foreign Language Speaking Assessment: Chinese Taiwanese

College English Teachers’ Scoring Performance in the Holistic and Analytic

Rating Methods. The purpose of this study was to investigate college English

teachers’ scoring performance of the Holistic and Analytic rating methods, their

views and concerns with the components of oral skills, and whether teachers’

background variables influenced their scoring performance. 20

The third study is by Daruwan Srikaew, Kamonwan Tangdhanakanond,

Sirichai Kanjanawasee, Chulalongkorn University, Thailand. English Speaking Skills Assessment for Grade 6 Thai Students: An Application of Multivariate Genenralizability Theory. This research studies an analytic rating scale for an English speaking skills assessment designed for Grade 6 Thai students learning

English as a foreign language. The results showed that fluency was the greatest

variance component of the composite score of the analytic rating scale, followed

by ideational function, cohesion, vocabulary and syntax, and pronunciation

19

Luu Trong Tuan. Teaching and Assessing Speaking Performance through Analytic Scoring Approach. University for Natural Resources and Environment, Ho Chi Minh City, Vietnam. Theory and Practice in Language Studies, (Vol. 2, No. 4, pp. 673-679, April 2012)

20

(39)

27

respectively. Reliability of the composite score for the speaking analytic rating

scale was high. It was found that the reliability coefficients for each component

would be reliable when six or more tasks are used and the number of raters is

from 6–10 and above. The dependability increased more when the number of

tasks increased and when the number of raters increased. It was also found that a

reliable high Phi Coefficient (over .80) could be obtained using only 6 tasks and

3 raters21

.

The forth study is by Chunguang Tian, Binzhou University. On the Construct Validity of an Analytic Rating Scale for Speaking Assessment. This paper aims to study the construct validity of an analytic rating scale for speaking

assessment. The Multi-facet Rasch Modeling method and the correlations

analysis are combined to investigate the construct validity. the analytic rating

scale can reflect the students’ speaking ability22

.

The fifth study is by Cynthia S. Wiseman, City University of New York. A Comparison of the Performance of Analytic vs. Holistic Scoring Rubrics to Assess L2 Writing. This study compared the performance of a holistic and an analytic scoring rubric to assess ESL writing for placement and diagnostic

purposes in a community college basic skills program. The study also

21

Daruwan Srikaew, Kamonwan Tangdhanakanond, Sirichai Kanjanawasee, Chulalongkorn University, Thailand. English Speaking Skills Assessment for Grade 6 Thai Students: An Application of Multivariate Genenralizability Theory. (2015, Vol 16, 47–66 p).

22

(40)

28

investigated whether each category in the two six-point rubrics were useful. Both

scales appeared to be measuring a single latent trait of writing ability23

.

The last study is by Nur Mukminatien, State University of Malang. The Advantages of Using Analytic Scoring Procedure in Speaking Assessment. This article intends to describe the advantages of Analytic Scoring Procedure and to

make a proof that Analytic scoring procedure is more advantageous compared to

the impressionistic (holistic) one. Besides directed to show that using an analytic

procedure in speaking assessment, in which the scoring procedure is based on

the analysis of features of speaking ability, is more advantageous than using an

impressionistic one (sometimes called a holistic approach) for two major

reasons.

Most of researchers find out the effectiveness of those scoring procedure by

comparing both of analytic and impressionistic. In this research, the researcher

wants to find out resemblance of the teacher’s rubric for speaking assessment

with the characteristic of Analytic Scoring rubric and make it related with

English for Specific Purposes (ESP).

23

(41)

CHAPTER III RESEARCH METHOD

This chapter deals with the procedures for conducting the study. It covers

Research design, subject of research, data and source of data, research process,

research instrument, data collection technique, and data analysis technique.

A. Research Design

This study designed to obtain information and description concerning

teachers’ rubric for speaking assessment then found out whether the teacher’s

rubric resemble to the characteristic of Analytic Scoring procedure or did not.

Therefore, the design of this study was descriptive-qualitative since the

purpose of this study is to describe the resemblance of a rubric. This

descriptive study used to gain certain information about a certain

phenomenon that happens when this study is conducted1

.

Based on Wallace, J, qualitative research is a method that uses to

analyze the data which cannot be counted. This research was coded the data

based on explanation of informants. Moreover, qualitative research was

concerned with developing explanations of social phenomena2

.

1

Bogdan and Biklen. Qualitative Research for Education. P 11 2

(42)

30

B. Subject of Research

The research subject of this research is the English teacher of

hospitality program in SMKN 1 Buduran, Sidoarjo. He is Mr. Mochammad

Anwar, which is only English teacher of hospitality program.

Mr. Anwar finished is graduated from English Education Department

of STKIP PGRI Sidoarjo in 1999. He started to teach since 2000/2001.

During his career, he had teaching experiences in three different schools.

First, he taught at SMA Tribhakti Tanggulangin from the academic year

2000/2001 to 2009/2010. Then, he started to teach English in MA Mambaul

Hikam Tanggulangin from the academic year 2002/2003 to 2009/2010. At

the same year, he also started to teach English in SMKN 1 Buduran. In 2010,

he decided to resign from two previous schools before and focused on

teaching English in hospitality program at State Vocational High School 1

Buduran.

C. Setting of the Research

1. Place

The research was conducted at SMKN 1 Buduran which was

located in St. Jenggolo No. 1B Sidoarjo. The research was focused on

Hospitality program because this is a program where the students have

(43)

31

their future career3. Therefore, there would be more chance for the teacher

to use authentic materials in English language teaching.

2. Time

The research was done starting from 6th June 2016. Document

analysis was started from 6th of October until a week before the last

interview. The interview was done on 11th June 2016. To complete the

missing data, the second interview was done on 29th of October 2016. By

conducting the interview after school hour and on the weekend, it would

not disturb the teaching and learning process.

D. Data and Source of Data

1. Data

The data that is used in this study is elements of the rubric used by the

teacher in assessing students’ speaking skill for two lessons and materials.

Particularly, the first rubric is a scoring procedure for individual

presentation which is used for ‘expression of intention’ lessons. The

second rubric is used for assessing short conversation and the material is

about ‘congratulating others’. In addition, teacher’s responds towards

interview given by the researcher will support the data got from the

rubric’s observation checklist.

3

(44)

32

2. Source of Data

Related to the data that are needed in this research, as the document is

the teacher’s rubrics; the documents is obtained the teacher’s file. Also,

the respondent of the interview of this study is the teacher as well.

Therefore, the source of the data here is the English teacher for seventh

grade of Hospitality students at State Vocational High School 1 Buduran,

Sidoarjo.

E. Research Instruments

1. The main instrument

The main instrument in this research was the researcher. The

researcher collected the data from Document study (Rubric Checklist)

and teachers. The researcher analyzed the rubric and interview from

the teacher.

2. Instrument Tools

There were two instrument tools that were used in this research:

a. Interview guidelines are questions for the subject of this research

and this is often used to obtain the subject perception and attitudes

of the issues. The methods are highly structured, formal interview,

taped, and transcribed. In this researcher, the researcher used

interview style semi-structured. From this style the researcher

(45)

33

Analyzing the interview data used two key ways. The first is

thematic generation is describe the perspective of the subject and

make the conclusion. The second is citation is directly quote the

subject’s sentences. In this case the researcher should directly write

the name of the subject4

. The researcher used both of two ways to

answer the research questions.

b. Observation Checklist

The observation checklist is needed to analyze the rubrics. The

document needed to be analyzed in the teacher’s rubric for

speaking assessment of hospitality students at State Vocational

High School 1 Buduran. To analyze the rubric, the result of

interview with the teacher was used. The result of both Checklist

rubric and interview answered the research question.

c. Document

Document study was the record or history of the data that the

research analyzed. The document in this study is the teacher’s

rubric for assessing speaking in tenth grade of hospitality students

at State Vocational High School 1 Buduran. To analyze the rubric,

the result of interview with the teacher was used. The result of both

Checklist rubric and interview answered the research question.

4

(46)

34

d. Tape Recorder

This interview has been done orally. In interviewing the teacher,

the researcher use recorder in order to keep the data. As the result

of interview is used as a supporting data as the in answering the

research question, it is very important to make sure that the data

would not be lost.

F. Data Collection Technique

In this research, the data was collected from the rubric used by the

teacher to assess students’ ability in speaking. The researcher asked the rubric

and analyzed it by using checklist rubric; did an interview with the teacher’s

to enrich understanding of the data. There are three kinds of interview, they

are: formal, informal, and standardized interview5. This study used formal

interview because the researcher had an interview with some of the teachers

of Hospitality program at State Vocational High School 1 Buduran Sidoarjo.

The result of both Checklist rubric and interview answered the research

question.

G. Data Analysis Technique

The researcher analyzed the data by using descriptive qualitative

method. The data was obtained from document and interview. After collecting

5

(47)

35

the documents, the researcher analyzed it by using checklist rubric about the

valid analytic scoring rubric for oral assessment adapted from two theories.

First, the theory from Mary J. Allen in her book Developing and Applying Rubric,6

and the Theory from Susan M. Brookhartin his book by the title

“How to Create and Use Rubrics for Formative Assessment and Grading”7.

So, to analyze the information from the whole data, it would be conducted by

using descriptive manner as follow8:

1. Identified the data based on the statement of problem

2. Classified each information based on each item of the interview and

observation

3. Verified and interpret the data

4. Draw the conclusion

H. Checking Validity of Findings

In checking validity of the findings, the researcher used triangulation

technique. As explained by Angen, triangulation in qualitative research is

generally used as technique to ensure that data of the research is rich, robust,

6

O'Malley, J.Michael, Authentic Assessment for English Language Learners. (London: Longmans, 1996). P. 46

7

Brookhart, Susan M.(2013) How to create and use rubrics for formative assessment and grading (Alexandria, Virginia, USA)

8

(48)

36

comprehensive, and well-developed.9 Additionally, Creswell also revealed

that using a single method can never adequately shed light on a phenomenon;

therefore multiple methods or techniques are needed in order to get deeper

understanding.10

In Denzin, there are four types of triangulation; those are methods

triangulation, triangulation of sources, analyst triangulation, and

theory/perspective triangulation.11 When analyzing data, the researcher also

crosschecked the findings using theory/perspective triangulation; as several

Analytic scoring procedure theories were regarded. Furthermore, analyst

triangulation was applied as well; besides the researcher, expert lecturers,

including the advisors also have ensured the findings.

I. Research Stages

In this research stages, the research explained the process of conducting

the research, including: preliminary research, planning, collecting data,

analyzing, and concluding data, as these following explanations:

1. Preliminary research

For the preliminary research, the researcher made an appointment with

the teacher at State Vocational High School 1 Buduran Sidoarjo to have

9

MJ Angen, Qualitative Health Research, vol. 10, 2000, p. 379,

10

John W. Creswell, Qualitative Inquiry and Research Design Choosing Among Five Traditions

(Thousand Oaks, CA: SAGE Publications Inc., 1998), p. 57

11

(49)

37

a consultation among this research. The researcher asked some

questions about practicing speaking and the way to assess it to make

sure that the teacher assesses it by using a rubric.

2. Planning

After conducting the preliminary research, the researcher made the

instruments for collecting the data of the study. A checklist rubric was

provided for assessing the compliance of teacher’s rubric with the

theory of analytic scoring rubric. Those checklist rubric as the

instrument of this study had to be validated first to the lecturer. After

passing the process of instruments validity, the researcher conducted the

research by using the instruments of research.

3. Collecting Data

In this stage, the researcher collected the data from a rubric used by

APH’s teacher to assess their students’ speaking ability. Then, the

teacher’s rubric was analyzed by using a document checklist to collect

some data from it.

4. Analyzing

After collecting the data, the researcher analyzed the teaching stages of

the lesson plans with the guidance of the theory in the literature review.

The result of the analysis was written descriptively. The researcher

(50)

38

answering the following research question and the result of the analysis

were written quantitatively at the findings and discussions.

5. Concluding data

The research summarized all information which found by the researcher.

The researcher served the data quantitatively. Then, the researcher

discussed all information concerning the research to obtain the research

data. Finally, the research concluded the research findings to answer the

(51)

CHAPTER IV

FINDING AND DISCUSSION

This chapter presents the collected data from teacher’s analytic scoring rubrics

and teacher’s responds towards the interview, also the analysis of it. The results of the

checklist also the teacher’s responds are showed as research findings. Furthermore,

the analyzed data is explained in discussion part.

A. Research Findings

Following the research question that discuss about the teachers’ rubric

compliance with the characteristic of analytic scoring procedure, the researcher

serves the data that found at State Vocational High School 1 Buduran. This

section contains on the detail information about the result of research which is

used two kinds of instrument those are observation checklist and interview. The

result of observation checklist presented in tabular form which is followed by the

result of teacher’s explanation supporting the checklist made by the researcher

which is taken from the interview section. The teacher’s explanation is presented

descriptively.

In particular, the two analytic rubrics are used in this research. Both of

those rubrics are rubric for assessing speaking skill of Vocational students

(52)

40

made by team teaching of that school then distribute to each proper class. Based

on the lesson plan which the researcher gained from the teacher, the first rubric is

created for assessing students speaking skill on learning material expression of

intention in individual presentation. The second rubric is for dialogue

congratulating others material.

Although those rubrics are created by the same person, but each rubric

has differences in some aspects based on created process and natures. From the

checklist of analytic rubric criteria, both of the first rubric and the second rubric

show several dissimilar results in some points of analytic rubric’s characteristics

stated by Allen as explained in theoretical framework in chapter 2.

1. The Compliance Assessing Rubric with Analytic Scoring Procedure Based on Created Process

Compliance with effective analytic scoring procedure has particular indicators

from two points of view. This section is focused on the created process’ point of

view; this following table explains about the result of the observation checklist

for the first rubric which assesses individual presentation and short conversation,

(53)

41

Table 4.1

Characteristic of Analytic Rubric based on Created Process

The table above is an illustration about the result of the observation checklist.

Specifically, related to the research question which tries to figure out the

compliance of the teacher’s rubric for assessing individual presentation and

rubric for assessing short conversation based on created process, these following

tables represent explanation about the result of the analysis using the observation

checklist and the interview.

(54)

42

Table 4.2

Characteristic of Analytic Rubric “Based on the Standard”

Aspects Rubric

The teacher just refers to the material that would be delivered. There is no reflection, and the rubric by using the standart of teaching refers to the material that would be delivered.

√

The table above shows that both rubrics are not based on the standard,

because the rubric developers did not make an official standard in creating the

rubric with the three steps based on the theory. In addition, the teacher’s respond

to the interview shows that there is no standard exactly in creating an assessment

rubric for speaking test, but he always considers what kind of task that he will