A CONTENT ANALYSIS ON A SECOND SEMESTER
TEACHER-COMPILED MODULE USED IN
SD TERUNA BANGSA FOR FOURTH GRADERS:
AUTOMATIC ENGLISH
A SARJANA PENDIDIKAN FINAL PAPER
Presented as Partial Fulfilment of the Requirements to Obtain the Sarjana Pendidikan Degree
in English Language Education
By
Ligolita Putri Fatoni Student Number: 071214108
ENGLISH LANGUAGE EDUCATION STUDY PROGRAM DEPARTMENT OF LANGUAGE AND ARTS EDUCATION
FACULTY OF TEACHERS TRAINING AND EDUCATION SANATA DHARMA UNIVERSITY
i
A CONTENT ANALYSIS ON A SECOND SEMESTER
TEACHER-COMPILED MODULE USED IN
SD TERUNA BANGSA FOR FOURTH GRADERS:
AUTOMATIC ENGLISH
A SARJANA PENDIDIKAN FINAL PAPER
Presented as Partial Fulfilment of the Requirements to Obtain the Sarjana Pendidikan Degree
in English Language Education
By
Ligolita Putri Fatoni Student Number: 071214108
ENGLISH LANGUAGE EDUCATION STUDY PROGRAM DEPARTMENT OF LANGUAGE AND ARTS EDUCATION
FACULTY OF TEACHERS TRAINING AND EDUCATION SANATA DHARMA UNIVERSITY
vii ABSTRACT
Fatoni, Ligolita Putri. 2014. A Content Analysis on a Second Semester Teacher-Compiled Module Used in SD Teruna Bangsa for Fourth Graders: Automatic English. Yogyakarta: English Education Study Program, Department of Language and Arts Education, Faculty of Teachers Training and Education, Sanata Dharma University.
Learning materials are important sources in the teaching and learning process. It helps the teacher to transform the curriculum into some interesting, useful, and practical activities in the classroom. One of the commonly found learning materials used by teachers is the module. Modules are said to be an effective tool to help teachers and students. This fact shows that a module should have certain criteria to help students understand the materials discussed at school. This research aims to examine whether or not the module used in SD Teruna Bangsa is included as an ideal module for students. There are two questions to answer in this study. The first is the criteria fulfilled by the module being analyzed, and the second is the strengths and weaknesses seen from the fulfillment of the criteria.
This research is a document analysis. There is a second-semester module for fourth graders as the subject for this research. The name of the module is
Automatic English. This module is analyzed using checklists for module evaluation. Some theories were used to support and provide the theoretical framework for this research. This research used a theory of workbook criteria, principles of language assessment, namely validity, authenticity, and reliability.
The result of the research indicated that the module, Automatic English, attained the criteria of a good module. There were different scores achieved in different criteria/aspects in the checklists. Regarding the result of the research, some suggestions were provided for the teacher to hopefully improve the module, especially to deal with the errors which are found in the module.
Keywords: content analysis, English module, module for fourth graders, module evaluation
viii ABSTRAK
Fatoni, Ligolita Putri. 2014. A Content Analysis on a Second Semester Teacher-Compiled Module Used in SD Teruna Bangsa for Fourth Graders: Automatic English. Yogyakarta: Program Studi Pendidikan Bahasa Inggris, Jurusan Pendidikan Bahasa dan Seni, Fakultas Keguruan dan Ilmu Pendidikan, Universitas Sanata Dharma.
Sumber materi pembelajaran merupakan suatu hal yang penting dalam proses belajar mengajar bahasa. Sumber materi pembelajaran tersebut dapat membantu seorang guru mengubah kurikulum menjadi suatu kegiatan yang menarik, bermanfaat, dan praktis. Salah satu sumber materi pembelajaran yang sering ditemukan dan digunakan oleh para guru adalah buku latihan. Buku latihan dipercaya sebagai media yang efektif untuk membantu para guru dan murid. Hal ini menunjukkan bahwa sebuah buku latihan seharusnya mempunyai kriteria tertentu untuk membantu para murid mengerti materi pelajaran yang sedang didiskusikan. Penelitian ini bertujuan untuk mengetahui apakah buku latihan yang digunakan oleh SD Teruna Bangsa termasuk sebagai buku latihan yang ideal bagi anak-anak atau tidak. Ada dua pertanyaan yang perlu dijawab dalam penelitian ini. Pertama, kriteria yang terpenuhi oleh buku latihan yang dianalisis, dan yang kedua adalah kekurangan dan kelebihan buku latihan yang dapat dilihat dari pemenuhan kriteria dalam checklists.
Penelitian ini merupakan analisis dokumen. Subyek dari penelitian ini adalah buku latihan semester dua untuk kelas empat SD yang berjudul Automatic English. Buku latihan tersebut dianalisis dengan menggunakan checklists berdasarkan teori-teori untuk menganalisis buku latihan.
Beberapa teori digunakan oleh penelitian ini untuk mendukung serta memberikan batasan pembahasan. Teori-teori yang digunakan adalah teori kriteria buku latihan, tiga prinsip penilaian pembelajaran bahasa, yaitu validity,
authenticity, dan reliability.
Hasil penelitian ini mengindikasikan bahwa buku latihan Automatic English dapat dikategorikan sebagai buku latihan yang baik tapi tidak semua kriteria tercapai secara maksimal.
Dengan melihat hasil dari penelitian ini, beberapa saran diberikan untuk para guru dengan harapan supaya dapat menjadikan buku latihan lebih baik, terutama mengenai kesalahan-kesalahan yang ditemukan dalam buku latihan.
Kata kunci: content analysis, English module, module for fourth graders, module evaluation
ix
ACKNOWLEDGMENTS
It is such a wonderful time that I could finally finish this long study. I would
like to express my gratitude to my Lord, Jesus Christ, who has blessed me and given
me guidance, courage, and help along this study. I am also grateful for those whom I
cannot mention all of the names, who have supported, guided, facilitated, and helped
for all this time.
Firstly, I would like to sincerely thank Ag. Prasetyo Hardi, S.Pd., M.A., my
sponsor who guided me, since without him my final paper would not have been
finished by now. My next gratitude goes to my beloved parents, Ch. Heri Fatoni and
Christina Maria Sudaryati, for their patience and their endless love for me and also
to my siblings, Sari ‘Pesek’ and Ian for their help, and supports.
I thank Drs. Barli Bram, M. Ed., Ph.D., for his meaningful help as my proof
reader. He helped me a lot by correcting the mistakes I have made. Then, my
gratitude goes to all PBI lecturers, for their knowledge and their guidance, so that I
can share my knowledge to others as well. My next gratitude goes to all the members
of Sanata Dharma University staff, especially in PBI secretary and in library, who
gave me a lot of help through the process of finishing this final paper. I also thank the
teacher from SD Teruna Bangsa, Esfie Lukmawati, S. Pd., for her help to share her
opinions about workbooks. It helped me a lot.
Then, I will never forget the power of friendship. I thank my great friends in
x
cannot mention one by one for teaching me about life, friendship, and boy-related
stuff. I will never forget the friendship, the tears and laughter we have shared. Million
of thanks for the priceless moments, adventures, and pictures we have been through.
Equally important, the Strong, Smart, and Sexy ladies in AZ dancers: Bubos
MbMae, Endah, Deli, Shela, Aning, Christin, Bibin, Melisonk, and still the others
I cannot mention all. I thank God for having them as my wonderful sisters. Thank
you, girls, for the sweats, tears, laughter, and the power. Last but not least, I would
like to thank my beloved sweetheart for the prayers, support, guidance, love and
experiences to share.
xi
TABLE OF CONTENTS
Page
TITLE PAGE ... i
APPROVAL PAGE ... ii
APPROVAL PAGE ... iii
DEDICATION PAGE ... iv
STATEMENT OF WORK’S ORIGINALITY ... v
PERNYATAAN PERSETUJUAN PUBLIKASI ... vi
ABSTRACT ... vii
ABSTRAK ... viii
ACKNOWLEDGEMENTS ... ix
TABLE OF CONTENTS ... xi
LIST OF FIGURES ... xiii
LIST OF APPENDICES ... xiv
CHAPTER I. INTRODUCTION ... 1
A. Research Background... 1
1. Content Analysis ... 7
2. A Teacher-Compiled Module ... 8
3. The fourth graders of SD Teruna Bangsa ... 9
4. Authenticity ... 9
5. Reliability ... 9
6. Validity ... 10
xii
CHAPTER II. DISCUSSION ... 18
A. Review of Related Literature ... 18
1. Principles of Language Assessment ... 18
a. Reliability ... 20
b. Validity ... 21
c. Authenticity ... 22
2. Approaches to Materials Evaluation ... 26
3. Evaluation Scheme ... 28
4. Description of an Automatic English Module ... 29
5. Learning Context ... 31
B. Findings and Discussion ... 32
CHAPTER III. CONCLUSIONS ... 39
xiii
LIST OF TABLES
II A.1. The Differences between Traditional and Authentic Assessment ... 23
II B.1. The Criteria of the Content Validity of the Module ... 32
II B.2. The Criteria of the Face Validity of the Module ... 33
II B.3. The Criteria of the Reliability of the Module ... 34
xiv
LIST OF APPENDICES
Appendix A. Checklists of Language Assessment Principles ... 43
1
CHAPTER I
INTRODUCTION
This chapter is allocated for discussing the background of the research and the
research method employed. Two questions are proposed in this study. In order to
make it more specific, this study is limited to three principles of language
assessment, namely authenticity, reliability and validity. The subject of this
research is a module for fourth graders in 2nd semester compiled by the English
teacher of SD Teruna Bangsa.
A.
Research Background
English language is spoken by millions of people, and the amount of
people using English as a means of communication are increasing time after time.
One of the reasons is that English is an international language. The development
of information, technology, education and other fields can be accessed through
English. People will be able to access what is happening in the world if they know
English. Working world nowadays is also demanding the employee candidates to
be able to speak English.
Knowing that fact, the Minister of Culture and Education in Indonesia
began to make a policy to include English as one of the local-content subjects to
teach at schools in order to prepare the students to the working world and
mentioned in Ministerial Decree of the Department Education and Culture
No.22/U/2006. It is hoped that since the school years, students will be accustomed
to English words and sounds and be able to speak it out and apply it to the real life
situation. As a foreign language, the careful preparation and examination of the
teaching and learning process should be taken into account. The things which will
support the process of language teaching and learning should be provided as the
media to transfer the knowledge and to reach the goals/ objectives which have
been set before, such as materials, media, and other supporting tools. Teachers‟
readiness is one of the factors that will greatly contribute in the process of English
language teaching and learning.
A teacher, as the key in the process of teaching and learning, needs to
realize how important his/ her role in the class is. He/ she should be able to
transform the curriculum, which is objectives, methodology, materials, and
assessment, to be the practical activities in the classroom. Therefore, the teacher
must have good background knowledge of the subject he/she is teaching. Skills
and abilities in the process of helping the students to get the knowledge and
assisting the students in classroom activities are also demanded for a teacher. In
other words, the English teacher should possess a good command of English and
sufficient knowledge about language teaching and learning in order to be able to
evaluate whether or not he/she is applying an appropriate teaching method and
approach and using meaningful materials.
Learning materials are important sources for the teacher to help him/her
modules, storybooks, recordings, brochures, flashcards, posters, video clips, and
others. Materials may serve the basis for the language input the learners receive
and the language practice in the classroom. Materials also provide the basis for the
content of the lessons, the skills taught, and the language practice that students
take part in. Since the learning materials have an important role in the process of
language teaching and learning, a teacher should be aware of selecting the
materials and putting it into practice.
This paper focuses on one of the learning materials used in English class
of SD Teruna Bangsa, namely Automatic English Grade 4 Semester 2 as the
subject of the research. Automatic English Grade 4 Semester 2 is a
teacher-compiled module and used as one of the assessment tools for the students. The
English teacher of SD Teruna Bangsa decided to use her own-compiled module as
to fulfill her students‟ needs and to support the achievement of the purpose and
objectives of learning English set by the schools.
Talking about language teaching and learning, the term assessment should
not be ignored for it is the integral part of the process of language teaching and
learning itself. It often happens that test and assessment are thought to be
synonymous. But, in fact, they are not the same. Assessment is different from test.
Brown (2003, p.4) explained that test is a part of assessment. “…Test is not the
only form of assessment that can be made. All tests are formal assessment, but not
all formal assessments are tests. A teacher can assess students‟ progress by getting
them make portfolios, researches, presentations, etc. Assessment covers a much
Furthermore, Brown (2003, p.4) argues that tests, which are a subset of
assessment which can provide authenticity, motivation, and feedback to the
learners, are essential components of a successful curriculum and one of several
partners in the learning process.
Assessment is needed in the process of language teaching and learning to
see how far the progress the students have made along the certain period of time
and also to evaluate the process of language teaching and learning itself.
Regarding to those issues, the writer uses some principles of language assessment
proposed by Brown (2003, p.19) in analyzing the “Automatic English” workbook
which is used as one of the materials in the process of English language teaching
and learning in SD Teruna Bangsa.
The principles of language assessment used in this paper are Reliability,
Validity, and Authenticity. Brown (2003, p.20) says that a reliable test is the one
which is consistent and dependable. A number of factors which contribute to the
unreliability of a test, as adapted from Mousavi (2002, p.804), are the possibilities
fluctuations in the students, in scoring, in test administration, and in the test itself.
A reliable test should yield similar results if it is given to the students in two
different occasions.
A test is said to be valid when the test measures what it is intended to
measure. A test cannot be valid unless it is reliable. The validity aspect covers
face and content validity. A valid test, as what has been explained by Brown
course work, clear and uncomplicated directions and items, and the appropriate
time allotment.
The last language principle being discussed in this paper is called
authenticity. Bachman and Palmer (1996, p.23) define authenticity as “the degree
of correspondence of the characteristics of a given language test task to the
features of a target language task.” In other words, a claim for authentic test is the
one that is likely to be enacted to the “real world”.
Those three principles proposed by Brown (2003, p.20) will be used to
analyze whether or not the Automatic English workbook is reliable, valid, and
authentic to use as the teaching and learning material and to analyze its strengths
and weaknesses.
The process of language teaching and learning will not run as what have
been expected if the components supporting the process are not well-prepared and
well-organized. Those components might be the media, environment, condition of
the students and teacher, and the learning materials as well. Learning materials
used are various depending on the objectives or goals of the lesson and the
creativity of the teacher herself. The sources of learning materials are for example
CD, cassette, recording, textbook, workbook, flashcard, handout, paragraph
written on the board, etc.
Nunan (1988, p.98) states that materials are, in fact, essential elements
within the curriculum, and they do more than simply lubricate the wheels of
models for desirable classroom practice. They act as curriculum models and at
their very best they fulfill a teacher development role.
In other words, teaching materials have direct influence on the process of
teaching and learning. The module, as one of the materials that is being used, of
course gives direct influence toward the way how the language is learned.
Knowing the fact that teaching and learning materials have an important effect
toward the progress/ result of the language learning, the significance of this study
is aimed to help the teacher to carefully examine the supporting materials‟
appropriateness toward the needs and characteristics of the students.
The study addresses two problems. The first problem is to know whether
or not the second semester teacher-compiled module: Automatic English Grade 4
has fulfilled the three criteria of principles of language assessment: reliability,
validity and authenticity. The second problem addressed is to know what the
strengths and weaknesses of the teacher-compiled module: Automatic English
Grade 4 Semester 2 as one of the means for assessing students are.
The objective of the study is to answer the questions proposed in the
problem formulation. The teacher-compiled module, Automatic English, is used
by SD Teruna Bangsa for certain considerations. The first reason is that the
English teacher feels that the school needs material sources that could
accommodate the needs of the students and the school. The English subject is
taught five days each week in this school and each meeting has time allotment 35
minutes. The English teacher often finds it is not efficient if she has to find
supported by the school committee, the English teacher decided to compile a
module to be used in the process of language teaching and learning.
Secondly, the teacher wants to accommodate what becomes students‟ need
more. She realizes how important the sources and kinds of materials are in the
process of language teaching and learning. Looking at these needs, the English
teacher of SD Teruna Bangsa compiled a module to be used for supporting the
process of language teaching and learning.
The study is concerned with the analyzing and evaluating the fourth grade
semester two elementary school module used in SD Teruna Bangsa, Automatic
English. The discussion is limited to the principles of English language
assessment. It attempts to find out whether or not the module used by SD Teruna
Bangsa, Automatic English, has fulfilled the principles of language assessment,
namely Reliability, Validity, and Authenticity.
The generalization of the research findings cannot be made to other
schools since they are not using the same module as what SD Teruna Bangsa is
doing.
To avoid different perceptions from what is being discussed, the
researcher has defined the specific terms used in the paper.
1. Content Analysis
Content analysis, which is often referred to as document analysis is one of
the approaches under the descriptive research. The aim of descriptive research as
stated by Ary, Jacobs, & Razavieh (2007, p.447) is to gain information about the
purposes of document analysis, some of which are to identify bias prejudice, or
propaganda in textbooks; to describe existing practice; to discover the level of
difficulty of material in textbooks or other publications; to discover the relative
importance of, or interest in, certain topics. Those purposes are proper to the
objectives of the research, which are aimed at discovering whether or not the
Automatic English module being analyzed has the criteria of book evaluation and
language assessment and to find out how those criteria are fulfilled or not, and to
see the module‟s strengths and weaknesses.
What is meant by content analysis in this paper is a process of analyzing
an Automatic English module used for fourth graders in SD Teruna Bangsa as one
of the assessment tools used by the English teacher. The analysis is done to see
the fulfillment of the criteria of book evaluation and language assessment and to
find out the strengths and weaknesses of the module. In other words, content
analysis is a procedure that involves examining learning materials to establish
their value (Tomlinson, 2003, p.146).
2. A Teacher-Compiled Module
A module is a book which is used in the process of teaching and learning
containing a lot of exercises and it plays an important role for the students to
practice. Poejosoedarmo (1978, p.317) defines a workbook as supplementary for
textbook besides teacher‟s manual book, and extensive reading book.
What it means by a teacher-compiled module in this paper is a book
compiled by the English teacher of SD Teruna Bangsa containing a list of
to be done by the students and arranged based on the certain topic/ theme/ unit.
The module is used as an ongoing assessment made by the teacher concerning
whether or not students have achieved the goals/ objectives of the lesson being
discussed.
3. The fourth graders of SD Teruna Bangsa
Fourth graders of SD Teruna Bangsa in this paper refer to the students
whose ages are ranging from eight until ten years old. The fourth graders consist
of thirty students from different social backgrounds. Teruna Bangsa is one of the
private schools in Yogyakarta located in Villa Seturan Indah D10 Caturtunggal,
Depok, Sleman, Yogyakarta.
4. Authenticity
It is one of the language assessment principles. Bachman and Palmer
(1996, p.23) define authenticity as “the degree of correspondence of the
characteristics of a given language test task to the features of a target language task,” and then suggest an agenda for identifying those target language tasks and
for transforming them into valid test items.
Basically, an assessment is said to be authentic when the tasks/ exercises
given have the as-natural-as-possible language, based on real-world task,
contextualized items, and meaningful topics organization (Brown, 2003, p.28).
5. Reliability
Reliability is one of the language assessment principles which concerns
about the consistency and dependability. This principle is affected by several
administration reliability, and test reliability. This paper is focused on test
reliability and it deals with the nature of the test itself. The sources of test
unreliability may from the inappropriate time allotment (the amount of time
provided is not suitable with the test items) and the poorly written test items (that
are ambiguous or that have more than one correct answer) (Brown, 2003, p.28).
6. Validity
An assessment is said to be valid if it measures what is intended to
measure. Validity is the extent to which inferences made from assessment results
are appropriate, meaningful, and useful in terms of the purpose of the assessment
(Gronlund, 1998, p.226). There is no final, absolute measure of validity, but
several different kinds of evidence may be used to examine the validity of the test
items. There are five types of evidences proposed by Brown (2003, p.22): Content
validity (content-related evidence), criterion-related evidence, construct-related
evidence, consequential validity, and face validity. From those evidences, there
are two aspects which will be discussed in terms of validity in this paper, i.e.
content validity and face validity.
Brown (2003, p.22) defines content validity as a situation in which a test
requires the test-taker to perform the behavior that is being measured. The content
validity can be identified observationally if the achievement being measured can
be clearly defined. In other words, content validity deals with the requirements to
perform the previously-learned lessons and represent the objectives of the unit on
The extent to which “students view the assessment as fair, relevant,
and useful for improving learning” (Gronlund, 1998, p.210) is popularly
known as face validity. Mousavi (2002, p.244) explains that face validity
refers to the degree to which a test looks right, and appears to measure the
knowledge or abilities it claims to measure, based on the subjective judgment
of the examinees who take it, the administrative personnel who decide on its
use, and other psychometrically unsophisticated observers. Face validity is
seen from students‟ points of view. The test is said to be face-valid if it has a
clear direction, a logically-organized structure of the test, an appropriate
difficulty level, has no „surprises‟, and an appropriate timing (Brown, 2003,
p.27).
B. Research Method
Document analysis is one of the approaches in descriptive research. There
are some approaches under descriptive research. Seven approaches are mentioned
by Ary et al. (2007, p.447): case studies, surveys, developmental studies,
follow-up studies, document analysis, trend analysis, and co relational studies. Moreover,
Ary, Jacobs, and Razavieh (2007, p.304) state that “documentary analysis, often
referred to as content analysis, is not confined to simple counts but can also be used to study sociological and psychological variables.” There are some purposes
of document analysis in educational research listed by Ary et al. (2002, p.443),
namely: identifying bias prejudice, or propaganda in textbooks; analyzing types of
level of difficulty of material in textbooks or other publications; discovering the
relative importance of, or interest in, certain topics.
The fact that this research is aimed at discovering whether or not the
Automatic English module being analyzed has the criteria of book evaluation and
language assessment and to find out how those criteria are fulfilled or not, and to
see the module‟s strengths and weaknesses, it can be concluded that identifying
bias prejudice or propaganda in textbooks, describing the prevailing practice,
discovering the level of difficulty of material in textbooks or other publications,
and discovering the relative importance of or interest in certain topics are proper
to the objectives of the research.
Automatic English Grade 4 Semester 2 module is the document which is
analyzed to discuss how the principles of language assessment are met. The use of
modules to support the process of language teaching and learning is commonly
found at schools. The widely published books demand teachers to carefully select
what become the most appropriate material sources to support the process of
language teaching and learning.
In this study, the researcher attempts to analyze the module used in SD
Teruna Bangsa entitled Automatic English. This module is used from the first
until the sixth grade. The students are already familiar with the module since they
already use it in the early time they study in that school. This study focuses on the
module used for fourth graders in the second semester. The module Automatic
English is compiled by the English teacher of SD Teruna Bangsa. She realizes
the students and the teacher herself. Because the time allotment for English
subject in SD Teruna Bangsais 5 days a week with each meeting‟s time allotment
35 minutes, the teacher feels that she needs some extra more materials to cover
the extra time allotment.
The researcher is the primary instrument for this research. Merriam (2009)
argues that a human instrument will be the ideal means of collecting and
analyzing data because human instrument is able to be responsive and adaptive to
the process of gathering the data.
The module as the document to analyze is the second instrument. Ary et
al. (2010, p.442) state, “Qualitative researchers may use written documents or
other artifacts to gain an understanding of the phenomenon under study.” Since
the researcher analyzes the module directly, the document used in this research is
considered as a primary source. For the module evaluation, the researcher
provides checklists.
The checklists used contain the following criteria: firstly, it is general
appearance. This criterion is related to cover, the font size and type of the letter
used in the module, the informative pages and table of contents, title chosen for
every unit/ lesson, the glossary, and references. Secondly, it deals with design and
illustration. Design includes the consistency in the use of headings, icons, labels,
italics, et cetera, whereas the illustration is related to images, charts, and tables
used in the module. Accompanying materials are the third criterion used in the
checklists. It is related to the recording, posters, flash cards, games used to
The objectives concern to the materials presented in the module. It should be
arranged based on the standard competence and basic competence appropriate for
fourth graders, so that the needs and wants of the students and teacher can be
accommodated. The fifth criterion is topic contents. It concerns about the ability
of the topics to interest students and the variety to accommodate and attract the
different types of learners.
Language skills which contain the criteria of whether or not the four
elements of language skills: listening, speaking, reading, and writing are
adequately given and allotted or not are the sixth criterion used in the checklists.
Teachability is going to be the seventh criterion in the checklists. This aspect is
related to the ability of the module to help the teacher in supporting the process of
language teaching and learning. The eighth criterion is flexibility. It measures the
ability of the module to be adapted in such a way to fulfill different style of
learning needs.
Practice and testing which deal with the numbers and kinds of exercises
provided in the module, the clarity of direction of the tasks/ exercises, and the
existence of the model of achievement test are used for the ninth criterion. The
tenth criterion is language content. This is related to the vocabulary,
pronunciation, and grammar used in the module. The last criterion available in the
checklists is social and cultural contexts. This aspect concerns about how the
module gives a positive impact toward students regarding to the social and
cultural contexts which includes the positive views toward ethnic origins,
be socially and culturally comprehensible to the students‟ cultural background to
be applied in daily life.
The data was gathered from the Automatic English module for fourth
graders in the second semester. The researcher analyzed the module based on the
criteria for book evaluation presented to get the data. The material evaluation
checklistss consist of seven columns. The first column is for number followed by
the principles of language assessment covered (reliability, validity, and
authenticity), then the third is for criteria/aspects which are measured, and the
fourth, fifth, and sixth column are provided for knowing the quality of each
criteria/aspects stated (ranging from 3 to 1), and the last column is for the score of
each criteria/aspects stated in each number. To measure the quality of the criteria
stated, the researcher uses numbers to indicate, those are 3 (three) to indicate that
the criteria/aspects are fully met in the module, 2 (two) to show that the module
has the criteria/aspects stated in the checklistss but not as perfect/appropriate/good
as the point 3 (three), and 1 (one) to indicate that the module does not have the
criteria/aspects mentioned in the checklists. The researcher bolds the column to
know the score of each criterion in the checklistss, and then writes the score in the
last column.
The calculation of the data gained is using this formulation:
Percentage= ∑X X 100%
N
∑X = the total criteria points which are met in the module for each aspect
Each principle of language assessment is also measured using the formula
presented above. Therefore, for the validity aspect, the calculation will be:
Percentage= ∑X X 100%
N
∑X = the total points of validity which are met in the module
N = the maximum score of the validity aspect (60 points)
Since there are two different aspects for measuring the validity of the
module (content validity and face validity), then the calculation is also applied for
each of the aspect. It will be as follows:
Percentage= ∑X X 100%
N
∑X = the total points of content/ face validity which are met in the module
N = the maximum score of the content/ face validity aspect (30 points each)
Whereas the calculation for reliability aspect will be in this way:
Percentage= ∑X X 100%
N
∑X = the total points of reliability which are met in a module
The calculation for authenticity aspect will be:
Percentage= ∑X X 100%
N
∑X = the total points of authenticity which are met in a module
N = the maximum score of the authenticity aspect (39 points)
The researcher uses these conditions to decide the result of the data analysis:
79 % - 100 % good
57 % - 78 % fair
18
CHAPTER II
DISCUSSION
Chapter II covers the theories supporting to the study. It deals with the
approaches used to evaluate the material, which is Automatic English module, and
the scheme of evaluation. The information about Automatic English module is
explained in chapter II along with the learning context of the module. The
research result is discussed in this chapter.
A.
Review of Related Literature
This chapter is allocated to review the literature related to the main topic,
evaluation of a module. It deals with a number of theoretical studies that
contribute to the field of module evaluation.
1.
Principles of Language Assessment
Assessment is a popular term in educational practice but it is sometimes
misunderstood. Assessment and test are not synonymous; they are different. As
what has been stated by Brown (2003, p.4), tests are prepared administrative
procedures that occur at identifiable times in a curriculum when learners muster
all their faculties to offer peak performance, knowing that their responses are
being measured and evaluated. Therefore, a test, in simple terms, is a procedure
for measuring a learner‟s ability, knowledge, or performance in a given domain.
Assessment, on the other hand, is an ongoing process that encompasses a
question, offers a comment, or tries out a new word or structure, the teacher
subconsciously makes an assessment of the student‟s performance. Based on what
has been explained by Brown (2003, p.4), tests, then, are a subset of assessment;
they are not the only form of assessment that a teacher can make. All tests are
formal assessment, but not all formal assessment is testing. Teacher can use
students‟ journal, portfolios as a formal assessment instead of testing. Assessment
is an integral part of the teaching-learning cycle. In an interactive, communicative
curriculum, assessment is almost constant. Tests, which are a subset of
assessment, can provide authenticity, motivation, and feedback to the learner.
Tests are essential components of a successful curriculum and one of several
partners in the learning process.
Basic principles about assessment as proposed by Brown (2003, p.16) are
assessment can increase motivation by serving as important events of student
progress; appropriate assessments aid to the reinforcement and retention of
information; assessments can confirm areas of strength and pinpoint areas needing
further work; assessments can provide a sense of periodic closure to modules
within a curriculum; assessments can promote student autonomy by encouraging
students‟ self-evaluation of their progress; assessments can encourage learners to
set goals for themselves; and assessments can aid to evaluating teaching
effectiveness.
One of the ways to see the learners‟ progress is by giving them one form
of assessments, i. e. a test. There are some principles which need to ponder and
(and also any kind of assessment) to be effective and appropriate. As proposed by
Brown (2003, p.19), there are five principles of language assessment which can be
used to see whether the test/ assessment being used are effective or not.
This paper focuses on the three principles proposed by Brown (2003), and
they are reliability, validity, and authenticity. Those three principles will be used
to see the fulfillment of language assessment‟s principles of the Automatic
English module for fourth graders in second semester and to see its strength and
weaknesses, as what has been mentioned in the purpose of this study.
a. Reliability
A reliable test is consistent and dependable. What it means by consistent
and dependable is that if a teacher gives the same test to the same students or
matched students on two different occasions, the test should yield similar results
(Brown, 2003, p. 21).
Establishing reliability is a prerequisite for establishing validity. Although
a valid assessment is by necessity reliable, the contrary is not true. A reliable
assessment is not necessarily valid. The issue of reliability of a test may best be
addressed by considering a number of factors that may contribute to the
unreliability of a test (adapted from Mousavi, 2002, p.804). The first factor is
from students (student-related reliability), and the second factor which may
contribute to the unreliability of a test is from rater‟s scoring system (rater
-reliability). Test administration (test administration reliability) also contributes to
unreliability of a test. The last factor which contributes to unreliability of a test is
The study focuses on one of the factors which may contribute to the
unreliability of a test, i.e. test reliability. Test reliability deals with the nature of
the test itself. Sometimes the nature of the test can cause measurement errors.
Inappropriate time allotment may affect the performance of students who do not
perform well on a test with a time limit. Another factor contributing to
unreliability of a test is the poorly written test items (which are ambiguous or have
more than one correct answer).
b.Validity
Everitt (2002, p.388) defines validity as the extent to which a measuring
instrument is measuring what was intended. In other words, it can be said that a
valid test is a test which measures what it is intended to measure. It requires
students to perform tasks that were included in the previous classroom lessons.
Brown (2003, p.22) explains that there is no final, absolute measure of
validity, but several different kinds of evidences may be invoked in support. Some
evidences which may support a test to be valid are content validity,
criterion-related validity, construct validity, consequential validity, and face validity. From
those evidences, there are two evidences which is used and discussed to help the
researcher analyzing the Automatic English module. Those are content validity
and face validity. A test is said to be content-valid if it requires the students to
perform the behavior that is being measured. For example, if a teacher is trying to
assess a student‟s ability to speak, then, a test which requires the learner to speak
within some sort of authentic context is a content-valid test. Face validity is the
improving learning (Gronlund, 1998, p.210). Face validity refers to the degree to
which a test looks right, and appears to measure the knowledge or abilities it
claims to measure, based on the subjective judgment of the examinees who take it
(Mousavi, 2002, p.244). In other words, face validity means that the students
perceive the test to be valid. Brown (2003, p.27) states that face validity will
likely be high if learners encounter a well-constructed with familiar tasks in an
expected format, a clearly doable test within the allotted time limit, items that are
clear and uncomplicated, directions that are crystal clear, tasks that relate to their
course book (content validity), and a difficulty level that presents a reasonable
challenge. Content validity is a very important ingredient in achieving face
validity. If a test samples the actual content of what the learner has achieved or
expects to achieve, then face validity will be more likely to be perceived.
c. Authenticity
Bachman and Palmer (1996, p.23) define authenticity as “the degree of
correspondence of the characteristics of a given language test task to the features
of a target language task. Its aim is to assess many different kinds of literacy
abilities in contexts that closely resemble actual situations in which those abilities
are used. For example, authentic assessments ask students to read real texts, to
write for authentic purposes about meaningful topics, and to participate in
authentic literacy tasks such as discussing books, keeping journals, writing letters,
and revising a piece of writing until it works for the reader. Both the material and
the assessment tasks look as natural as possible. In other words, a test is said to be
natural language, contextualizing the test item, giving meaningful (relevant,
interesting) topics for the learners, providing thematic organization to the item
(e.g. through story line or episode), giving test which represent or closely
approximate real world task.
An example of authentic assessment that can be made is after completing a
first-grade theme on families in which students learned about being part of a
family and about the structure and sequence of stories, students might illustrate
and write their own stories with several parts, telling a story about how a family
member or friend helped them when they were feeling sad.
The formats for performance/authentic assessments range from relatively
short answers to long-term projects that require students to present or demonstrate
their work. These performances often require students to engage in higher-order
thinking and to integrate many language arts skills. Consequently, some
performance assessments are longer and more complex than what it is in
traditional assessments.
Traditional --- Authentic
Selecting a Response --- Performing a Task
Contrived --- Real-life
Recall/Recognition --- Construction/Application
Teacher-structured --- Student-structured
[image:38.595.104.525.304.765.2]Indirect Evidence --- Direct Evidence
There are some characteristics which differentiate an authentic assessment
from traditional assessment. Firstly, it is selecting a response to performing a task.
On traditional assessments, students are typically given several choices (e.g., a,b,c
or d; true or false) and asked to select the right answer. In contrast, authentic
assessments ask students to demonstrate understanding by performing a more
complex task which is usually representative of more meaningful application.
The second characteristic which differentiates an authentic assessment
from a traditional assessment is that it is contrived to real-life. It is not very often
in life outside of school that students are asked to select from four alternatives to
indicate our proficiency at something. Tests offer these contrived means of
assessment to increase the number of times students can be asked to demonstrate
proficiency in a short period of time. More commonly in life, as in authentic
assessments, students are asked to demonstrate proficiency by doing something.
Thirdly, it is recognition of knowledge to construction/application of
knowledge. Well-designed traditional assessments (i.e., tests and quizzes) can
effectively determine whether or not students have acquired a body of knowledge.
Thus, as mentioned above, tests can serve as a nice complement to authentic
assessments in a teacher's assessment portfolio. Authentic assessments often ask
students to analyze, synthesize and apply what they have learned in a substantial
manner, and students create new meaning in the process as well.
The fourth factor is teacher-structured to student-structured. When
completing a traditional assessment, what a student can and will demonstrate has
attention will understandably be focused on and limited to what is on the test. In
contrast, authentic assessments allow more student choice and construction in
determining what is presented as evidence of proficiency. Even when students
cannot choose their own topics or formats, there are usually multiple acceptable
routes towards constructing a product or performance. Obviously, assessments
more carefully controlled by the teachers offer advantages and disadvantages.
Similarly, more student-structured tasks have strengths and weaknesses that must
be considered when choosing and designing an assessment.
The last factor which differentiates between an authentic assessment and
traditional one is the indirect evidence and direct evidence. Even if a
multiple-choice question asks a student to analyze or apply facts to a new situation rather
than just recall the facts, and the student selects the correct answer, what do you
now know about that student? Did that student get lucky and pick the right
answer? What thinking led the student to pick that answer? We really do not
know. At best, we can make some inferences about what that student might know
and might be able to do with that knowledge. The evidence is very indirect,
particularly for claims of meaningful application in complex, real-world
situations. Authentic assessments, on the other hand, offer more direct evidence of
application and construction of knowledge. As in the golf example above, putting
a golf student on the golf course to play provides much more direct evidence of
2.
Approaches to Materials Evaluation
Tomlinson (2003, p.15) defines materials evaluation as „a procedure that
involves measuring a value (or potential value) of a set of learning materials‟. An
evaluation focuses largely on the needs of the users of the materials and makes
subjective judgment about their effects. An evaluation might include a question,
for instance, „Do the reading texts sufficiently engage learners?‟, which elicit
responses containing a necessarily subjective value judgment.
Evaluation can be carried out pre-use, in-use, or post-use. The main aim of
evaluating materials pre-use, according to Rubdy (2003, p.42), is to measure the
potential of what a teacher and learners can do with them in the classroom. In-use
and post-use evaluations are important in establishing how successful learning
materials are (McDonough & Shaw, 2003, p.71).
With the widespread of commercially produced textbooks supporting the
teaching and learning process, teachers need to be careful selecting the materials
used in the process of language teaching and learning. Some modification,
adaptation, and development are needed in order to meet the needs and demands
of the learners.
A number of theoretical evaluative frameworks regarding materials
evaluation have been published (e.g. in Williams, 1983; Breen & Candlin, 1987;
Sheldon, 1988; McDonough & Shaw, 2003; Cunningsworth, 1995; and McGrath,
2002). These have mostly been checklist-based, usually in the form of questions
to be answered to determine the extent to which the materials fulfill a set of
The advantages and disadvantages of checklists have been pointed out by
several writers. Not only can checklists be systematic and comprehensive, they
are also time and cost effective, and the results are easy to understand, replicate
and compare (McGrath, 2002, pp.26-27). Sheldon (1988, p.242) suggests that
considerable modification of any set of culturally restricted criteria needs to be
done to make checklists applicable to most cultural contexts.
Tomlinson (2003, p.16) argues that materials evaluation is a procedure that
involves measuring some or all of the following: the appeal of the materials, the
credibility of the materials to learners, teachers, and administrators, the validity of
the materials, the reliability of the materials, the ability of the materials to interest
the learners and the teachers, the ability of the materials to motivate the learners,
the value of the materials in terms of short-term and long-term learning, the
learners‟ and teacher‟s perceptions of the value of the materials, the assistance
given to the teacher (in terms of preparation, delivery, and assessment), the
flexibility of the materials, the contribution made by the materials to teacher
development, and the match with administrative requirements.
In this study, a checklist-based system is employed, with some
modification required to make the evaluation more suitable for the researcher‟s
aim and context. A questionnaire given to the teacher is also used to gain more
information concerning about the focus of the study. The questionnaire consists of
the following categories: the introduction of the subject/ course, course‟s subject
matter, aids, exercises, and activities, the teacher‟s manual, the course book layout
3.
Evaluation Scheme
Hutchinson and Walters (1987, pp.37-38) suggests an interactive view of
material evaluation. They emphasize the deeper level of materials evaluation by
asking the question why materials are the way they are. They claim that materials
evaluation plays such an important role in language teaching that its potential for
influencing the way teachers operate is considerable. Materials evaluation can and
should be a two-way process which enables teachers not just to select a textbook,
but also to develop their awareness of their own teaching/learning situation.
Hutchinson and Walters (1987, p.96) give this definition, „Evaluation is a
matter of judging the fitness of something for a particular purpose.‟
Cunningsworth (1984, p.64) puts forwards the idea „…that the process of
evaluation could not be a purely mechanical one and that professional judgment
was involved at every stage.‟He goes on to say, „Professional judgment, founded
on understanding of the rationale of language teaching and learning and be backed
up by practical experience, lies at the base of evaluation procedure‟. In other
words, materials evaluation helps us to make decisions in selecting materials,
form professional judgment as well as raise awareness of or reflect on our
teaching and learning experience.
Having seen the significance of evaluating materials for language teaching
and learning process, let us move to the things concerned with the ways of
evaluating materials and the criteria followed in the process of evaluation.
Sheldon (1988, p.237) maintains that since no one‟s set of criteria is
checklists or scoring system. The process of evaluation is inevitably a subjective
activity. To avoid the danger of allowing subjective factors to influence judgment
in the analysis, Hutchinson and Waters (1987, p.97) emphasize the importance of
objectivity in evaluation. „Evaluation is basically a matching process: matching
needs to available solutions. If this matching is to be done as objectively as
possible, it is best to look at the needs and solutions separately.‟ Thus, the
evaluation process is divided into four major steps, i.e. 1.defining criteria,
2.subjective analysis, 3.objective analysis, and 4.matching.
The materials evaluation scheme used in this study is a checklist-based
using the quality of the certain criteria in each aspect ranging from 1 to 3. The
evaluation assesses the materials in relation to the following areas: general
appearance, design and illustration, accompanying materials, objectives, topic
contents, language contents, social and cultural contexts, language skills,
teachability, flexibility, practice and testing. The evaluation scheme is designed
with the aim of evaluating the ability of the materials to fulfill the principles of
language assessment which are shown in the areas of materials evaluation above
in order to promote the teaching and learning process and to see the strengths and
weaknesses of the material being used.
4.
Description of an Automatic English Module
The module Automatic English which is compiled by the English teacher
of SD Teruna Bangsa is used for the students of SD Teruna Bangsa in the first
until the sixth grades. The study is going to focus on the module used by the
The module consists of five units of the lesson to cover in the second
semester. There is an extra material accompanying each unit. This extra material
is given for the students who are fast-pace type of learners so that they still have
activities to do and to avoid disturbing the other students. A list of vocabulary is
given in the end section of the module with Indonesian translation. The topics
arrangement of the second semester module of Automatic English is as follows:
Unit 6 Clothes and Extra
Unit 7 Health Problems and Extra
Unit 8 In the Mall and Extra
Unit 9 On the Street and Extra
Unit 10 On the Phone and Extra
Vocabulary
References
Each topic of the lesson in this module includes four skills activities
namely listening, speaking, reading, and writing. Song is also given in the
end of each topic with the picture of illustration. There is a bible quote in the
beginning of each topic which is related to the topic presented. English lesson
in Teruna Bangsa is taught five days a week with the time allotment each
meeting 35 minutes. The teacher feels that it is not efficient if she has to find
additional materials to cover the lesson and she realizes how important the
materials which can fulfill students‟ needs are in the process of language
teaching and learning. Moreover, the decision for making and using the
those reasons, the English teacher of SD Teruna Bangsa decided to compile a
module to be used in the process of language teaching and learning.
The module is printed in an A4-size paper with 81-page width. The
module is not that thick. Therefore it is easy for students to bring. The fonts
of the letters are clearly printed with an appropriate size. To avoid boredom,
the activities/tasks available in the module are various, ranging from
matching, filling in the blanks, dialogue practicing, and drawing. The pictures
and illustrations presented in the module are in black and white ink. The
references of the module are taken from the internet.
5.
Learning Context
The learning context is a private school in Yogyakarta. The students
have a five-day English subject in a week with time allotment 35 minutes in
each meeting. The focus of this paper is fourth grade learners aged between
eight and ten. The learners of this school have been introduced to English
subject since the first grade. All learners are familiar with the Automatic
English module series for they have used it throughout their studies at the
school.
One of the measuring tools for learners‟ progress is by doing the
exercises of the module. Besides this, the teacher gives regular tests to
measure learners‟ progress. The learners are given certain time allotment to
complete the tasks/exercises on each topic of the module. This will be used
B.
Findings and Discussion
There are three principles of language assessment discussed in this study,
namely validity, reliability, and authenticity. The checklists used to analyze and
evaluate the module consist of 47 items/criteria/aspects with the distribution as
follows:
1. Validity : There are 20 items for analyzing the validity of the module, with
10 items each on content validity and face validity. Those 20 items are
divided into two parts.
The first part is to see the content validity of the module. It consists of 10
items. The items deal with objectives, teachability, and practice and testing.
[image:47.595.102.511.197.688.2]The table is shown in table B.1:
Table B.1. The criteria of the content validity of the module
No. Principles of Language
Assessment Covered Criteria/Aspects
1. Validity (content-valid) Generally, the module fulfills the general objectives of teaching English language in Indonesia (objectives) 2. Validity (content-valid) Generally, the module fulfills the general objectives of
teaching english for elementary level (objectives) 3. Validity (content-valid) The objectives specified explicitly in the module are
based on some theoretical background (objectives) 4. Validity (content-valid) The objectives define the desired degree of mastery
(objectives)
5. Validity (content-valid) The objectives meet the needs and wants of fourth graders (objectives)
6. Validity (content-valid) The objectives are clear and precise (objectives) 7. Validity (content-valid) The objectives are measurable (objectives) 8. Validity (content-valid) The module helps teachers exploit the activities to meet
students' expectation (teachability)
9. Validity (content-valid) The module provides periodical revisions for diagnostic purposes (practice and testing)
The second part of the validity is to check the face validity of the module. It
consists of 10 items. This part deals with general appearance, objectives, language
skills, flexibility, practice and testing, language contents, and social and cultural
[image:48.595.103.508.235.546.2]contexts as shown in table B.2.
Table B.2. The criteria of the face validity of the module
No.
Principles of Language
Assessment Covered Criteria/Aspects
11. Validity (face-valid) The cover is informative and attractive to young learners (general appearance)
12. Validity (face-valid) The font size and type used in the module are appropriate for fourth graders (general appearance)
13. Validity (face-valid) The module has a complete and detailed table of contents (general appearance)
14. Validity (face-valid) The objectives suit the level of fourth graders (objectives) 15. Validity (face-valid) The four skills are adequately covered (language skills) 16. Validity (face-valid) The module provides opportunity for teachers and students
to localize activities (flexibility)
17. Validity (face-valid) Every exercise has a clear direction (practice and testing) 18. Validity (face-valid) The tests are valid and contain correct language (practice
and testing) 19. Validity (face-valid)
The materials for teaching grammar, vocabulary, and pronunciation are graded in an appropriate manner (language contents)
20. Validity (face-valid) The social and cultural contexts in the module are comprehensible (social and cultural contexts)
2. Reliability : 14 items.
The reliability of the module is analyzed using 14 items from the checklist.
The criteria/aspects analyzed are from general appearance, design and
illustration, accompanying materials, language skills, teachability, flexibility,
practice and testing, and language contents. Table B.3 shows the reliability
Table B.3. The criteria of the reliability of the module
No.
Principles of Language
Assessment Covered Criteria/ Aspects
21. Reliability The module has an appropriate glossary (general appearance) 22. Reliability The module has a complete references (general appearance) 23. Reliability There is enough white space to achieve clarity (design and
illustration)
24. Reliability There is consistency in the use of headings, icons, labels, italics, etc (design and illustration)
25. Reliability The illustrations are functional (design & illustration) 26. Reliability Recordings that accompany the module are suitable
(accompanying materials) 27. Reliability
Listening material is well recorded, as authentic as possible, and accompanied by background information, questions, and activities (language skills)
28. Reliability
There is sufficient reading material (there is a range of varied and interesting reading text that can engage students
cognitively and effectively) (language skills)
29. Reliability There is sufficient material for spoken english (e.g. dialogues, role-plays, etc) (language skills)
30. Reliability The module helps teachers to minimize their preparation time (teachability)
31. Reliability The module caters for different preferred learning styles (flexibility)
32. Reliability There is a reasonable and appropriate number of exercises (practice and testing)
33. Reliability The module includes adequate materials for teaching vocabulary (language contents)
34. Reliability The module includes adequate material for pronunciation work (language contents)
3. Authenticity : 13 items.
There are 13 items stated in the criteria/aspects for measuring the authenticity
of the module. Those criteria/aspects deal with general appearance, design
and illustration, accompanying materials, topic contents, flexibility, practice
Table B.4. The criteria of the authenticity of the module
No.
Principles of Language
Assessment Covered Criteria/ Aspects
35. Authenticity Every lesson is given an appropriate title (general
appearance)
36. Authenticity There is a variety of design (design and illustration) 37. Authenticity The illustrations are varied and attractive (illustration design &
)
38. Authenticity The illustrations stimulate students to be creative (design &
illustration)
39. Authenticity The posters and flashcards that accompany the module are suitable (accompanying materials)
40. Authenticity
The topics of the module are varied and engaging to appeal the fourth graders with different interests and personalities (topic contents)
41. Authenticity The module avoids potentially embarassing or disturbing topics (topic contents)
42. Authenticity The module is appealing and useful to the students (flexibility) 43. Authenticity
The module provides a variety of meaningful and mechanical exercises and activities to practice language items and skills (practice and testing)
44. Authenticity
The module provides communicative exercises and activities that help students carry out their communicative tasks in real life (practice and testing)
45. Authenticity The content of the module is free from stereotypical images and information (social and cultural contexts)
46. Authenticity
The modul
