English Teachers’ Assessment Literacy and Online Formative Assessment Implementation in Teaching Writing
(Teaching Writing: A mixed method study )
By
SYIFA DWI MUTIAH 21190140000004
GRADUATE PROGRAM OF ENGLISH EDUCATION FACULTY OF EDUCATIONAL SCIENCES
SYARIF HIDAYATULLAH STATE ISLAMIC UNIVERSITY JAKARTA
2022 M / 1443 H
Abstract
This research attempted to investigate the relationship between English teachers’
assessment literacy levels and the implementation of online formative assessment scores in a rural district area of Cilacap regency, precisely in the Kawunganten district. This study further discovered the imbalance between the teachers’
assessment literacy and their implementation scores of online formative assessments.
As the next step, this research tried to explore the mediated factors that contributed to the relationship between them. The assessment literacy and the implementation of online formative assessment were simultaneously measured through the adapted Classroom Assessment Literacy Inventory tests and questionnaires. After that, an interview was needed for more in-depth results. Automatically, this research used the explanatory sequential mixed method as its design. Twenty-eight teachers incorporated in Junior High School English Teacher Association (MGMP) participated in this study.
The result showed that there was no relationship between English teachers’
assessment literacy and their implementation of online formative assessment scores since the two tailed significant result was larger at 0.174 than the r table at 0.05.
There were three teachers obtained highest scores of assessment lietarcy. They obtained 100 score with 73,54 average scores. In contrast, the implementation of online formative assessment scores was low. It evidenced by only one teacher got 100 scores, one other teacher obtained 60 as the second highest score, while the rest obtained below that it. There were three teachers obtain the lowest score; three scores, and two teachers with five scores. In addition, It was collectively at 23.07 average.
This mismatch between the levels of assessment literacy and online formative assessment scores emerged some factors that contributed to it by analyzing some high frequent correct answer questions and the lowest number of answers in the questionnaire. Four questions from CALI and five from the questionnaire were used as the interview benchmark.
Furthermore, these results followed the interview section of six teachers who obtained an imbalance scores between the assessment literacy and online formative assessment implementation in teaching writing. It showed that the factors that mediated those gaps above were mainly caused by the facility provision, students’ motivation, and internet signal accessibility.
Keywords: Assessment Literacy, Online Formative Assessment, Mediated Factors between Assessment Literacy and Online Formative Assessment.
Abstrak
Penelitian ini bertujuan untuk mengetahui hubungan antara nilai assessmen literasi guru dan nilai implementasi mereka terkait penilaian formatif secara daring di desa Kawunganten, Kabupaten Cilacap. Lebih jauh lagi, penelitian ini juga menginvestigasi nilai yang tidak seimbang antara literasi assessmen ketagori guru dan implementasi penilaian formatif mereka dalam mengajar writing.
Ketidakselarasan nilai tersebut menimbulkan hasil kalkulasi yang menyatakan bahwa tidak ada hubungan antara nilai assessmen literasi guru dan nilai impelemntasi penilaian formatif mereka didalam mengajar skill menulis secara online. Kemudian, penelitian ini juga menyelidiki faktor-faktor yang mempengaruhi terjadinya ketidak seimbangan tersebut. Aspek literasi assessment diukur dengan menggunakan tes Classroom Assessment Literacy Inventory (CALI), sedangkan nilai implementasi penilaian formatif secara daring didapatkan melalui kuesioner implementasi penilaian formatif yang telah diadaptasi kedalam lingkungan online. Hasil dari kedua instrument tersebut kemudian dianalisis dan dilanjutkan dengan interview. Maka, metode yang digunakan pada penelitian ini adalah metode campuran dengan desain Explanatory sequential. Dua puluh delapan guru yang bergabung dalam komunitas Musyawarah Guru Bahasa Mata Pelajaran (MGMP/KKM) Bahasa Inggris telah mengikuti tes literasi assessmen (CALI) yang telah diadaptasi dan survey implementasi penilaian formatif online. Hasil dari tes hipotesis (1 dan 2 tailed) menunjukan nilai signifikansi hubungan antara keduanya lebih besar (0.174) daripada nilai rtabel (0.005).
Rata-rata nilai guru sebesar 73.54 dengan tiga guru meraih nilai tertinggi, yakni 100 poin. Disisi lain, hasil survey tentang implementasi penilaian formatif secara online menunjukan rata-rata nilai mereka adalah 23.07. Hanya ada satu guru yang mendapatkan nilai 100, dan nilai 60 yang diraih oleh satu guru; nilai 60 merupakan nilai tertinggi ke-dua setelah 100. Tiga guru mendapatkan nilai 3 sebagai nilai terendah, diikuti nilai 5 yang diraih oleh 2 guru lainnya. ketidakseimbangan nilai antara dua hasil variabel tersebut menunjukan beberapa standar/nomor yang memiliki ketidak seimbangan. Hasil tersebut kemudian di lanjutkan dengan menginterview enam guru yang memiliki hasil nilai tes dan survey yang tidak seimbang tadi. Hasil dari interview menunjukan bahwa faktor-faktor yang mempengaruhi ketidak seimbangan nilai literasi assessmen dan implementasi penilaian formatif secara online dalam ranah writing mereka sebagian besar berasal dari ketersediaan fasilitas, motivasi siswa, dan akses sinyal internet, literasi digital, dan kelelahan guru.
Kata Kunci : Literasi Assessmen, Formatif Assessmen Online, Faktor yang mempengaruhi Literasi Assessmen dan Formatif Assessmen
Table of Contents
CHAPTER I ... 1
INTRODUCTION ... 1
1.1. Background ... 1
1.2. The objective of the study ... 21
1.3. Limitation of study ... 21
1.4. Research significance ... 23
CHAPTER II ... 27
THEORETICAL FRAMEWORK ... 27
2.1. Teachers’ Assessment Literacy ... 27
2.1.1. Assessment Literacy Definition ... 27
2.1.2. Teachers Assessment Literacy Principles ... 30
2.2. Online learning and Online Assessment ... 49
2.2.1. Online learning... 49
2.2.2. Online learning technology ... 50
2.3. Online assessment ... 56
2.3.1. Online Assessment Principles ... 57
2.4. Formative Assessment in the writing classroom ... 68
2.4.1. Formative Assessment Strategies in the writing classroom .... 69
2.5. The factors creating challenges and opportunities for teachers in implementing online learning in rural areas ... 78
2.5.1. Internet signal acess ... 78
2.5.2. Facilities provided ... 79
2.5.3. Low economic condition... 80
2.5.4. Teachers' and students' digital literacy... 80
2.5.5. Students low motivation ... 81
2.6. Mediating factors of assessment literacy ... 82
2.7. Previous related studies ... 83
CHAPTER III ... 88
RESEARCH METHODOLOGY... 88
3.1. Research Setting ... 88
3.2. Research Method ... 89
3.3. Participants ... 90
3.4. Research Instruments ... 92
3.4.1. Classroom Assessment Literacy Inventory (CALI) ... 93
3.4.2. Questionnaire ... 98
3.4.3. Interview ... 102
3.5. Data Collection Procedure ... 103
3.6. Data Analysis Technique ... 104
3.6.1. Quantitative Data ... 105
3.6.2. Qualitative Data ... 109
3.6.3. Trustworthiness ... 110
CHAPTER IV ... 115
RESULT AND DISCUSSION ... 115
4.1. Results ... 116
4.1.1. English teachers’ assessment literacy level ... 116
4.1.2. Online formative assessment implementation in writing ... 118
4.1.3. The relationship between the teachers’ assessment literacy levels and the implementation of online formative assessment scores in teaching writing ... 120
4.1.4. The factors contributed to the relationship between English teachers’ assessment literacy and their online formative assessment implementation. ... 123
4.2. The integration between quantitative and qualitative methods .... 127
4.3. Discussions ... 130
4.3.1. English teachers’ assessment literacy level ... 130
4.3.2. English teachers’ implementation of online formative assessment ... 137
4.3.3. The relationship between English teachers’ assessment literacy levels and the implementation of online formative assessment scores in teaching writing ... 143
4.3.4. Factors that contribute to the relationship between teachers’ assessment literacy levels and their online formative assessment implementation ... 144
C ... 161
HAPTER V ... 161
CONCLUSION AND SUGGESTIONS ... 161
5.1. Conclusion ... 161
5.2. Implication ... 163
5.2.1. Assessment research ... 164
5.2.2. Online writing assessment practice ... 165
5.2.3. Policy practice ... 166
5.3. Suggestion ... 167
5.4. Limitation ... 168
REFERENCES ... 170
List of Appendices ... 189
List of Table Table 1Technologies in Online Formative Assessment(Koc et al., 2015) .... 51
Table 2 Basic Questions to Do Self-assessment (William & Leahy, 2015) .. 62
Table 3Success Criteria and The Goals from Lee (2017)... 70
Table 4. Basic Metacognitive Questions in Story Writing Form... 75
Table 5. The Purposive Sampling of the Research ... 92
Table 6. The Classroom Assessment Literacy Inventory Framework by (Council, 1990) ... 93
Table 7. Online Formative Assessment in Writing Practice Questionnaire Framework ... 98
Table 8 The Scale of Teachers' Attitude to Online Formative Assessment Implementation ... 99
Table 9. Reliability Test Criteria ... 101
Table 10. English teachers' assessment literacy level ... 116
Table 11.Online Formative Assessment total score in teaching writing .... 118
Table 12. The score pairs of assessment literacy and online formative assessment ... 120
Table 12. The relationship between English teacher's assessment literacy level and online formative assessment implementation... 121
Table 13. The Factors of the Gap between Assessment Literacy and Formative Assessment Implementation ... 124
Table 14. The Frequencies of the Mediated Factors ... 126 List of Figures
Figure 1. Dialogue between Students Writer and Peer Assessor adapted from Lee (2017) ... 73 Figure 2. Students learning log of writing (Lee, 2017) ... 77 Figure 3. Explanatory Sequential Mixed Method Design Process (Cresswell
& Clark, 2018) ... 90 Figure 4. Validity Test of CALI Result... 97
1 CHAPTER I INTRODUCTION 1.1.Background
The educational environment has dealt with many new terms from time to time because education has exponentially grown in the past decades (Kay & Li, 2019). One term that was recently attached is
“Online learning.” It is an activity of teaching and learning delivered through a digital device, and its purpose is mainly to assist learning (Ferri et al., 2020). Although online learning has been utilized in Indonesia for a long time, its utilization is not equally done all over the regions. Therefore, when the school's closure happened due to the Covid-19 emergency crisis, online learning became an unavoidable term for all students and teachers worldwide (Tria, 2020). It became the only alternative suggested by the government to keep education continuity. As well as Dhawan (2020) asserted that the sudden shift from face-to-face to online learning is done to keep the learning continuity rather than to maximize the quality intentionally.
Consequently, it creates a new massive utilization of online learning techniques based on their different readiness. It implies that the sudden changes in teaching mode could create other problems, such as readiness. It is where the teachers formed three main readiness
issues in the teaching preparation, online learning activities, and online learning assessment (Ardiasih, Lidwina et al., 2020).
Teachers' pedagogical competencies because the teachers had limited preparation and training regarding teaching with technologies (Avillanova, 2019). These technical difficulties consume much online learning time and limit interaction between teachers and students (Ingemarsson, 2021). Lack of familiarity with utilizing digital platforms caused the teachers' feedback to be lacking either(Nugroho.
et al., 2021). The teachers limited control controlling their students' engagement in online learning (Sumardi & Nugrahani, 2021). The disparity of quota substitution policy in the outlined regions of urban areas (Sajida & Ranjani, 2020).
In Indonesia, the government has enacted the policy of learning from home for K-12 grades, which is online, and designing the emergency curriculum ultimately based on the decree of the Ministry of Education and Culture of the Republic of Indonesia number 719/P/2020. The circular letter 15, 2020, regarding learning from home in an emergency due to the Coronavirus disease outbreak. It generally explained the approach for the teachers, students, and parents to implement online learning cooperatively from choosing the course and learning model until the assessment practice.
Although the government has provided a guideline for online learning from home, many teachers still do not maximally apply it correctly. Ardiasih and Yundayani (2021) found that teachers' readiness affected the activity of online teaching and learning effectiveness. Additionally, the unstable connections and limited direct instructions caused the ineffectiveness of online learning (Nartiningrum & Nugroho, 2020). Furthermore, Bakerson et al. (2015) explained some struggles that often happen in online learning, such as technology limitations, since students learn the technology first rather than retain the main content of the material. Besides, the students would get frustrated if they lost the material and focus from the beginning — less on personal connections feeling from the students (Ingemarsson, 2021). Furthermore, the limitation of reliability and validity issues include un-proctored exam cheating, using outside materials, not being in current learning (replaced by others), plagiarism, and misinterpretation of feedback (Sumardi & Nugrahani, 2021). Assessment design could limit online learning reliability and validity challenges (Nguyen et al., 2020).
The progress of students' learning is measured through an assessment because the assessment and learning are intertwined and mutually dependent (Alderson et al., 2017). Moreover, Stiggins (2005)
believed that assessment provides evidence of students, teachers, and the system's success because it shows students' standard mastery that could help teachers decide their meaningful instructions. Besides, Kristiyanti (2021) defined assessment as a method in the learning- teaching process where the teachers collect their students’ ability level, learning outcomes, strengths, and needs. So that after gaining them, the teacher could fit their teaching with the students’ current conditions to achieve greater levels of learning.
Therefore, teachers should set the appropriate teaching assessment technique as well as Husna (2020) argued that assessment design needs to be carefully planned. It could be a helpful tool for teachers as a grading system, yet it could be harmful if they did not design it properly (Shepard, 2000). Boud (1995) stated that students can escape bad teaching yet avoid the assessment. In other words, teachers should consider certain aspects that can minimize the students’ pressure during the assessment process, in line with Xie and Lei (2019). They found that formative assessment provided the students chance to face the assessment process with less pressure and anxiety. Defianty (2018) defined formative assessment as an evaluation of students’ competencies development through feedback
and students’ involvement and teaching adjustment, which is done continually.
Furthermore, Defianty et al. (2021) stated that formative assessment could provide valuable evidence of learning output and become a barometer of daily improvement. Farrag (2020) explained that formative assessment is done at the end of the course and in the middle or based on teachers' and students' needs. Besides, Rahim (2020) asserted that teacher engagement with formative assessment is crucial because it could facilitate teachers' professional development and student learning during Covid-19 learning. As well as the research of Defianty et al. (2021) found that students expected their teachers to implement formative assessment during emergency remote teaching because it could accommodate students' needs more thoroughly.
Furthermore, research showed that students' and teachers' engagement with formative assessment could provide the students with writing development from what and how they learn (Krishnan, 2019). Online writing platforms for student collaboration are now ubiquitous (Krishnan et al., 2019). Online collaborative writing through google docs made students participate equally during online learning (Fan & Woodrich, 2017). Besides, Chu et al. (2017) found
that Wiki created higher output of writing quality. Jong and Kim Hua (2021) also found that students improved their writing significantly by using Padlet (an online writing platform).
An online writing platform would automatically make the teachers utilize it in the grading system, too, precisely in the assessment process (Koc et al., 2015). Previous research has been done in formative assessment, especially in writing. The study revealed that formative assessment could improve students' writing and provide teachers with diagnostic feedback (Lee, 2017). Formative assessment also makes teachers easier in designing writing instruction continuously (Xie and Lei, 2019). Nurhayati (2020) found that formative assessment helps students improve writing in grammatical and language use aspects. Besides, a positive attitude has also appeared because students could evaluate their performance from the feedback in writing (Fan & Woodrich, 2017).
Krishnan (2020) described that formative assessment in writing would make the students engage with the learning process due to their direct involvement in the assessment process. Besides, in language teaching, formative assessment can inform the teacher of the evidence of the student's state of language proficiency. In Indonesian online
learning today, Kemdikbud (2020) has emphasized that teachers should use formative assessment more frequently in online learning.
In other words, teachers should deal with formative assessment as the central basis for classroom writing assessment (Lee, 2017). It is because formative assessment could give the students guidelines for criteria brainstorming, proposal and language into assessment rubrics and provide self-assessment and writing goals formulation by themselves (Zou et al., 2021).
Despite the various online writing platforms available and their ease in the assessment process, online learning during the pandemic still creates a different state of preparation and readiness (Ingemarsson, 2021). Hodges et al. (2020) stated that online learning in emergency conditions could emerge various outputs based on the institutions' preparation and readiness. The research found that inequality in technology readiness still happened in Indonesia. It is because of some reasons, such as the teachers' and students' inability to access online platforms because of limited knowledge about it (Lie et al., 2020), teachers’ lack of giving appropriate and adequate feedback (Nugroho et al., 2021), and less support from the parents to the online learning (Yulianti et al., 2019). This inequality could create
a gap between online learning continuity and teachers, students, and parents' readiness to implement online learning.
Many studies asserted that online learning could cause disparity among several remote areas in Indonesia because of the limitation of technology and IT connectivity. The research of Gamage et al (2020) discovered that online learning in remote regions emerges the potential academic disintegration due to the limited teachers’
control of students’ engagement. Furthermore, Lie et al. (2020). They found that the teachers’ engagement in online teaching in rural regions of Indonesia is still rudimentary up to an intermediate stage.
Additionally, Octaberlina et al. (2020) asserted that online learning in remote areas was hard to implement through online live conference video. However, it needs to be implemented because of its significant benefit in an online environment. Some research reported that Java province has effective and efficient online learning due to the stable network, connection (Harefa & Sihombing, 2021), and infrastructure (Amelia & Istianah, 2021).
However, the rural areas in Java province are also undergoing a disparity in online learning effectiveness (Mukarromah & Wijayanti, 2021). The rural area of Indonesia is described in Law number 7 of
1991 about the income tax. It defined rural areas as places with much economic potential in the form of natural resources in agriculture, forestry, mining, tourism, and industry. However, the economy, infrastructure, and facilities are still limited. So both government and investors must build their infrastructure and facilities, such as roads, ports, electricity, telecommunications, water, employee housing, health services, schools, places of worship, markets, and others, to convert the available economic potential into real financial strength (Pasutri, 2020).
It was recorded that central Java province has several regencies categorized as rural areas with different effectiveness in implementing online learning (Badan Pusat Statistik, 2020b). Aprilisa and Kartowagiran (2022) found that teachers in the rural area of Java province have a high interest in free online digital platforms, although many of them still learn about how to use other platforms except WhatsApp. Meanwhile, slow internet connection on Java island still happened (Cakrawati, 2017). Lack of parental involvement in supporting the children to engage in online class also cause the challenge of teaching English online (Yulianti et al., 2019). Different regions and its policy can cause these different challenges of online learning implementation. Additionally, The disparity in
socioeconomic levels between urban and rural areas is high (World Bank, 2021).
There are pretty many rural areas numbers in several big provinces of Indonesia. It is based on Law number 120 in 2020, which routinely observes, classifies, and categorizes urban and rural areas.
For instance, Sumatera has 5258 rural areas, Kalimantan has 5777 rural areas, Sulawesi has 7548 regions, and Java has 8133 rural areas (Badan Pusat Statistik, 2020a). From the classification numbers of rural areas, Java island still has many rural areas. However, Java is believed to have many possibilities to upgrade its area categories to be more developed to become urban because it is located on the same island as Indonesia’s capital city (Kristiyanti, 2021; Rahiem, 2020). Central Java reached a high number of rural areas in 2020. It has 3236 areas (Badan Pusat Statistik, 2020a).
Cilacap is one of the largest regencies of central Java province and has 24 rural districts. One district is reported to have a high under- age number of workers, Kawunganten (BPS Kabupaten Cilacap, 2021). Kawunganten is a village with a highly increased number of productive-age kids (BPS Kabupaten Cilacap, 2021). It implied that many children become students and workers simultaneously, which
could distract the students’ focus on learning. Cilacap should provide enough space for productive-age kids to learn properly and focus. In reality, It has only four state junior high schools with 2786 total students and 662 teachers that create 1: 46-classroom ratio.
Meanwhile, ideally, law number 14 of the year 2005 regulated that the ratio between students and teachers should be 20:1. Besides, It has no state madrasah but five private madrasahs with 70 teachers and 1038 students. In other words, the ratio between the number of teachers and students and the ratio of the schools' availability would possibly be unideal (Idris et al., 2019). As it is stated too by Rahiem (2020) stated that rural areas tend to have a limited number of schools.
Moreover. the number of teachers and students in junior high schools under the ministry of education and ministry of religious affair remain unbalanced in the Kawunganten district (BPS Kabupaten Cilacap, 2021). This disparity ratio between teachers and students has become an unavoidable challenge for remote and rural areas (Hatip &
Setiawan, 2022; Idris et al., 2019).
Today Indonesia is experiencing a shift in teaching and learning environment from face-to-face to online because of the covid- 19 effect. Therefore, all educational institutions in Indonesia should
implement the online teaching and learning system. Indeed, the educational institution and schools in Cilacap, with its current conditions of the unideal ratio between students and teachers, schools and teachers’ numbers, cannot avoid it either.
As a result, many studies found that some obstacles hinder online learning. These are teachers' professional development (Avillanova, 2019). It is supported by Ardiasih and Yundayani (2021), who stated that teachers in Madrasah in Java were still not fully ready to implement online teaching. Additionally, the unemployment state and poverty of Java province are increasing in 2020 (Badan Pusat Statistik, 2020b). It potentially affects the online learning continuity because of parents’ less support (Pratama & Firmansyah, 2021). It has happened since online learning needs additional costs for specific devices and quotas. Therefore, the student's financial background would also be affected (Rahiem, 2020; Widayanti & Suarnajaya, 2021). Consequently, the student's willingness to learn online emerged differently in rural areas (Prastikawati et al., 2020). In other words, despite connection stability, students' socio-economic status, and parents' support, teachers’ readiness could be the main reasons behind online learning challenges.
Sintasari et al. (2020) discovered that teachers in rural Cilacap regencies try to implement online learning maximally despite many struggles. Many teachers' professional development training regarding the online platform tools to aid the online teaching process has been done there (Avillanova, 2019). It described that teacher professional development in the rural areas of Cilacap has been done to maximize online learning implementation.
Despite much professional development training to prepare teachers for online teaching available, implementing those training, especially in assessment, created different output stages among the teachers. Many studies have been done on the utilization of online educational platforms such as Google docs (Fan & Woodrich, 2017), Quizizz (Basuki & Hidayati, 2019), and Google Classroom (Wahidiyati, 2020). These platforms are helpful for teachers, but the assessment process that should be easier for a teacher could be more complicated when the teachers are not ready and literate in the assessment process in online mode (Zou et al., 2021). In other words, there is a dissonance between the available platforms for teaching and its practice, especially in online environments (Yan et al., 2018a).
Furthermore, the assessment process became a barrier to online learning since the government did not set specific guidelines, especially for the assessment process. The government sets only the regulation for teachers to assess students by some considerations which focus on students’ needs, weaknesses, and competence based on their exact condition. It is in line with McNeil (1977), stated that the evaluation process must consist of three crucial pieces of information:
placement, mastery, and diagnosis. The student's level of learning could reflect the placement term, where they can be challenging but not frustrating. Besides, the mastery term will describe the students’
prior knowledge and abilities to go to the next level—the last diagnosis term, which reflects the students’ current difficulties.
In the online learning context during the pandemic in Indonesia, Kemdikbud (2020) suggested that diagnostic assessment could be implemented in the formative assessment form during online learning. Besides, Fautley and Savage (2010) declared that diagnostic assessment becomes the subsumed formative assessment. In other words, the government has suggested that teachers implement formative assessments based on the students’ current competencies, weaknesses, and strengths. At the same time, measurable and directed references or guideline is needed to implement the education program
properly (Utami et al., 2021). However, the government did not explicitly prepare online learning guidelines, including the assessment part (Kristiyanti, 2021). In other words, the current regulation of online learning does not provide teachers with enough guidelines. The teachers become the main critical decision-maker in the assessment process, including the designer of the feedback production in the online environment (Thumvichit, 2021).
Sufyan et al. (2020) discovered that receiving less feedback in the assessment process became a challenge in online learning assessments. Additionally, Lie et al. (2020) found that teachers still struggle to enhance the quality of learning based on the learning activity and assessment. Besides, the research of Arrafii (2020) found that teachers’ conception, understanding, and practice of formative assessment innovation practice would be based on a clear vision and teachers’ assessment literacy, which will also affect the design, implementation, and effectiveness. In other words, teachers’
assessment literacy has a crucial role in their formative assessment practice.
Teacher assessment literacy is a teacher's understanding of assessment principles and the required skill of students’ to assess in
line with Stiggins (1999). He initially defined assessment literacy as a teacher’s assessment concept regarding the learner's skill understanding. So, the assessment literate teacher would know what, how and what challenge would be faced in the assessment process (Stiggins, 2005).
The researchers and practitioners conceptualized the meaning of assessment literacy as the method used in arranging the evidence of students’ performance in the form of reliable data, how to use assessment to support students learning and how to deliver the result of assessment effectively and accurately (Zulaiha et al., 2020).
Some studies found that teachers’ assessment literacy could affect their practice of assessment in the classroom ( Thumvichit, 2021; Zou et al., 2021; Zulaiha et al., 2020). These researches revealed that teachers’ assessment literacy influenced the assessment practice, assisting students in performing their output maximally. However, teachers' regional infrastructures and different assessment literacy levels have become a gap, especially in the online formative assessment, whether in Indonesia's rural or remote regions.
Poedjiastutie et al. (2021) found that teachers in remote and rural areas face several challenges in teaching English. There was an
economic situation, community beliefs, the administering national curriculum, the lack of teachers’ knowledge of assessment (Nugroho et al., 2021), and teacher engagement (Zou et al., 2021) conception and understanding of formative assessment (Arrafii, Mohammad, 2020).
As well as Ivone et al. (2020) found that remote and rural areas endured extra effort in implementing online learning. Their lack of infrastructure, teachers' knowledge of ICT, and teachers’ transferring their pedagogical knowledge to an online mode, such as lesson planning and assessment, have become the reasons. It can be said that teachers’ assessment literacy in rural areas remains lacking so is the practice.
Zamili (2021) discovered that the teachers’ pedagogical competence, including the teachers' assessment literacy in Java province, is still at the medium to low level. Yamin and Syahrir (2020) found that teachers in Indonesia were still unaware and had problems designing assessment instruments that focus on students’ literacy skills, while literacy now becomes urgent. It is supported by the Merdeka Belajar program recorded that Indonesian students still have low literacy and numeracy skills. The main focus of secondary students' national assessment is literacy and numeracy (Kementerian pendidikan dan kebudayaan, 2020).
Additionally, Putri et al. (2019) discovered that literacy activity such as writing is essential for high school students to prepare for the academic world at the university level. Therefore, in Indonesia's current curriculum for junior and senior high school language subjects, writing becomes students' last and most complex learning skill (Pajarwati et al., 2021). Consequently, writing in a language subject needs more attention and preparation.
Therefore, this study attempts to investigate the teachers' assessment literacy and their practice of online formative assessment in writing since the research of Krishnan (2020), and the theory of Lee (2017) have been used as patent guidelines for many studies such as the studies of (Shim, 2009a; Zou et al., 2021; Zulaiha et al., 2020).
Although previous studies about formative assessment implementations have been done to enlighten this research, the researcher found that they were not linking it to the assessment literacy aspect. At the same time, the research of Anam and Putri (2021), Arrafii (2020), and Farmasari (2020) have discovered that teachers’
assessment literacy is linked with and becomes a sub-theme of formative assessment implementation (Council, 1990). Therefore, the assessment literacy concept should be involved as the new concept of
this research, especially the literacy regarding collaborative writing in the online environment.
Furthermore, the researcher found that those previous studies’
results did not answer today’s current condition, where face-to-face learning has been shifted to the online environment. Therefore, this research was automatically done to discover and describe further the assessment literacy of the teachers and the implementation of formative assessment in teaching writing in online conditions.
Another new term discovered in this research was the research site. The previous research regarding only formative assessment has been conducted in the West Java region (Defianty, 2018) and the East Java region (Arrafii, Mohammad, 2020). The results found that the formative assessment implementation was based on the teachers’
assessment concept. Meanwhile, the research of Octaberlina et al. ( 2020), Rahiem (2020), and Kristiyanti (2021) found that the implementation of online formative assessment in a remote area (outside of Java) was hindered by external factors such as infrastructure and internet connection access. Therefore, these studies evoked the researcher's curiosity about the challenges in implementing online formative assessment in different regions, which different
factors could cause. Therefore, this research was done in a rural area of Central Java, Cilacap regency. Precisely in Kawunganten district, where the previous studies were done in East and West.
Furthermore, this current research was also conducted in an online environment to see whether the assessment literacy concept in Java island was linked to the implementation of online formative assessment or not.
Lastly, this research used a different method from the previous studies that used either qualitative or quantitative methods (Defianty, 2018; Krishnan et al., 2021; Zou et al., 2021). It aimed to gain both general descriptions of assessment literacy and online formative assessment implementation through the followed-up activity in a qualitative way. Therefore, this study combined the data collection process to answer the problems in the background of the study with many possible pieces of evidence.
Based on the above description and explanation of the problem until the possible data collection methods, this research formulated the research questions below. (1) What is the assessment literacy scores of junior high schools English teachers in the Kawunganten district? (2) What is the scores of online formative assessment implementation of
English teachers in Kawunganten District? (3) Is there any relationship between English teachers’ assessment literacy level and their implementation of online formative assessment in teaching writing ? (4) What factors contributed to the relationship between the assessment literacy and the implementation of online formative assessment scores in teaching writing?
1.2.The objective of the study
1. To know the assessment literacy of junior high school English teachers’ association of Kawunganten district.
2. To know English teachers' practice of online formative assessment.
3. To know and identify the relationship between the teachers’
assessment literacy and their implementation of online formative assessment in teaching writing
4. To know some factors that contributed to the relationship between the assessment literacy level and the implementation of online formative assessment in teaching writing
1.3.Limitation of study
This research focuses on the online formative assessment in rural areas of the Cilacap regency to avoid overwhelming and unfocusing data obtained. This research would focus on the schools incorporated into a group of English Teachers Association of
Kawunganten district. That is why the probability sampling would be suitable for gaining the teachers in the English teachers’ association of Kawunganten area assessment literacy and the practice of online formative assessment (Teddlie & Tashakkori, 2009). Secondly, the teachers’ assessment literacy is expected to be aligned with the teachers’ online formative writing assessment practice. Therefore, the researcher would focus on the relationship between assessment literacy and practice of online formative assessment implementation. This research adapted the research of (Shim, 2009a), which has been updated by (Zulaiha et al., 2020), with the explanatory sequential design. Other designs would not be enough to cover this study because this study mainly wants to investigate the factors that hindered the relationship between assessment literacy levels and online formative assessment practice more in-depth.
Furthermore, since this current research focuses on the online environment and writing skills, the fundamental theories guideline is updated. Therefore, the theories of Lee (2017) regarding formative assessment in writing skills collaborated with the online assessment theories by Palloff and Pratt (2009). Additionally, the documentation of assessment design artifacts will be taken as the triangulation process of this study.
1.4.Research significance
Based on this research objectives, the researcher has considered some opportunities coming from this research as the research significance.
This research significant classified into three main parts.
1. Theoretically
The previous studies have explored many theories regarding teachers' assessment literacy and formative assessment. This research employed those theories and created new theoretical contributions by extending the views of the assessment literacy aspect (Perry, 2013) and the implementation of formative assessment in writing (Lee, 2017; William & Leahy, 2015) in the current condition of online environment (Koc et al., 2015; Palloff & Pratt, 2009).
Furthermore, this research contributed to pedagogical knowledge as an essential input to English teachers because this research described the assessment literacy standard that became the standard of being a professional teacher (Arrafii, Mohammad, 2020; Defianty, 2018; Zamili et al., 2020).
Therefore, English teachers knew they were expected to own that professional assessment standard (Council, 1990).
Besides, this research added the extended theories from the
previous studies, which could be used by the next researcher interested in this field.
2. Practically
The guideline of formative assessment practice used in this research automatically lead the teachers to understand the teachers’ professionalism regarding assessment literacy and the implementation of online formative assessment that is expected and regulated by the government. The research of Perry (2013) declared that research which provided the assessment literacy standard was still rare, so it is necessary to do research that also used the assessment literacy benchmark to be a teachers’ reflection guideline in teaching.
This research enlightened the teachers regarding implementing online formative assessment, especially in teaching writing online since this research clearly described the implementation procedure from the theories (Lee, 2017;
Shim, 2009a; William & Leahy, 2015). Automatically writing assessment guidelines in an online environment would benefit teachers today. Therefore, this research used the online assessment environment as a grand theory (Koc et al., 2015;
Palloff & Pratt, 2009). Besides, this research provided the
teachers with clear guidelines and benchmarks for assessment literacy because its research instrument has been adapted and modified to measure the assessment literacy level and online formative assessment in teaching writing.
3. Society
This research not only provides the new concept of online formative assessment in teaching writing generally but also offers some possible solutions and the current precise needs of English junior high school teachers in the specific rural district. Exactly in Cilacap regency since the research finding revealed the factors that mediated the teachers’
assessment literacy levels and their implementation of formative assessment in the online environment. This significance worked as a contra evidence of Rahiem's research (2020) because she discovered that technology implementation on Java island tends to be more effective than outside of Java island.
This research portrayed not only the teachers’
challenges of online formative assessment implementation but also the teachers’ supporting aspects from the factors that created the opportunities for the teachers to assess students in
the online environment. It was done in rural Cilacap regency, which still faces many challenges in online learning even though they are on Java island.
Furthermore, the government could consider this research in future policies or regulations regarding online teaching and learning during the pandemic. The regional government could also use this research to better understand the teachers' needs to improve their literacy assessment level.
Following these studies (Kristiyanti, 2021; Sajida & Ranjani, 2020; Utami et al., 2021), whose results created needs that should be considered in the future.
Therefore, the government would minimize the disparity in implementing online learning between the urban and rural areas of Java island by stating the regulation of online assessment based on the current condition of a rural district in the Cilacap area.
27 CHAPTER II
THEORETICAL FRAMEWORK
2.1.Teachers’ Assessment Literacy 2.1.1. Assessment Literacy Definition
Assessment literacy is a body of information about what should be assessed and how it should be done (Arrafii, Mohammad, 2020).
Besides, assessment literacy is described as "an individual's awareness of the core assessment concepts and techniques that are likely to affect educational decisions” (Popham, 2011). Lee (2017) asserted that assessment literacy is a form of assessment for learning or formative assessment practice. It means that teachers with assessment literacy will practice formative assessment in their teaching activities.
Furthermore, a teacher’s assessment literacy relates to their proven comprehension of the principles behind selecting and designing tasks, grading student work, and interpreting and applying assessment data to assist student learning (Koh et al., 2018). Stiggins (1991) defined assessment literacy as teachers' knowledge of underlying principles and the skills needed to evaluate student learning. Furthermore, assessment-literate people
will understand what to assess, how to assess, what challenges might arise, and how to solve them.
Additionally, Zulaiha and Mulyono (2020) explained assessment literacy as the teachers’ knowledge about collecting reliable data on students’ performance and the assessment result usage to support students’ learning and communicate it comprehensively and accurately. Yazdani and Ghasedi (2021) stated that assessment literacy refers to a teacher's knowledge of evaluation principles, techniques, criteria, tools, and tactics. In other words, assessment literacy is the teachers’ knowledge of assessment principles, strategies, and assessment results followed up activities to make the learning move forward.
The research of Dwiyanti and Suwastini (2021) found that formative assessment has two essential strategies. There are feedback and diagnostic features because this assessment aims to explore and improve the learning to move forward. It is in line with Tinoca and Oliveira (2013), who conclude several things that have become the feature of formative assessment. Those include a cognitive process that is regulated by motivational beliefs.
Moreover, formative assessment and feedback should be powerful
strategies to empower the students’ self-regulations during learning.
Furthermore, the basic concept of teachers’ assessment literacy and engagement in formative assessment is shaped by several factors, such as teachers' contextual factors and beliefs (Krishnan et al., 2021). In other words, the national regulations and the teachers' beliefs in their strategies would be the main contributions to the formative assessment implementation, which strongly relates to assessment literacy. Arrafii (2020) research found that teachers will follow the national curriculum regulation. However, it does not mean that the teacher omitted their old strategies for assessing students. It proved that the teachers’ strategies in assessing students were from their beliefs, which also reflected their assessment literacy level.
Above all, despite teachers' beliefs about assessment strategies, assessment literacy can be defined as a benchmark of the possibilities of their formative assessment implementation, which the government announced to be used as a diagnostic assessment during online learning in Indonesia. Being used as a benchmark of teachers’ professionalism, especially in assessment competence, makes the teachers aware of the term literacy. However, online
assessment and assessment literacy research remain lacking (Kristiyanti, 2021). Therefore, the researcher conducted this research to delve into the term assessment literacy and the implementation of formative assessment in teaching writing online in this chapter.
2.1.2. Teachers Assessment Literacy Principles
After reviewing the term assessment literacy, the researcher first described an essential part of it. That is the principle of assessment literacy. Perry (2013) defined principle as a basis of national policy and regulation that professionals legalize and expert organizations that are used as a foundation of the implementation of the assessment. Besides, assessment principles are related to the teachers' professional expertise since their role is to become the basis of any instructional decision and evaluation of learning (Aria et al., 2021). In other words, the assessment principles have a significant role in teachers’ assessment literacy because they became the basic and root of ideal instructions and assessment practices. The more teachers understand the principle, the higher their assessment literacy level (Anam & Putri, 2021;
Aria et al., 2021).
A recent study by Arrafii (2020) stated that those concepts of language assessment literacies were summarized by the National Council on Measurement in Education (NCME), the National Education Association (NEA), and the American Federation of Teachers (AFT) in 1990. The criteria mentioned over there were used to describe the evidence of language teachers’ assessment literacy. The description is described below.
2.1.2.1.Teachers are selecting appropriate assessment methodologies for instructional decisions.
The assessment-literate teacher should know the importance of selections (Council, 1990). Teachers would be familiar with many alternatives of assessment with its strength and weakness. So, their choice will be appropriate with the following up instruction since the instruction decision of the teacher would be based on the teacher’s choice. So, instructional decision has become essential things in assessment. The instructional decision means the teachers can connect their chosen assessment method and learning objectives (Arrafii, Mohammad, 2020). It can be said that when the teacher does not have an instructional decision, it will affect the learning objective too, that consequently, the students will not get the learning purpose or their new skills(Aria et al., 2021).
Therefore, understanding how to select the assessment method based on the learning objective is essential.
Selecting the assessment method is where the teachers could choose the appropriate assessment method for the instructional decision later. Furthermore, Stiggins (1999) added that in this stage, the teacher would automatically use the competency of connecting assessment to a clear purpose. The teacher could select the assessment methods following the purpose of the assessment.
Therefore, teachers would also conversantly know many potential options for assessment (Popham, 2009). In other words, the teachers here can choose the assessment based on its purpose in many possible and potential ways. Lee (2017) believed that assessment-literate teachers could design connected learning instructions and assessment types. So, the teacher will focus on the success criteria shared with their students.
Additionally, Zulaiha et al. (2020) described that assessment- literate teachers understand how to collect reliable data about student performance, use assessment to assist student learning, and effectively and accurately communicate assessment results. It can be said that teachers in this stage could consider and adjust their assessment methods and teaching instructions based on their
students’ needs and lacks. The assessment-literate teacher ideally selected the methods and instructions from the previous readiness checking that was done in the very first meeting. It is in line with the formative assessment principle that mainly suggests teachers do a diagnostic assessment before selecting the assessment method since it would give the teachers deeper information about students’ needs in the next meeting.
2.1.2.2.Teachers are developing appropriate assessment mechanisms for instructional decisions.
Previously, an instructional decision was considered the important part that the assessment literate teachers should own because teachers can appropriately adjust it based on their purpose of learning regardless of the student's culture and social classes. In line with Popham (2009), literate teachers could be more straightforward in deciding on both classroom assessments; the daily assessment made by teachers who know how the students engaged with the instructions and the accountability assessment, a traditional test used in accountability testing.
Mellati and Khademi (2018) stated that teachers’
instructional decisions would control the objective learning focus so that the teachers would make the instruction based on the grade
and realistically did not make the students get a lower objective or too high objectives. They would accept their adjusted instructions.
After knowing the instructions' purpose and fitting them with the appropriate assessment, the teacher could contextualize the assessment or appropriate instructions. Stiggins (1999) support it, stating that literate teachers should be competent in applying proper assessment methods. It is supported by Fautley and Savage (2010) suggested that teachers develop the assessment in line with the instruction. As well as Lee (2017) emphasized that assessment- literate teachers should be able to link the learning objective, success criteria, and instruction because they cannot be separated.
They are formatively unitary in assessment.
To reach and practice this instructional decision, teachers need to develop it through several mechanisms. William and Leahy (2015) suggested that the teacher could use peer and self- assessment when the learning decision is to raise students’
awareness about discovery learning because they would find others or themselves lack strength during learning. Next, they suggested questioning techniques and waiting time to invite the students’ understanding of the learning objectives and instructions. Announcing and showing the ideal model of
students’ writing work to the students will help them have a clear picture of the ideal writing product. Additionally, letting students’
do peer feedback by training them to give feedback properly (Lee, 2017). As well as Fautley and Savage (2010) also stated that assessment development could happen in the beginning, middle, or end of the learning process as the teachers reflect on their linkage of the learning objective, instruction, and students’ output.
2.1.2.3.Teachers use external and teacher-created assessment methods to administer, rate, and interpret the results.
2.1.2.3.1. Using external and teacher-created assessment methods
It is not enough for teachers to select and develop the assessment method since they must also be administered themselves. The assessment-literate teachers should be able to administer and implement it in their classrooms (Council, 1990). Further, the external and created assessment methods are essential for the teachers. It is where the teachers adjust their students’ lacks and needs according to their ability to create the assessment process consisting of collaborative sources that come from their created assessment and external assessment method. The
teachers could also adjust the external methods, such as using a state book and developing them (Arrafii, Mohammad, 2020). Therefore, the teachers' ability to select and combine the assessment method would become the teachers’ assessment literacy (Aria et al., 2021). The teachers could run the assessment process made by them or externally not made by them comprehensively. This stage has a relation with the selection of the assessment method process because, in this stage, teachers could also select and combine their own method from other sources outside so they can collaborate and justify it based on their students’ needs.
2.1.2.3.2. Administering assessment methods
After using external and made assessment methods, the assessment-literate teachers should administer them in the teaching activities. As well as Federation and Council (1990) standardize the literate assessment teachers category. It described that the assessment-literate teachers should be able to administer the assessment method that has been
collaborated from the external and made by the teachers.
It is where the teachers can administer the standardized test based on the guidance or blueprint that has been analyzed comprehensively by them so that the teachers would do the blueprint of the test that is collaborated by external and internal methods of assessment comprehensively (Yazdani & Ghasedi, 2021).
In this stage, teachers have selected and collaborated on their assessment method based on their students’ needs. What they should do further is arrange the mechanism of it thoroughly. In administering the test, the teacher should be able to create valid and reliable tests comprehensively with correct procedures.
In making test items, teachers must create a blueprint divided from the learning objectives they have distributed. Consequently, the test item they administer would be more valid and reliable.
2.1.2.3.3. Rating the assessment methods
After administering the collaborative assessment method, the assessment literate teachers would know how to follow up the assessment method through several steps. Furthermore, the reflection of the assessment-literate teachers Federation and Council (1990) stated that assessment-literate teachers should be able to interpret and analyze the students’
assessment results. Lee (2017) explained several steps of using both summative and formative assessments. It is by using an error log that summatively records the students' accuracy of targeted language (in writing) and a learning log that records the students' self-reflection independently.
Both will be useful when teachers cope with the administration to rate and interpret the students learning objectives summatively. Besides, it aimed to share the students' abilities diagnostically. In other words, the teacher with assessment literacy would be able to collaborate on the assessment methods based on their role as teachers in the institution and facilitators in students’ learning. In this stage, teachers could consider
and rate the assessment method that they used to be evaluated and upgraded for future instructions.
2.1.2.3.4. Interpreting the result of the assessment
Teachers could interpret the commonly reported scores: percentile ranks, percentile hand scores, standard scores, and grade equivalents. They conceptualized the summary indexes commonly reported with assessment results: measures of central tendency, dispersion, relationships, reliability, and measurement errors (Mohamadi, 2018). Teachers could apply these concepts of score and summary indexes to enhance their use of the assessments they develop.
They analyzed the assessment results to identify pupils’
strengths and errors. If they get inconsistent results, they will seek other explanations for the discrepancy or other data to resolve the uncertainty before arriving at a result of the assessment (Council, 1990). After setting the test minimum and maximum score, the teacher should be able to infer their students' scores accurately to make the next move of teaching instruction forward.
2.1.2.4.Teachers use assessment data to judge individual students, teaching plans, curriculum development, and school improvement.
2.1.2.4.1. Using assessment data to make judgments about individual students
When the teachers have various valid and reliable data on their students’ assessment, they will automatically enable themselves to get improvements and changes in improving their instruction so that the students' outcomes would be increased too (The Center Standard & Assessment Implementation, 2018).
Therefore, it is believed that teachers should first read their students’ lacks and needs to record their
achievements individually or non-
individually(grouped). As well as Stiggins (2005) stated that the assessment-literate teacher could clarify the expectations/success criteria attainability which can be used as the improvement criteria to improve the student's achievement. So that teachers could determine the student's current ability and the expected criteria accurately.
2.1.2.4.2. Using assessment data to make teaching plans After interpreting the students' assessment results, the teacher could know the student's needs and further improvement steps. So, teachers need diagnostic assessment (Fautley & Savage, 2010a).
Additionally, Mellati and Khademi (2018) discovered that assessment-literate teachers acted appropriately in deciding on the class, such as evaluating their instruction after seeing the test results to make better instruction in the meantime and in the future. Those judgments should go to the next stage.
The teacher's competency in connecting the assessment and its apparent purpose will be a literate teacher’s basis in deciding on the next assessment method (Stiggins, 1999). Rahman et al. (2021) found that the improvement of assessment data has become the basic idea as the benchmark of teaching planning and syllabus development that could be done more than three times; at the beginning, implementation, and end of the teaching process. In other words, assessment- literate teachers should be prepared for every result of
their students' scores and directly execute their preparation to maintain and develop their instructions based on the student's diagnostic results too.
2.1.2.4.3. Using assessment data to make curriculum development
As the assessment data employed in the teaching plans, it would also influence the curriculum development since it would provide several benefits, such as reviewing, considering, and instituting changes, assisting teachers in developing their classroom’s curriculum, and monitoring the effort of change (Siler, 2009).
Additionally, Brown and Sally (2005) emphasized that assessments and tests are essential to a successful curriculum because they become teaching and learning process partners and are used as students, teachers, and school improvement barometers. They played as a student's motivation and a school closure periodic tool. Therefore, the assessment data would help review, change, adjust, and evaluate the curriculum development to avoid the gap between the
curriculum expectations and the actual assessment data (Arrafii, Mohammad, 2020).
2.1.2.4.4. Using assessment data to make school improvement The curriculum changes and developments are based on the assessment data as the school improves.
Besides, the ongoing development process would automatically affect the school's improvement.
Moreover, it has been stated that the teachers’ essential role in assessment literacy is to raise school improvement standards (Green, 2013). The standard was decided by American Federation and Council (1990). They suggested that assessment-literate teachers should be able to use their assessments' data as a school improvement tool at this level of literacy.
Teachers can use accumulated assessment information to organize a sound instructional plan to facilitate students’ improvement (Shobeiry & Baktash, 2022).
Besides, they also use assessment results to plan and evaluate instruction and curriculum.
Automatically, teachers will interpret the results correctly and avoid common misinterpretations and
improve the remedial program from the classroom area until the school scope. The accurate assessment data helped the schools to enhance their curriculum too.
2.1.2.5. Developing accurate grading procedures for students The teacher could do a proper grading procedure by developing the assessment exercises, scoring rubrics, and sampling qualitatively (Stiggins, 1999). It is primarily for grading, which is vital in teachers’ professional practice (Green, 2013). Teachers, in this stage, should consider the two essential things in preparing for the assessment process. These are validity and reliability. The indicators of classroom test quality are based on them (Yasin et al., 2020).
It is, first, ensuring validity. It is where the test made is measuring what it is supposed to measure (Alias, 2005).
Validity is the criterion of a good test, which mainly means the test is measuring what it is measured (Fautley & Savage, 2010b). There are three ways to increase the validity of the test.
There are face validity and content validity. Face validity can be seen from the form that aligns with the target skill being assessed.
In contrast, content validity is the validity that needs experts or other sources to look into it to judge whether the test content is representative of the skills being measured (Alias, 2005). Meanwhile, reliability is where the test creates consistent results even though it is conducted at several different times (Jayanti et al., 2019). It has three types of reliability. Firstly, internal consistency. It is the consistency of every item in the test. Next, Inter-scorer reliability is the consistency between different teachers' scores. In contrast, Intra-scorer reliability is the consistency between the score given by the same teacher at different test times. Therefore, the teachers’ role to be consistent in grading procedures would increase the accuracy of the assessment method itself.
2.1.2.6. Informs various stakeholders about the assessment results The teachers who meet this standard would be able to understand and deliver appropriate information regarding the students’ socioeconomic, cultural, language, and other background factors related to their assessment (Council, 1990).
Besides, teachers could explain that the students’ background would not cause the assessment result since it will limit their significant educational development (Shim, 2009a).
