Chapter 6. Quality of Student Learning as Measured by International Tests
7. Key issues and challenges for improving performance
Despite a decade of investment and reforms in education, the performance of Indonesian students on PISA and TIMSS, has remained unacceptably low. Across all the tests, the majority of Indonesian students are below minimum competence on the basic skills. This means they are entering the workforce or senior secondary education with an average lag of two to three years behind their peers internationally.
Table 21. Summary of performance on international tests Subjects
assessed
Av. score
& country rank
Trend Gender gap Benchmark analysis PIRLS
Grade 4
Reading 428 (42/45)
Increase Girls’ scores signif.
higher
34% below low benchmark 38% at the low benchmark TIMSS
Grade 8
Maths 386 (38/45)
No change Girls’ scores signif.
higher
57% below low benchmark 28% at the low benchmark Science 406
(40/42)
Decrease Girls’ scores signif.
higher
46% below low benchmark 35% at the low benchmark PISA
15 years old
Maths 375 (63/65)
No change None – data unreliable
75.7% below minmum proficiency Science 382
(64/65)
No change Girls’ scores signif.
higher
66.6% below minimum proficiency
Reading 396 (60/65)
Increase Girls’ scores signif.
higher
65.2% below minimum proficiency
While the results in Science and Maths are low and not improving, Literacy while still low, is improving. The positive trend in Literacy is evidence that improvement is possible and can be sustained over time.
Summary interpretation of performance on international tests
Average performance – The mean score of Indonesian students is very low on all tests. Indonesia is near the bottom of the rank order of participating countries and is closest in overall performance to Malaysia, Colombia, Morocco and some Gulf countries. The mean scores are around one standard deviation below the OECD mean, equivalent to a lag of about 2.5 years.
Subject trends – In Literacy there is a significant upward trend on PISA which is also emerging on PIRLS (Grade 4). In Maths there has been no change in a decade of PISA or TIMSS. In Science there has been no change in a decade of PISA and a decline in over a decade of TIMSS tests.
City schools Population ≥ 100,000 Score difference of city schools
over other schools
Score adjusted for ECSC Town schools
Population 3000 to 100,000
City schools higher by 36 points Significant
Significant Rural schools
Population ≤ 3,000
City schools higher by 48 points Significant
Not Significant
Gender – There appears to be a significant gender gap in favor of girls across all subjects and all tests, except the PISA Maths gap 2011 which was not regarded as reliable because of the number of students who scored too low to be included in calculations of significance.
Benchmark analysis – Focusing on PISA and TIMSS which assess students just prior to the end of basic education, it appears that the majority of students are not achieving minimum competency and there are no students at the advanced level or level 6.
Critical areas to be addressed – As well as overall development, the two areas for critical attention are Maths and boys’
achievement.
7.1 Targeting low performance
The immediate challenge for Indonesia is to reduce the number of students below the minimum level of competence as this has important implications for the individual students and for Indonesia’s growth and national competiveness.
Improving the bottom end of the distribution does not mean neglecting to nurture high performance.
Different strategies (e.g. talent development programs) can be implemented in different schools and even in the same schools and the same classrooms. Effectively targeting low performance requires rich assessment data from school and national tests with sufficient diagnostic information to identify the particular competencies and problems that need to be worked on, along with regular monitoring.
Effective targeting is needed at two levels: by local government to identify schools for intervention and by classroom teachers to identify students for within-class attention. Both of these need robust and meaningful data about performance beyond the information from international tests. Teachers and education officials need improved capacity and support to design, implement and use assessment information effectively.
7.2 Assessing performance in the early grades
Information about performance in the early grades is imperative to establish a successful improvement program that will ensure the foundational skills for later grades. Steps towards this could include -
• strengthening basic skills training in teachers’ preservice education for the elementary school,
• support for more effective classroom assessment,
• developing specific early grades assessment instruments and learning readiness checklists for Indonesia,
• promoting parent education programs to support literacy in the home and engaging with parents on expected levels of achievement,
• establishing an effective national sampling test to monitor performance in key skills and participating in international tests (e.g. TIMSS Grade 4) to provide system level information.
With the support of development partners many districts have been implementing the Early Grades Reading Assessment (EGRA) which has been customized for Indonesia and provides broad information for teachers on the level of student reading achievement. Training to administer the tests is also a form of professional development for teachers, providing them with practical ideas for teaching reading more effectively.
The best teachers should have responsibility for teaching in the early grades. While this may seem counter- intuitive to some, i.e. more skill is required for the academic competencies in higher grades, many education systems around the world now recognize the key role of early grades teachers in establishing the foundation for basic skills and the foundation of positive attitudes towards learning that influence the entire school experience.
7.3 Improved classroom assessment within a more comprehensive assessment system
Improving and enriching classroom assessment is a goal of Kurikulum 2013. To improve the learning of all students, teachers need to become highly skilled in formal and informal assessment techniques so they can provide constructive feedback to students and they can capture data about progress that will inform their teaching program and approach. This is the main type of assessment that will lead to improved learning outcomes. However, teachers, principals and policymakers also need reliable system level information to benchmark what is being achieved in schools, and parents and employers need some comparative information to understand a student’s level of achievement. These purposes need different tools.
A comprehensive assessment system is needed which has purpose-built tools for each different assessment level (e.g. student evaluation, school mapping, performance management). Each tool must be the best it can be in providing reliable and valid information for its purpose and be fully implemented. Filling the current gap in reliable national sampling of basic skills achievement in the primary years must be a priority to provide the data that will drive improved performance all the way through Primary and Junior Secondary. In a comprehensive assessment system, exams would be improved in scope and made more rigorous, and existing quality assurance strengthened.
7.4 Compensating for socio-economic disadvantage
The international test data show the extent to which socio-economic circumstances influence performance, as well as the practices associated with higher-performing schools. In general terms, there are three main types of interventions that can be implemented simultaneously. Indonesia already has appropriate program responses but they can be implemented more effectively to target students better.
1. Direct welfare assistance to individual students to ameliorate home background factors (e.g.
access to pre-primary, scholarships, meals, uniforms, equipment, cash payments to increase the educational resources at home).
2. Targeted programs and funding to disadvantaged schools to address school level factors (e.g.
educational resources; specialist teachers; better qualified and experienced teachers and principals; time for teachers to work together to develop the features of high performing schools).
3. Implementing policies system-wide which are known to be associated with improved performance for all students (e.g. quality assurance, using data for improvement, improving school-based management and collaborative planning, giving schools more autonomy and increasing teachers’
professionalism).
7.5 Targeting boys’ education
Over the past two decades, an international trend has emerged for boys to perform lower at school and university than girls, especially in literacy, and within literacy, specifically in writing. In Indonesia the gender trend on international tests appears to be generalized to Maths and Science as well.
Experts point to a range of factors in boys’ socialization which appears to underlie these trends9. These include a low motivation stemming from perceived irrelevance of schooling, a preference for active rather than passive learning, peer culture which may be mildly or strongly anti-authority, low self-esteem and confidence, uncertainty about jobs and the relevance of school to work, exposure to anti-establishment themes and values in popular media, and weak discipline at home. Boys’ lower achievement may also be related to their higher absenteeism, higher rates of suspension, earlier drop-out and disengagement from schooling even if still enrolled – so they have less exposure to teaching and less opportunity to learn.
The immediate need is for research in Indonesia to identify how widespread is the trend, how significant and what action may be needed to address the gap. The gender gap in other countries is associated with disadvantage. Indonesian research should therefore seek to answer questions such as which boys, why and what works?
7.6 Strengthening the instructional leadership of principals
Improving learning for all students requires leadership and support, and principals are the ones who must be the instructional leaders of their schools. The task can be shared of course, through a model of distributed leadership, but principals are ultimately responsible for the quality of the learning program in their schools.
Their focus on learning must be relentless. Nothing can be more important to them on a daily basis than student learning and development.
The current reforms in the selection, appointment, preparation, continuing professional development and performance management of principals need to be implemented comprehensively, in all local governments.
Similarly, supervisors need to be equally well qualified and prepared for their role in supporting principals to be effective pedagogical leaders. Paper and pencil tests of knowledge and attitudes are insufficient to judge the effectiveness of leadership.
7.7 Changing teaching practice
Given that the biggest influence on student performance after home background is at the teacher level, the low performance on international tests must be addressed through changing teachers’ classroom practices.
It is highly likely that in Maths for example, teachers themselves are unfamiliar with and unskilled in a teaching/learning approach which requires thinking skills – reasoning, inferring, problem solving and applying knowledge to contexts, rather than memorization and recitation as the main form of teaching and assessment. Teachers must become competent themselves in higher order thinking skills, problem solving and applying knowledge and skills to contextual situations before they can develop these skills in their students.
Changing teachers’ conceptual framework and classroom practices should capitalize on systematic implementation of the reforms which have been introduced, at all stages from recruitment and pre-service to induction and continuing professional development. However, it may also require a shift in the mindset of policy-makers from programs which develop teachers as individuals, one by one (e.g. through access to qualifications or in-service), to programs which develop the learning community within the school. School improvement literature of the last two decades highlights building the social capital of the school as an effective means of transforming and sustaining good teaching practice.10
7.8 Having a clear goal
Indonesia faces three major challenges. One is to systemically reduce the percentage of students below the low benchmarks. The second is to nurture, extend and develop students who have acquired basic skills to think more critically, reason and solve problems to become high performers. Thirdly, these tasks must be approached in such a way as to ensure equity. This should become the objective of all levels of government.
The first step in facing these challenges is for all levels of government to have valid and reliable data which accurately reflects what students know and can do. The most critical data must be gathered from the primary and lower secondary grades. International benchmarks provide reference points but locally-derived diagnostic testing is needed for use at the classroom level. The proposed Indonesian National Assessment Program (INAP) would be an appropriate tool for this purpose. There are also other options which could be
explored. However, whatever decision is made, the improvement strategy needs reliable tools for assessment and monitoring of achievement.
Schools should set their own realistic targets for improvement, recognizing current levels of performance (e.g. to reduce the percentage in the lowest performance band by a stated % and to increase the percentage in the higher bands by a stated %) and they must be supported in designing and implementing their individual improvement program. It is highly relevant to note that one of the factors associated with higher performance of Indonesian schools on PISA was schools having an improvement plan.
7.9 Urgency for a national focus on Maths
Maths is the area with the highest percentage of students not achieving minimum competence by the end of basic education, and this low performance appears to be entrenched. This makes it a significant target for improvement. Acknowledging the problem at the highest level may be needed to provide the motivation and political will to spark and sustain a comprehensive program to improve Maths teaching and Maths performance nationally.
It is acknowledged that Literacy is at least equal in importance to Maths as a foundational skill as it is language that mediates learning. However, on the international tests, Literacy is improving, while Maths is not. Good teaching practices in Maths at the Primary level will readily spill over to other areas of the curriculum. At Junior Secondary level, Indonesia already has a strong Maths Teachers Association that could be empowered to take a leading role with teacher education institutions, quality assurance institutes (LPMP) and others in developing provincial and local programs. The Maths strategy should have an action research approach so that information about what works in classrooms can be quickly tested in a range of contexts and generalized.
7.10 Monitoring the impact of Kurikulum 2013
Kurikulum 2013 has been developed in part to address gaps in curriculum and students’ lack of experience in applying Mathematical knowledge and processes to contextualized problems.
PISA researchers point out, however, that it is not just the content of the curriculum which is important, but the process of teaching it, which makes a difference in student performance. Monitoring of Kurikulum 2013 must focus most clearly on how teachers are teaching and assessing, not just whether teachers are adapting their lesson plans to new content or whether teachers can pass a knowledge test about the new curriculum.
The issues of pedagogy and assessment are addressed more comprehensively in the chapter on curriculum.
7.11 Improving efficiency of expenditure
From an analysis of the performance and education expenditures of the 63 countries participating in PISA, OECD researchers conclude that beyond a minimum level of expenditure, there is no statistically significant relationship between country spending on education and average performance on the tests. How countries expend their funds matters as much, if not more, than the amount they spend. This is one of the key issues for Indonesia and is a recurring theme in the Background Study. Spending more, to do more of the same, will not deliver the desired improvements. The lack of improvement in Maths and Science over more than a decade is strong evidence for this conclusion.