Table of Contents
Lecture 1: Introduction ... 2
Lecture 2: Quantitative and Qualitative Data ... 7
Lecture 3: Principles of Measurement and Tutorial Notes ... 12
Lecture 4: Standardised Testing and NAPLAN ... 18
Lecture 5: Quality of Data - Reliability ... 26
Lecture 6: Quality of Data - Validity ... 33
Lecture 7: Data - Rubrics, Assigning Grades & Reporting ... 38
EDST4200 Exam Notes
Week 1 Lecture & Tutorial Notes Education data needs to be quantifiable.
• Underlying idea = positivism = idea that there is a single, measurable reality. Underpins data used to inform classroom-based decision.
• School teachers who use educational data greatly improve the academic outcomes of students (Hattie). Hattie reported synthesis of over 500 meta-analyses (20-30 million students). Identified more than 100 factors influencing student achievement. Attributes of schools, homes, students, teachers, and curriculum. Effect size of 0.4 is average.
Top pedagogical factors associated with using educational data: self-assessment and reporting, formative feedback to teacher & student, student prior achievement.
Hattie’s work has come into some criticism lately. He acknowledged that there are problems with ranking by effect size.
International Sample Assessments:
1. Programme for International Student Assessment (PISA)
2. Trends in International Mathematics and Science Study (TIMSS) 3. Progress in International Literacy Study (PIRLS)
Measurement, assessment, and evaluation are used interchangeably in education
Dimensions Definition/description
Measurement A set of rules for assigning numbers to represent objects, traits, attributes, or behaviours
(NAPLAN assigns numbers that represent an individuals performance) The process of determining a quantitative or qualitative attribute of an individual or group that is of academic relevance
Assessment
Has become a broader connotation than measurement
Relates to a related set of measures used to determine a complex attribute of an individual or group of individuals.
Evaluation
Includes the notions of merit and worth.
The process of determining to what extent the educational objectives are being realised.
A judgement on how well or poorly the educational objectives being realised.
Assessment encompasses measurement but it is a BROADER TERM.
Assessment for learning FORMATIVE
To enable students, through effective feedback to fully understand their own learning and the goals they are aiming for (ie. formative assessment, progress evaluations)
Student self-assessment, reflective task, written and oral feedback, interviews with students, collection of student work, student portfolio.
Not a product but part of learning cycle.
Assessment of learning SUMMATIVE To grade or summarise performance (summative learning)
Collect evidence of student knowledge, skill or
proficiency
Occurs during units of work, towards end of term, used to rank/grade students.
Benchmark tests, formal essays, reports, end of term work, portfolio’s,
Learning product
Assessment as learning Occurs when students are their own assessors.
Students monitor their own learning, ask questions, and use a range of strategies to decide what they know and can do, how to use
assessment for new learning Students monitor their own learning, ask questions, decide what they know and can do, how to use
assessment for new learning.
Weekly journals, reflections of learning.
Norm Referenced (Standardised)
The process of giving meaning to scores by referencing them to the cohort or normative group is called norm or cohort referencing.
SAT, IQ tests, school readiness test, NAPLAN (test against rest of Australian cohort)
Criterion Referenced
Candidates are measured against defined (and objective) benchmark. You are either competent or you are not. (Pass/fail)
Driver’s test, end of year exam Maths test for summative performance to account for a percentage of overall grade.
Standards Referenced
An expansion of the criterion-referenced approach. The performance of students is mapped against a range of performance
descriptors rather than a single benchmark level of performance (i.e., pass/fail).
Multiple choice, short answer, writing task, maths test NAPLAN test to determine if students have attained benchmarks of proficiency Ipsative
Referenced
Compares your current performance with a previous performance for example your personal best.
Spelling test, running speed, reading level (beginning vs end of term)
Diagnostic Standardised Informal Formal
Designed to identify areas of weakness and strength. May include: checklists, running records, placement tests, and formal assessment tools.
Formal, reliable and valid, often norm referenced tests
administered and scored in a particular way.
Evaluation not to grade
performance using metrics.
evaluation to grade performance using metrics
Week 1 Reading – Evidence Based Approach to teaching and learning (Bruniges, 2005)
There are four main ways in which we can use the information we gain from assessment to maximise student learning and outcomes. This includes using evidence to:
1. Improve the focus of our teaching (diagnostic capacity)
2. Focus students’ attention on their strengths and weaknesses (a motivation capacity) 3. Improve programming and planning (a means of program assessment)
4. Report on assessment (a means of communicating student achievement).
To effectively support students, evidence to inform teaching and learning must be explicit, accountable, equitable, representative, valid and reliable.
Minimal and nondescriptive information about assessment can disengage students and limit their capacity for achievement.
• Be open
• Use questioning
• Conversation
• Give explicit standards and criteria and feedback to students.
• Use of criterion referenced assessments (in favour or in collaboration with traditional norm- referenced assessment) has enabled the use of data to identify particular strengths or weaknesses in curriculum – at classroom, school, and system levels. For instance, the development of the rubric has been powerful in supporting students learning (gives them informative feedback, and evaluation of their final products).
• Conversation with students about assessment is crucial for giving the learner the opportunity to discuss how goals are set, how performance is measure and how performance can be improved. The learner has an active role in the process.
• Hattie’s evidence shows that “excellence in teaching is the single most powerful influence on achievement.” Teachers account for about 30% of the variance in student achievement – the major source of within-school variance.
• Alan Luke (1999) argues that effective education requires alignment of the three ke message systems that exist in education: curriculum, pedagogy and assessment
• Teachers should play a lead role in evidence-based approaches to teaching and learning.
Week 1 Reading - Hattie Article (2005):
Key Findings:
1. Ask relative questions of effectiveness
2. The use of effect-sizes in classrooms to underpin the discussion on effectiveness
- Effect-sizes assists teachers in the delivery of their program, forces schools to have clear goal and standards of student performance. Teachers can collect and review information to inform themselves about their levels of success with their students in reaching those standards.
3. The importance of learning intentions and success criteria
- Students often rarely know the learning criteria for a particular lesson.
- Using LIs and success criteria makes evidence easier.
4. Assessment data is optimised when teachers conceive such data as about them (and not the students)
- Teachers need to move away from considering achievement data as saying something about the student, and start considering it as saying something about their teaching.
- Brown (2004). Four major conceptions (among teachers): assessment improves teaching and learning, assessment makes schools and teachers accountable, assessment makes students accountable and assessment is irrelevant.
- Irving (2005). There should be a greater use of student evaluation of teaching. He found that students could dependably discriminate between NBC and non NBC teachers. The data is there, but do we have the courage to use it?
5. Movement towards student empowerment of teaching and learning
6. Enhancing teacher performance to improve student learning is conditional upon evidence - Not enough PD focused on enhancing student learning. Too often, PD is related to
working conditions and correlates of student learning.
- In a recent synthesis of literature Timperley was only able to locate 17 articles related to the effects of PD on student learning.
- Timperley proposed 5 elements of professional learning communities:
1) Development of shared values and expectations about children, learning, teaching and teachers’ roles and the relationship of these to the environment.
2) The collective focus on student learning that becomes part of the normative control of the professional community
3) Collaboration, foster sharing expertise, creating shared understandings
4) Deprivatised practice, time and opportunity to talk to each other about teaching.
5) Reflective dialogue, self-awareness, in depth conversations about teaching and learning.
Key idea:
• moving from discussion of data -> interpretation
• moving from student outcomes -> teaching successes and improvements
• moving from accountability models located about schools -> located in the classroom to support evidence based learning and teaching.
The major sources of evidence relate to diagnosis and formative assessment models and are centre on three major questions:
1) Where are we going?
2) How are we going?
3) Where to next?
It is possible to devise a national accountability model based on evidence critical to teachers.
