Authentic assessment for life competence
Presentation to Shanghai University of Finance & Economics October 2019 Professor Gavin T L Brown [email protected]
Professor of Educational Assessment
◦ Previously: Standardised test developer for K-12 testing in NZ
◦ Teach U’grad and P’grad courses in assessment & testing in NZ and HK
◦ Research focus on the impacts of assessment methods, policies, and purposes on the beliefs, attitudes, and behaviours of teachers and students
◦ Currently supervising multiple PhD studies into HE assessment in PRC
◦ Developing a software system for use in UoA to evaluate MCQ tests/exams and help instructors set appropriate standards
Introductions
Not to give you definitive answers to a complex and organic situation.
Instead deepen and extend a dialogue that may allow us to do and understand better.
I am here to lend an expert view, but that is not the same as telling you the answer.
I see my role as helping contribute to and move forward meaningful dialogue in a high-value area.
Role and intent
What is authenticity?
Role of a university in society
Role of a university as an educational organisation
Role of intellectual depth
Role of Risk Management
Outline
Assessing a real world practice
Hair Dressing
◦
You’ve been a client, you’ve had hair cut, styled, etc.
◦
what criteria and levels of performance would you create to rate or assess the last hair dresser or barber you
visited?
Personality? Skill-level? Price? Communication skills?
Attractiveness?—what makes it a good haircut?
Discuss
◦If this is hard, imagine how hard it is to score authentic work in complex fields
Discuss
What makes an assessment authentic?
What makes it seem authentic?
What are the characteristics of authenticity that we can replicate in our teaching-learning environment?
Authenticity
of undisputed origin and not a copy;
genuine; real; verified; not false;
origin supported by unquestionable evidence;
trustworthy; authenticated;
entitled to acceptance because of agreement with known facts or experience;
What does the dictionary say?
Good
Assessment Authentic Assessment
What is authentic assessment?
Good Authentic
Valid Mimics exhibition of real behaviours Reliable Requires holistic judgment
Efficient Inefficient, slow
It depends on who you ask
◦ Roles, responsibilities lead to different ideas about authenticity
◦ Disciplines differ: Authentic law vs authentic welding?
◦ Inducting new students from being ‘interested outsiders’ into
‘competent insiders’
How to come to synergy?
◦ Need to agree on what that looks like
◦ Need to agree on how to assess it and report it
Why are there so many tensions around
authentic assessment?
Roles & Responsibilities
Employer Student
InstitutioHE
n Society
• We prepare students for society and work by teaching them things that
universities are good at.
• But we are not job factories.
• We are not
responsible for everything.
What is the purpose of a university?
Employability: graduates get jobs.
◦ THUS, authentic assessment has to align with skills, knowledge, and functions that predictably will occur within the multiple domains of employment.
But a university has a mandate that involves local, national and global citizenship and the competencies that speak to these.
◦ So, this must be part of defining authenticity.
The role of a university
Specialist knowledge & abilities (Technical, Discipline-specific)
◦ Yes it is authentic to test knowledge (declarative, procedural) because it is required in the ‘real world’
General intellectual skills & capacities
◦ Supposedly if you can pass our assessments, you must have these?
Personal qualities, dispositions, attitudes, attributes
◦ Supposedly obtained, but do we know? Are these incidental, rather than deliberate?
Authentic HE outcomes
Does coming to HE assure society that graduates have these to some degree?
How can they be sure, if we aren’t sure or never even look at it?
Or are these just aspirations????????
Developmental pattern of intellectual skills
Where are
university students in China?
More advanced in Learning than
Knowledge?
Why is technical discipline not enough?
Academic Requirements Life, Citizenship, & Employment
Mastery of discipline-specific technical knowledge & skills
Communicative ability
Cognitive skills
Self-management
Inter-personal competencies
Appropriate technical competencies
Communicative abilities
Cognitive abilities
Self-management
Inter-personal competencies
It’s not just accounting, welding, history, etc.
Interpersonal skills: responsive to verbal and non-
verbal communication utilising effective communication methods when interacting with people
Managing relationships: establishing and building relationships with stakeholders using empathy, rapport, networking, and cultural and ethical understanding
Team working: working with stakeholders utilising group facilitation and management, conflict resolution and accepts and gives positive and constructive criticism
Employability skills: transferring knowledge, problem solving, work experience, self-management and career building skills to employment related situations
Leadership: taking decisive control over situations in an assertive manner, manages and develops subordinates, and builds a positive sense of group purpose
Flexibility, creativity, and innovation: using flexible open-minded approaches with resourcefulness to create solutions to innovate situations whilst also demonstrating compromise and adaptability
Critical thinking and communication: critically analyses situations, generating hypotheses and clear communicate strategies using verbal and non-verbal communication methods
Self-reflection and self-motivation: reflecting on
abilities and aspirations to identify learning challenges and actively seeks to accomplish them
Defining HE ‘soft skills’
outcomes
Slade, M. (2019). Understanding Taylor College student academic attributes. (M.Ed. unpublished thesis), The University of Auckland, Auckland, NZ.
UoA Graduate Attributes
Old New
27 skills, but curriculum
focused on discipline specific knowledge
Leadership treated them as largely aspirational & ignored
Presumed achieved
◦ Not assessed directly
◦ Not reported
4 big ideas
◦ Leadership; Scholarship;
Innovation; Global Citizenship
◦ 6 major themes
Leadership support
Disciplinary Knowledge & Practice Critical Thinking
Solution Seeking
Communication & Engagement Independence & Integrity
Social & Environmental Responsibilities
OLD Graduate Profile
◦ “The blunt answer is, it doesn’t [play a role], if you mean that we consult the Profile. But it does [play a role] as far as the ideas in the Profile are ones that we live by”
◦ “[The purpose] is really to define the fundamental body of knowledge that we require for our students and what we guarantee as a faculty that our students can do.”
◦ “sets university graduates apart from graduates of other tertiary
institutions (e.g., trade/tech schools). University graduates were believed to be more broadly educated than others who undergo specialised
vocational training.”
Aspirational, important, but ignored?
Interview study with senior leaders
UOA
Students engage in:
• Big picture thinking
• Foundational concepts
• Planning and preparing
Students engage in:• Practicing
varied
applications
• Reflecting
• Responding to feedback
EXREPLO
EX
TEN D EX
HIBI T
ENTHU SE
Students engage in:
• Real and
relevant tasks
• Using multiple capabilities
E
XPLOREStudents see the learning is:
• Relevant to
• Important
• Achievable
E
XTENDE
NTHUSEE
XHIBITwww.futurereadygrads.co.nz
NEW Framework: A pedagogical framework for
teaching future-ready graduates
Authentic assessment is educational
Academia Real World
Learners; Not yet competent
Purpose: prepare generically for beginner role
Design: Analytic, sequenced, scaffolded, selected &
controlled, well-structured
Professionals; competent
Purpose: achieve a specific work-related goal
Design: Integrated, dynamic, simultaneous, unpredictable, ill-structured
Learners who can fail
Instructors who support
Workers who must succeed Informal
instruction?
Remember we are teaching learners, not developing already graduated professionals….
Every task we set is an opportunity to do 2 things
◦ Judge quality, rank, report grades (Accountability)
◦ Assess or diagnose success and needs, and prescribe solutions (Improvement)
Feedback opportunities have to be designed into the sequence of assessed tasks
◦ Task 1 must be simpler than following tasks
◦ Task 1 must be essential for later tasks, so feedback is relevant
◦ Task 1 diagnostic information must help improve Task 2 performance
In terms of knowledge, understanding, skills
Designing for Feedback
A mental model that guides the program and all participants in deciding what to
assess
◦ Assessment = Demonstration
◦ Show me what you understand & can do with your knowledge to meet learning goals
Authentic Assessment is Exhibited
http://www.futurereadygrads.ac.nz/the-4es/
I already know you can answer MCQ or write an essay….
How can I get you to apply the that knowledge and skill in a novel way, so that I can see your ability to integrate and
synchronise multiple objects, processes, and ideas simultaneously?
◦ Because this is the way of the world…it’s authentic.
Example: My own course Educ 224 Assessment &
Evaluation in Education within the BA degree
Authentic assessment exhibits integrated
knowledge & skills
Use standardised test
Feedback Reports of standardised test
Write MCQ, get &
give peer feedback
Create test, administer,
score, feedback,
evaluate validity
Educ 224
Assessment Design
50%
25%
15%
10%
Final Grade
A complex skill supported with scaffolded
tasks & lectures & tutorial which contribute to the total. Feedback leads to improved
performance. The challenge is using theory, not repeating it.
Lectures Readings Tutorials
Discuss
◦ How (or what) would you restructure the sequence and nature of assessments in your program so that students:
Are supported in developing component skills
Demonstrate that they can integrate knowledge with competencies
Receive feedback in such a way that performance provides insights into improvement
Receive sequences of assessments within courses that support developmental acquisition of knowledge and skills
◦ What systems do you want to ensure your program supports learning?
Design for Learning, not just
judgement
Real learning occurs when learners do things that are achievable but challenging
◦ Higher-order thinking skills,
◦ do more complex; not just “more in less time”
Cognitive challenge not just long and hard
How can we ensure that we build challenge into all aspects of curriculum, course content, and assessment?
◦ Biggs & Collis (1982): Structure of Observed Learning Outcomes (SOLO)
◦ Let’s look at the cognitive characteristics of the exhibited learning
Authenticity not just difficulty
SOLO: Structure of Observed Learning Outcomes
Surface Deep
Unistructural (Fail)
Multistructural (Satisfactory, C)
Relational (Good, B)
Extended Abstract
(Excellent, A)
Understandin g of
Awareness &
Evaluation of Multiple Stances
Selection &
Application in Static well- structured simulation
Selection &
Application in Dynamic ill- structured simulation
Authentic Assessment
Works toward Professional Competence
MCQ Test or Exam Course-work Assignment or
Written Exam
Performance Assessment under Lab conditions
Performance Assessment under Real-world conditions (Practicum/Internship)
Advanced Introductory
But at Year 1 students should be doing more than remembering! So should go more quickly—esp. post Gao-Kao
Emotionally intelligent
Principled
Ethical and professional
Know the theories
Critical thinkers
Proactive
Know the
‘facts’
Creative Independent
thinkers
Teaching – a shift in emphasis
Flexible Leaders
Problem solvers
Open minded Globally minded
Culturally competent
Becomi ng Doing
Knowin g
Transformationa l teaching
Hard to teach
Easy to teach
Hard to forget; Last a life-time Easy to forget
Become out-dated
28
Objective marking vs. Professional judgment
◦ Trivial content?
Easy to mark content vs. Hard to mark
◦ Time to manage?
Knowledge about vs. Knowledge to do
◦ Does one lead to the other?
Quantity of knowledge vs. Quality of performance
◦ What do our grades and qualifications mean?
Assessment Tensions
Quality Judgment
?
Discuss:What does your end-user community expect?
What does your institution mean by grades—quality or quantity?
Is quantity correct equal to high quality performance?
What threats are there for this claim?
Excellent; Highly Competent (A)
Good; Competent (B) Adequate; Minimally
Competent (C)
Inadequate; Not Adequately Competent (D)
Standards in Education:
Academic Grades
Grade Description
A Highly Competent. High to exceptionally-high quality;
excellent knowledge and understanding of subject matter and appreciation of issues; well formulated arguments
based on strong and sustained evidence; relevant literature referenced; high level of creative ability, originality and
critical thinking; excellent communication and presentation skills.
B Competent. Good to strong grasp of subject matter and understanding of major issues though not necessarily of the finer points; arguments clearly developed and based on
convincing evidence; relevant literature referenced;
evidence of creative ability, originality and critical thinking;
good communication and presentation skills.
Possible Grades & Standards:
Sample from University of Auckland
Possible Grades & Standards:
Sample from University of Auckland
C Minimally competent. Adequate knowledge of subject
matter and appreciation of main issues though possibly with some lapses and inadequacies; arguments developed and supported by some evidence and references; creative
ability, originality and critical thinking present but limited;
adequate communication and presentation skills.
D (Fail)
Not adequately competent. Lacks breadth and depth and generally has gaps. Frequently takes a simple factual
approach and understanding and coverage of material is inadequate; does not attempt to interpret the material;
indicates a need for considerable effort to achieve
improvement; communication and presentation skills are poor.
If you have rubrics and standards how do you know if your judgments match each other or the standards expected
outside by employers or graduate schools or society?
MODERATION Panels
◦ Show sample work and rubric: ask for judgment and reason
◦ Compare (target 70% or better exact agreement)
◦ Force discussion and consensus before using grades
◦ Create benchmark exemplars
Agreement?
Valued but not implemented?
◦ Easy in Teaching & learning, but assessment?
Change Management
◦ Difficulty in moving from static knowledge to dynamic implementation
◦ Performances are harder to judge, easier to use MCQ, essays
◦ Students don’t like novelty in evaluation practices
Managing Change
◦ Accountability effects
Change may not get same result. Will this be used to evaluate instructors &
courses?
◦ Fear punishment
Might lead to ‘badge engineering’; accommodate new, but do the old
◦ Manager discussion, approval & support for risk taking
Time & safety to experiment and develop
Managing Risks in
Authentic Assessment
Requires serious discussion and agreement around
◦ Role of a university in society, employment, and life
◦ Role of a university as an educational organisation
◦ Role of intellectual depth
◦ Role of risk management
◦ Expectations of employers and society
Authentic assessment
