Transcript of video
This video is available from https://www.qcaa.qld.edu.au/senior/assessment/quality- assurance/endorsement/resources/chief-endorser-briefing
Slide 1 The following briefing is intended to support
Chief endorsers in their preparations for an endorsement meeting.
Slide 2 This presentation will cover:
• Endorsement overview
• The endorsement meeting
• Roles and responsibilities of chief endorsers
- Calibration
- Responding to messages
- Accessing and understanding reports in the Endorsement application, and - Quality assuring directives
We will also cover
• Resources.
Slide 3 Endorsement is a quality assurance process
designed to give schools confidence that their assessment instruments provide valid
opportunities for students to demonstrate what they know and can do. Endorsement builds teachers’ capacity to develop quality assessment that is comparable across Queensland schools.
It is important that we, and the assessors on our teams, remember that this is a process for approving assessment instruments for use with students. Implementation decisions are the responsibility of the teacher. What we are providing is a quality assurance check that
Preparing for an endorsement meeting
Chief endorser briefing
focuses on validity (a match to syllabus) and accessibility.
At each meeting we will be reviewing
assessment for all 48 syllabuses, from schools not just in Queensland but those offering the QCE in our offshore schools, so it is important to be flexible when evaluating assessment instruments as school contexts may be quite different.
Slide 4 Detailed information about the endorsement
process can be found in the
• QCE and QCIA policies and procedures handbook, which can be accessed from the QCAA Portal, and
• the Endorser manual, which can be accessed from the endorsement assessor webpage.
A link to the assessor webpage will be provided later in the presentation.
Slide 5 The endorsement process is spread over two
separate events for each cohort of students.
This allows schools to develop their
assessment closer to the time it will be used and provides them with an opportunity to reflect on their implementation of the syllabus, and other assessment instruments, as they begin to develop their new tasks.
An endorsement event includes all stages that are defined in the endorsement process, as well as internal quality assurance activities that support the process. The event includes:
• a one- or two-day meeting for Application 1.
All assessors in the subject team are involved in Application 1 quality assuring of directives before the QCAA publishes decisions to schools to ensure that they are error free and in scope
• a period of consultation that is offered to schools before they revise and resubmit assessment that was not endorsed at Application 1
• Application 2 reviews of resubmitted assessment that are completed by lead endorsers
• the Intervention process for schools that were not successful at Application 2. This activity is completed by the chief endorsers once all Application 2 decisions for all subjects are finalised.
When Intervention is completed, this is considered the end of the event.
Sampling by chief endorsers occurs after the February endorsement event is closed.
Sampling and the subject report are part of the endorsement process but are not considered part of an endorsement event. As chief endorser, you will be provided with a briefing and materials before the start of the sampling process.
Today’s briefing will focus on the endorsement meeting and the quality assuring of directives before they are published.
Slide 6 This workflow diagram can be found in the
endorser manual. The steps shown for Application 1 occur at the endorsement meeting.
Before they commence endorsing, all
endorsers and lead endorsers must complete a calibration exercise.
Once the endorsers and leads submit their response to the calibration exercise, as chief endorser you will be able to view them. Chief endorsers and their subject PEO provide feedback to their assessors about their decisions and directives. We will revisit how this is done later in this briefing.
After completing the calibration exercise, endorsers and lead endorsers move directly onto the assessment instruments submitted for this event. They should not compare
assessment instruments from one year to the next or compare schools. Collaboration with peers or the degree of change from one year to the next is the school’s decision. Each assessment instrument must be evaluated based on the evidence that is provided.
Once both an endorser and lead endorser have completed their review for an
assessment instrument, the directives can begin to be quality assured by the chief endorser and subject PEO.
In the week after the meeting, the QCAA will
publish the endorsement decisions to all schools.
Slide 7 There are a number of QCAA staff working
behind the scenes to support you and your team of endorsers and lead endorsers.
Decisions about disaster management, closing the meeting and approving the messages to assessors is the responsibility of the Manager of the Quality Assurance Unit and the Director of the Senior Assessment and Certification Branch. If the queue of assessment
instruments can’t be completed by the close of the meeting, the Manager Quality Assurance Unit will identify what actions will be taken and extensions to timelines.
The officers from the Quality Assurance Unit are available to:
• advise on process
• coordinate the approval and sending of subject-specific messages that will be sent to assessors through the application
• The application superusers are responsible for ensuring assessors are able to access assessment and will be responsible for reallocating roles and assessment instruments if needed.
Operations coordinators are responsible for meeting logistics and oversee issues with venue, travel and accommodation.
They will be monitoring attendance and are responsible for sending communications to assessors through the application. They may also follow up with assessors about timesheets for pay purposes.
There are a number of teams providing IT support. There will be officers who can provide desktop support to assessors at the meeting, as well as those who manage the
Endorsement and Assessor Jobs applications.
Slide 8 Learning areas support their chiefs and
assessors at the endorsement meeting by:
• making decisions about team strength, such as promoting a reserve lead endorser to fill a vacancy due to absences during the meeting
• helping you to monitor the achievement of targets
• assisting the chief endorser to quality assure endorsement directives helping chiefs provide feedback and conduct professional conversations when calibration of assessors is needed
• managing the ceasing of assessors once the assessment queues are completed.
If you are unable to attend in person, your subject PEO will be in touch with you before the meeting to make arrangements about how they will work with you remotely.
Slide 9 A schedule for each meeting is updated and
published on the Endorsement assessor webpage. It identifies the subjects that are meeting each day, the start, break and expected time the meeting will close.
This page also identifies venue information including which assessors have been invited to attend in person, and who is expected to work from home.
In most cases chief endorsers will be invited to attend the meeting in person with QCAA staff.
If you are unable to attend in person, you can nominate to join the meeting online.
The endorsement meeting starts at 8 am each day. The Endorsement application will be unavailable to assessors until the event is opened.
Lead endorsers start at 9 am on the first day of their meeting to ensure there are assessment instruments in their queue for them to begin reviewing.
All assessors start at 8 am on Day 2 if they have a two-day meeting.
We will have scheduled breaks and will send a message to all assessors during the day to remind them of the break times.
Slide 10 The main responsibilities of the chief endorser are to:
• review calibration responses and provide feedback
• respond to assessor questions in the Endorsement application
• monitor meeting progress and contact assessors if necessary
• quality assure directives.
The next section will show you how to use the Endorsement application to complete these tasks.
Slide 11 When accessing the QCAA Portal, please
make sure you have selected QCAA from the menu at the top right of your screen before you open the Endorsement application.
If you don’t see the QCAA organisation or the Endorsement application, please let your subject PEO know and they can pass the message on so that we can make sure you have the correct access.
Google Chrome is recommended for accessing the application.
Slide 12 The calibration sample is the first activity the
endorsers and lead endorsers complete. It should take no more than 20 minutes. The Calibration report has been updated to allow chief endorsers to provide feedback to their assessors about their performance in the calibration scripts.
Select the year > Event > your subject. You will also be required to select the sample. One sample is uploaded for each meeting day, unless you are otherwise advised. If the sample dropdown is blank, this means no assessor has submitted a response yet.
The report displays the number of issues we expect the endorsers and lead endorsers should identify.
Total differences score is the assessor’s match to the expected practices that should elicit a
‘no’ decision and a directive. A score of 0 is a direct match to the expected response. A score of 1 or more means either the assessor has identified extra issues or they missed an expected issue.
To see the assessor’s response, click on the Action menu to the right and select View calibration.
This will show you:
• the assessment instrument they have reviewed
• their completed assessment evaluation
• the directives they wrote.
Slide 13 Once you have reviewed the calibration
responses, you will be required to send feedback to the assessors in your team.
You may wish to speak to your subject PEO to confirm which assessors need to have a discussion to help them improve their decisions or their directives (or both), and which assessors can be sent one of the pre- populated feedback statements.
To send feedback:
• click on the feedback button above the assessment evaluation form
• select the appropriate feedback option. The options are
- Decisions and directives are appropriate - Decisions are appropriate but the
assessor is reminded to adhere to the communication strategy
- ‘Custom feedback’ to write brief messages tailored to a specific issue, such as missing one of the significant assessment practices, or
- A notification that the chief endorser will be in contact for a calibration discussion.
If you select the option to have a calibration discussion, you will be provided the assessor’s
phone number through the Endorsement application. Once you’ve made the phone call you can mark the discussion as completed to help you track your progress through the queue.
Slide 14 Endorsers and lead endorsers can
communicate about an assessment instrument that they are reviewing, in the Endorsement application, to the chief endorser and PEO.
They can’t use it to communicate with each other.
Chief endorsers need to use the Event
statistics menu and select Messages to check for any new messages. There is no prompt popping up on your screen to say there is a new message as this could interfere with other things you are doing, particularly if there are many messages being sent. Please work with your subject PEO to plan when messages will be checked and who is responding.
Once you have selected messages, there are two tabs:
• one with all the unread messages, and
• a second tab that includes all messages, in case you want to check what you said to another query, or to add extra information if needed.
Click the View button to the right of the screen.
Your message access lets you view the assessment instrument and any directives completed by the assessor, as well as their question.
Slide 15 The lead endorser can also see any messages
sent by an endorser for an instrument that is assigned to them, once the assessment instrument arrives in their queue. This means they can see the reply you provided so they don’t have to ask the same question; however, they can also add a new question and you can see all correspondence about that instrument.
The lead endorser cannot write a message to the endorser — it can only go to the chief and PEO.
If you are getting overwhelmed with queries or messages that are being sent, please contact your PEO for support. Your subject PEO can also read and reply to messages written by endorsers and lead endorsers in the
application; however, as much as possible we want our chief endorsers to be the main point of contact.
Slide 16 This table is a summary of the communication
you might be involved in during the meeting and the preferred mode of communication.
For the most part, your communication with endorsers and lead endorsers should be about subject-specific information and workflow support, and this should occur using the messaging feature in the application.
Your subject PEO can provide you with a contact email or phone number for assessors who require a recalibration conference. This might be as a result of the calibration script, or it might be identified during the meeting when reviewing their submitted assessment
evaluation during the quality assurance of decisions.
If you are no longer able to participate in the meeting due to an emergent situation or illness, please contact the Internal Assessment Operations Unit using the email on the screen and make sure you advise your subject PEO.
Slide17 The Endorsement application includes a
number of reports that help chief endorsers and PEOs to monitor the progress of assessment instruments that are being reviewed.
It is important to monitor progress and action issues early. For example, if an assessor seems to be making little progress, either you or the PEO should call them to identify if they:
• are working, have forgotten about the meeting, or are taking an unexpected break
• have many assessment instruments on hold while they are waiting on decisions or
• are having access issues in the application.
Assessors who are not working, or who are underperforming, impact on the rest of their team. It is not an acceptable solution to
redistribute work to other assessors — this just transfers pressure onto schools and lead endorsers if the assessment instrument is not endorsed and the lead endorser ends up with a high volume of consultations to complete in very short timelines.
Your chief endorser IT quicksteps explains each type of report and its use.
Slide 18 All reports you have available to you are in the
Event statistics menu.
When running any report, make sure you are selecting the appropriate exit year. This may not be the current calendar year, as the endorsement process always starts in August of the year prior for each cohort. Select the event number (Event 1 or 2) and Application 1.
Before the meeting starts, some of these reports won’t have data in them yet. If you would like a reminder of the data that displays, you could select data from the last
endorsement event.
Please note: when the report is being run, a red progress bar runs across the top of the screen. When a lot of data is being used, the reports can be a little slow. The progress bar shows if the report is still being generated.
In many of these reports, you will need to select your subject from the menus. Others will restrict you to your subject only. For this reason, we need to give you a quick reminder about being respectful of the confidentiality of this information.
Slide 19 The event progress is a big picture view of the
endorsement decisions for your subject and the progress towards completion.
The endorser and lead endorser reports help you keep tabs on whether individual assessors are working too slowly (or have stopped altogether) and also give some data on whether they are endorsing/not endorsing instruments — are they saying ‘no’ to all assessments?
Slide 20 There are two reports that provide data about which Priorities of assessment are being identified as an endorsement issue.
We recommend using the subject version for this meeting.
The Assessment by priority is used for our data for the subject report.
The priority reports identify patterns for instruments not endorsed for the subject. Is there an issue that requires recalibration?
Should we be sending a message to all the assessors in the team?
Slide 21 The Application outcome is used to quality
assure directives. This will be covered shortly.
The Application outcome report may also help you identify issues to target at the next
opportunity for training our assessors.
The Rate of endorsement report is useful for the subject report once the two events have been completed. You can ignore this report for the subject meeting. You can also disregard the chief sampling report for this meeting.
The event progress chart will be unpacked in the next slide.
Slide 22 This report graphically represents when we
expect the assessors to complete their queue and how they are actually working. From the data accompanying this chart, you will see how many endorsers and lead endorsers are currently available to work at the meeting.
The important features of this chart are:
• Data is drawn from the Assessor Jobs application so that the most recent data for assessor availability is used. As assessors notify that they are available or unavailable, the chart will recalculate the predicted completion times based on those values.
These predictions are available before the meeting starts and may change during the day.
• The number of assessment instruments submitted is shown for each subject.
• Endorsers’ and lead endorsers’ rate of work is mapped throughout the meeting. You will see that data change in real time.
• There are reference lines to show expected progression if assessors are working according to the piece rate for their assessment type. This helps us identify if they are working slower than expected, in which case we may need to send a bulk message or contact individuals to identify if they need support.
Slide 23 Chief endorsers and subject PEOs conduct a
quality assurance check to ensure directives issued to schools are in scope and error free.
We are not interrogating decisions; however, we may identify that some directives are not correct or appropriate and therefore the decision for the assessment practice may be overturned.
Quality assuring of directives can be started during the endorsement meeting; any directives not quality assured at the meeting become the responsibility of the learning area, not the chief endorser.
To quality assure directives use the Event statistics >Application outcome report.
Slide 24 When we are quality assuring directives, we
only need to check the ‘Not endorsed’
decisions. Endorsed assessment instruments are excluded from checking because there are no directives provided. Only the directives of the lead endorser are quality assured because endorser directives are not published to schools.
You can apply the filters to show only the not- endorsed assessments from the list.
If required, chief endorsers and PEOs can contact a lead endorser to revise their directives during the meeting. Alternatively, a chief endorser or PEO may make a change on behalf on the lead endorser if it is a simple change or if the lead endorser is no longer working.
When you have completed the review for a decision, you can tick the Ready to publish button to identify the ones you’ve finished.
This is just a flag to say that task has been
checked; it does not publish the decision.
You can then additionally filter to show only the ones left to do (not ready to publish). This will save you scrolling if you have a large subject.
Slide 25 Quality assuring of directives should look out
for the following common issues with directives:
• Assessors need to use the communication strategy. Many directives did not follow the communication strategy and were not actionable. This caused difficulty for chief endorsers and intervention officers when the action or the reasoning was not clear.
• Assessors should choose the best
assessment practice that matches the issue and write only one directive. It is not
appropriate to repeat the same directive under multiple assessment practices if the action is the same. Do not write ‘as above’.
If there is nothing different to be done and the instruction is the same, remove the extraneous directives and change them to yes. The school will still see there is an issue to address and the directive that identifies the changes required.
- If there are many actions required as the result of a fundamental error, each action can be written as its own directive, specific to the changes that need to be made. For example, if students are directed to write their response in the wrong format or genre, the task description may need rewording (which is Directive 1) and the scaffolding might need to be rewritten to match the specifications (which would be written in Directive 2).
• Schools are directed not to copy the QCAA samples as their own, therefore we need to be careful not to recommend that the assessment is revised to match the samples. If the assessment satisfies the assessment specifications it should be endorsed.
Slide 26 • Please remove references to formatting, style or design that are school business, for example:
- moving information from one part of the assessment template to another section anything pertaining to removing bold text, aligning bullets, American or English spelling.
• Also remember, a small amount of repetition is okay. Only comment if it is excessive.
• Do not provide directives for checkpoints unless they clearly contradict conditions of the syllabus. For example, schools do not need to include a specific week or date.
This is a school decision and dates are always subject to change.
• Finally, remember that assessors approve, not improve assessment. Unless the assessment is not valid (it doesn’t match syllabus requirements) or it is not
accessible, it should be endorsed.
Slide 27 Learning areas are recording ideas for
improvements so please pass any ideas or observations on to them to include with their notes. This may relate to trends you identify during quality assuring directives that can be included in future assessor training,
assessment samples that would be good future calibration samples or frequently asked assessor questions.
Slide 28 You can find detailed information about the
endorsement process on the assessor webpage. Meeting details are also posted here.
Slide 29 No voiceover
Slide 30 Thank you for viewing this briefing. We look
forward to catching up at the endorsement meeting.
If you have any questions, please phone or email your subject PEO or email
© State of Queensland (QCAA) 2023
Licence: https://creativecommons.org/licenses/by/4.0 | Copyright notice: www.qcaa.qld.edu.au/copyright — lists the full terms and conditions, which specify certain exceptions to the licence. |
Attribution (include the link): © State of Queensland (QCAA) 2023 www.qcaa.qld.edu.au/copyright.
