Step 5: Cull and sort the selected texts
2.5. Literature Review
The first stage of the two-stage literature review included a review of moderation and eModeration. The second stage focused on HCI, with search terms including usability, user experience, quality in use, and product quality. The literature review was conducted on different data sets for each stage. The flow of information during the phases of the literature review process is depicted in Figure 2-1.
Figure 2-1: Flow of Information through the different Phases of the Literature Review Adapted from Moher, Liberati, Tetzlaff and Altman (2009, p. 267)
38
2.5.1. Literature Review of Moderation
This section outlines the literature review related to the moderation of assessments, beginning with an overview of what moderation is and a brief discussion of the importance of moderation, followed by a discussion of eModeration. This is followed by a discussion of the systems currently used in educational environments. This section concludes with a discussion of usability as applicable to an eModeration system and outlines the requisite components of an eModeration system. This literature review principally draws from the works of Adie (2009, 2010, 2011, 2012), Adie, Lloyd and Beutel (2013), Beutel, Adie, and Lloyd (2014), Bloxham (2009), Bloxham, Hughes, and Adie (2016), Klenowski and Wyatt-Smith (2010), and Adie and Klenowski (2016), who are recognized as specialists in moderation practices.
2.5.1.1. What is moderation?
Moderation is the process of reviewing and adjusting the grading of assessments to ensure that they are fair, consistent, and accurate. This process is necessary to ensure that students are evaluated based on the same standards and that the evaluations are valid and reliable (Connolly, Klenowski, & Wyatt-Smith, 2012; Krause et al., 2013; Wyatt-Smith, Alexander, Fishburn, &
McMahon, 2017; Williams, 2019).
Researchers have described moderation in various ways. Adie (2012) explains moderation in terms of a technology-mediated interaction among teachers from geographically dispersed areas. Adie and Klenowski (2016), Beutel, Adie, and Lloyd, (2017) and Gourdin et al. (2019) on the other hand, emphasize the physical communication between teachers when collaboratively developing assessment criteria. Thus, moderation is deemed to be a social, quality assurance activity to establish a shared understanding of standards among different assessors. Bloxham et al. (2016) specify different moderation practices and examine the purposes of moderation. In contrast, Adams and Anderson (2019) locate moderation as an efficient method of professional learning and accountability for teachers. According to Handa (2018), moderation is a process by which teachers communicate their knowledge and expectations concerning standards to improve the reliability of their judgement of students’
learning. Krause et al. (2013) describe moderation as a process incorporating peer reviews to develop uniformity in assessment judgements.
39
Moderation, as defined by Maxwell (2010, p. 457), is a “process for producing consistency across assessors in qualitative judgments of student performance or achievement”. While moderation is important at individual institutions, Krause et al. (2013) highlight the importance of regulating standards of student achievement across HEIs and within common courses to allow inter-institutional judgements of student assessments. Inter-institutional moderation is acknowledged as a valuable approach to address challenges in creating effective and dependable assessments. Likewise, Gourdin et al. (2019), Handa (2018), Newhouse and Tarricone (2016), Teltemann and Jude (2019), and Wyatt-Smith et al. (2017) emphasise the importance of consistent standards that are subject to ongoing discussions, evaluation, and validation within educator communities. As a quality assurance process, moderation ensures that there is consistency amongst judgments of student assessments before they are reported on (Maxwell, 2010c).
In the SA context, assessment practices are governed by the SA Qualifications Authority (SAQA). SAQA describes moderation in assessment as the internal and external verification of the credibility of an assessment system, and that assessments are fair, valid, reliable, and practicable (SAQA, 2015). This description is in line with international definitions. Adams and Anderson (2019, p.13), for instance, describe moderation as an “activity to develop tasks and activities which provide learners with fair and valid opportunities to meet the standards and expectations required of them”.
Despite the disparate foci of the aforementioned studies, the commonality that has emerged is that moderation is a procedure for guaranteeing that awarded marks are impartial and reliable, and that marking guidelines are used consistently (Wyatt-Smith et al., 2010). Drawing on the works of Maxwell (2010), Beutel et al. (2017), Handa (2018), Dessai and Kamat (2018), SAQA (2015), and Vergés Bausili (2018), moderation can be explained as a quality assurance process where the moderator reviews the assessment judgements of the initial assessor and expresses their opinions on the grades, the uniformity of the grading and feedback provided by the assessor to align assessors’ judgements and promote a shared interpretation of the relevant standards.
40
2.5.1.2. Rationale for moderation
Whilst research into moderation practises in secondary schools in SA is not extensive, several international studies claim generalisability in driving efforts at reforming the moderation process and increasing professional and quality standards (Maxwell, 2010; Colbert et al., 2012;
Connolly et al., 2012; Wyatt-Smith et al., 2017). Notably, prevailing research is supportive of a standards-based paradigm, so as to improve quality standards (Connolly et al., 2012; Krause et al., 2013; Booth et al., 2016; Dessai & Kamat, 2018).
Beutel, Adie, and Lloyd (2017, p. 4) refer to “consensus” or social moderation when discussing the use of common frameworks by assessors when making judgements of assessment tasks.
Social moderation is described as a process in which assessors work collaboratively to establish a common understanding of assessment standards (Adams & Anderson, 2019; Connolly et al., 2012; Newhouse & Tarricone, 2016; Williams, 2019). Social moderation can take the form of face-to-face peer moderation or online technologically mediated moderation forums. Teachers engage with moderation processes and a community of moderators, irrespective of where they are physically located, by asynchronous (e.g., email) and synchronous (e.g., video- conferencing) online systems (Newhouse & Tarricone, 2016).
Moderation has largely become a component of the cycle within the assessment process, evolving to integrate validation, monitoring educators’ professional development, and most significantly accountability (Adams & Anderson, 2019). A study of 20 countries by Teltemann and Jude (2019) indicates that the use of external moderation as a method of quality assurance at schools is relatively widespread.
The literature suggests that moderation is a critical process in ensuring the quality and consistency of assessments, and that it plays an important role in establishing a shared understanding of standards among assessors.
2.5.1.3. Moderation in the SA context
Moderation processes in the SA secondary school context are discussed in terms of exit examinations, the nature of school-based assessment and the moderation thereof, and the requirements for digital submissions of assessments.
41
2.5.1.3.1. Exit examinations
In SA schools, Grade 12 is the exit point of the schooling system. As such, the Grade 12 curriculum is regulated by a national assessment and certification process. The Council for the Quality Assurance in General and Further Education and Training (UMALUSI) is a quality assurance body whose responsibility is to ensure that exit examinations are quality assured (Punt, 2010). The key quality assurance processes, as outlined by UMALUSI, are:
Moderation of question papers as well as internal and practical assessments;
Monitoring of the readiness to conduct, administer, and manage the writing and marking of examinations, verification of marking, and the reporting of irregularities;
Discussions of marking guidelines;
Managing concessions; standardisation, statistical moderation, and resulting.
Approval of the release of results (Volmink, 2018).
2.5.1.3.2. Internal School-Based Assessment (SBA)
In addition to exit examinations, internal assessment, also referred to as School Based Assessment (SBA), is fundamental in calculating the final results of students (Independent- Examinations-Board, 2015; Punt, 2010).
The National Curriculum Statement (NCS) promotes an assessment-led method for student learning, supporting the idea of assessment for learning together with an assessment of learning. Continuous assessment of learners during the school year underpins the implementation of the NCS (Independent-Examinations-Board, 2015). Valid, fair, and reliable evidence of learner performance against the requirements of the curriculum is gathered and recorded as evidence via the continuous assessment process (Independent-Examinations- Board, 2015). It is an NSC requirement that the standard and quality of SBAs must be quality assured to ensure uniform standards (Punt, 2010; IEB, 2020).
42
2.5.1.3.3. Processes used at private schools for the moderation of SBA tasks
The IEB moderates SBAs using Regional Moderators (RMs) during the year and a national moderation process at the end of the academic year (Independent-Examinations-Board, 2015).
RMs are allocated specific schools whose tasks are to be moderated. Prior to COVID, the IEB provided three different models that the RM could apply when moderating:
Model 1: A cluster of schools may be asked to send all files to a central school within their