This deficit provides the rationale for a study of the components of a framework for evaluating the user experience of an eModeration system. User experience requirements for digital moderation systems in South Africa: Using participatory design in design science research.
Rationale
UX is typically measured using constructs related to a system's usability (Hussain et al., 2018). Confidence in the validity and reliability of student results is an important issue associated with the e-evaluation process (Durcheva et al., 2019).
Research Problem
Furthermore, a lack of theorizing about the components of a usable eModeration system in the high school environment is evident (see table 1-1). The focus of their study was on the user experience of an eModeration system in the context of higher education rather than high school.
Research questions
These arguments point to a gap in current knowledge and support the need to investigate a user experience framework for evaluating eModeration systems in secondary schools. The problem formulation can thus be formulated as: There is currently no evidence-based framework that can be used to evaluate the user experience of an eModeration system in secondary schools.
Theoretical Grounding of the Research
- Literature review approach
- Philosophical paradigms
- Theoretical framework
- Technology Acceptance Models
- IS Success Models
- Human-Computer Interaction (HCI)
- Components included in the theoretical framework for this study
SLRs provide new perspectives to the topic under review, thus generating new knowledge (Pérez-Sanagustín et al., 2017). The researcher's assumptions shape how the research questions are understood, the data collection methods used and how the results of the study are interpreted (Saunders et al., 2019).
Research design
Data collection and data analysis methods
The choice between qualitative and quantitative research methods depends on the nature of the research problem (Creswell, 2014). This study used an exploratory sequential approach combining qualitative and quantitative data (Cresswell, 2014), starting with a qualitative phase to explore participants' opinions (see section 4.2.2.2). section 5.3) was introduced in the second, quantitative phase, where the qualitative phase was used to refine the prototype and determine suitable instruments for use in the quantitative phase (Creswell,.
- Research flow process
- Contributions
- Theoretical Contributions
- Practical Contribution
- Research planning
- Thesis Layout
- Conclusion
An analysis of the UX evaluation criteria (version 2) was used to extract domain-specific UX evaluation criteria that guided the development of an initial UX evaluation framework (version 1) and modified pre- and post-use guidelines (version 2). Finally, the pre- and post-use criteria, extrapolated from an analysis of the findings, formed the basis for the development of guidelines for the evaluation of an eModeration system.
Rationale for a Literature Review
The importance of conducting a comprehensive literature review in research is widely recognized by scholars such as Pickering and Byrne (2014), Boell and Cecez-Kecmanovic (2015b), and Barn, Barat, and Clark (2017). Assessing the strengths and limitations of the existing literature is a crucial aspect of a thorough literature review.
Systematic Literature Review explained
The use of predefined and explicit methods in conducting SLR helps to ensure the integrity, impartiality and transparency of the literature review process, which facilitates the replication of the process (Barn et al., 2017; Boell & Cecez-Kecmanovic, 2015a, 2015b). It is important that the literature review clearly records the steps used in its development.
Literature Review Protocol
The modified assessment can then be uploaded and digitally accessed by the original assessor (Van Staden et al., 2017). The search process becomes complex due to the combination of the need for broad reading and the extensive IT-related terminology (Pretto & Curró, 2017).
Begin from the general to the specific
In contrast, Van Staden et al. 2017), describes eModeration as using an online tool for digital quality assurance. The summaries of the articles provided a basis for eliminating those that did not meet the exclusion criteria (see Table 2-1).
Scan and survey all technology related journals and books
However, searches in Scopus, Inspec and WoS using the keywords: "cloud native digital moderation" yielded no results. For example, one related article was found from the reference lists of the four initial articles in this study using the search phrase “user experience rating moderation systems”.
Refine the selection
A Google search using the keywords “digital moderation project+New Zealand Qualifications Authority” produced newspaper articles about a cloud-based digital moderation project in New Zealand schools. Full articles were read and a flagging tool was used to highlight the relevance and value of the content.
Read the selected sources with careful attention to IS terms used
Articles about systems that were not specifically focused on digital tools in education were eliminated. Relevant articles were added to favorites in Mendeley for easy identification during the literature review.
Cull and sort the selected texts
Literature Review
- Literature Review of Moderation
- What is moderation?
- Rationale for moderation
- Moderation in the SA context
The flow of information during the stages of the literature review process is shown in Figure 2-1. This section describes the literature review related to rating moderation, beginning with an overview of what moderation is and a brief discussion of the importance of moderation, followed by a discussion of eModeration.
A cluster of schools may be asked to send all files to a central school within their region. The moderator then travels to that school to moderate files
The IEB moderates the SBA using regional moderators (RMs) during the year and a national moderation process at the end of the academic year (Independent-Examinations-Board, 2015).
The moderator may ask schools to courier files to their school or a central venue
Purpose of an eModeration System
The shift to digital assessments has highlighted the need for robust moderation processes that can adapt to the challenges of technology-mediated assessments. Despite certain limitations, such as the need for reliable and stable internet connection, the shift to digital assessments has the potential to improve the effectiveness and efficiency of the moderation process (Van Staden, 2017).
Existing studies and ICT systems relevant to eModeration
The focus was on eliciting teacher requirements to develop an online eModeration system (New Zealand Qualifications Authority, 2016b). The focus of this study was on a framework for evaluating the user experience of the eModerate system.
Requirements of an eModeration system
User requests are aspects of the eModeration system that have a direct impact on users. System requirements refer to the basic features of the eModeration system that are critical to its effective implementation.
Review of Human-Computer Interaction
Human-Computer Interaction
- Usability
- Definitions of usability
- Usability as Quality in Use
- User Experience
- Definitions of user experience
- Relationship between usability and user experience
A holistic measure for evaluating an IS should consider the impact and quality of the system (Gable et al., 2008). Hedonic attributes refer to the system's ability to meet the user's non-task needs (Hassan & Galal-Edeen, 2017).
UX criteria extracted from the literature
Conclusion
The following chapter compares the different technology acceptance models and maps the IS success dimensions to the identified eModeration requirements to determine their applicability in the eModeration context. Chapter Two outlined a literature review of moderation and the HCI field and outlined the requirements, components and criteria for an eModeration system.
DSR artefacts
The purpose of this chapter is to present the theoretical basis that underpins the development of this study. Models establish a connection between a problem and its solution, enabling exploration of the consequences of design decisions in the real world, thereby improving understanding of both the problem and its solution (March . & Smith, 1995; Hevner et al., 2004; Cleven et al., 2009).
Frameworks and models
Constructs define classification systems that outline problems and solutions, thereby enabling the creation of models (Hevner et al., 2004; Cleven, Gubler, & Hüner, 2009; Visser, 2017). In a user-centered design process, it is essential to consider teachers' attitudes and perspectives when designing an ICT tool (Schulz et al., 2015).
Technology Acceptance Models
- UTAUT
- Task Technology Fit Model
- HOT-Fit Framework
- Summary of applicability of technology acceptance models to identified eModeration usability constructs eModeration usability constructs
TAM focuses only on user perceptions of the ease of use of technology (Mafunda et al., 2016). The Hot-Fit framework, which involves a fit between human, organization and technology, is therefore important in evaluating the ease of use of an eModeration system (Bain et al., 2020).
Information system success
In addition, System Importance and Utility were added to the D&M Success Model and Organizational Impact and Individual Impact were removed (Seddon & Kiew, 1996). The D&M model is relevant in the evaluation of an eModeration system as it takes into account the interdependence of factors that contribute to the success of an IS.
User experience frameworks
- Framework 1: User Experience Research Framework
- Framework 2: M-health User Experience Framework
- Framework 3: User Evaluation Framework for eModeration
- Applicability of existing UX frameworks to eModeration
- User experience constructs
- Definition of UX for this study
User motivation yes The usefulness of the eModeration system will determine how motivated the user is to use the system. Temporal context refers to the time dedicated to using the given context constraints system.
Theoretical Framework
- Applicability of theories to eModeration
- IS constructs included in the theoretical framework
Service quality refers to the quality of support users receive from IT support (Petter et al., 2013; TryWindy et al., 2020). Service quality affects the use and satisfaction of users of the eModeration system (Pitt et al., 1995).
Usability constructs pertinent to an eModeration system
- Theoretical framework for the evaluation of an eModeration system
- Revised UX requirements for an eModeration system based on theoretical framework framework
The physical technology and task environment are contextual factors that influence the user experience (Law et al., 2009). When designing tasks for quality in use (e.g. for evaluating efficiency and effectiveness in use), content and functions are embedded in the task design itself rather than as features of the software, as user experience relates to the use of a system (Lew et al., 2010).
Conclusion
The identified criteria are used as a basis for developing an evaluation framework for an eModeration system in Chapter Six. Findings from different streams of literature were synthesized to propose a theoretical framework within which to position this research into the UX evaluation of an eModeration system.
Research Methodology
- Philosophical World View
- What is a paradigm
- Comparison of paradigms
- Interpretivism vs Pragmatism
- Pragmatism and interpretivism aligned to this study
- Research Design
- Design Science Research
- Participatory Design
- Alignment between PD and DSR
- Mapping of PD to DSR
- Positioning the user within DSR
- Action Design Research
- Participatory Action Design Research
- Participatory Action Design Research approach (PADRE)
- Alignment between DSR and PADRE
- Research Method
- Data collection
- Participatory Design workshops
- Data Collection Instruments
This study requires an understanding of the perspectives of educators and moderators in evaluating the UX of an eModeration system. The Intervention component of the PADRE cycle (see Figure 4-7) occurred during the Design Cycle of DSR.
Data Analysis
UEQ was used in its entirety to define the UX of the eModeration prototype. A quantitative data analysis complemented the qualitative data analysis in this study to enable data triangulation to yield more relevant and unbiased results.
Research quality and rigour
- Reliability of quantitative data
- Reliability of measuring instrument
- Normality of measuring instrument
- Validity and reliability of qualitative data
Peer debriefing: feedback was sought from the research supervisors to test insights based on the analysis of the data. The integrity of the findings lies in the data and must represent the "situation being researched".
Ethical Clearance
Adequacy: the researcher must compile the data, analysis and findings in such a way that the reader is able to validate the adequacy of the findings (Nowell et al., 2017). Referential adequacy was tested by reviewing the raw data and comparing it to the developed themes to ensure that all conclusions were supported by the data (Nowell et al., 2017).
Conclusion
Authenticity was ensured by including a range of voices, along with dissenting views, in the participants' own words to further explain the researcher's interpretations (Fossey et al., 2002). A comparison of the criteria for an eModeration system that were derived from the literature and the criteria obtained from the PD workshops is presented in Table 5-7.
Descriptive analysis
Activity Five: Participants were each allocated a total amount of R200.00 and were encouraged to indicate how much they valued each design idea. The following sections discuss the collective results of the activities completed during both workshops, starting with the responses to the user feedback forms, the ranked themes, and the brainstorming session that resulted in the completed web of ideas.
Analysis of qualitative results
- Thematic Analysis
Accordingly, the semantic content of the data was examined using the six phases of TA as postulated by Braun and Clarke (2006), namely,. Sheet three of the spreadsheet (see Figure 5-3) contains a transcript of the data from each group's idea sites.
One of the first steps in qualitative data analysis is to prepare and organize the data for TA (Lester et al., 2020). ATLAS.ti® is a widely used tool that streamlines the qualitative data analysis process to enable deeper analysis of the data (Castleberry & Nolen, 2018).
Benefits of eModeration Cost savings Cost savings: printing Cost savings: paper Cost savings: transportation Communication Communication: messages. Requirements and functions were included as function requirements and user experience requirements respectively (see table 5-4).
Possible sub-themes and codes for the overarching theme of Benefits of eModeration are shown in Table 5-4. Consistent with the literature, a decision was made to consider requirements as "must-haves" (University-of-Colorado, 2021, p. 170) and features as additional functions that would enhance the user experience.
Results from group activities: Workshop One
- Current moderation practices
- Likes, dislikes and challenges
- Requirements of an eModeration system
- Identification of stakeholders
The dominant themes relate to facilitation of the moderation process, communication, and hardware and system requirements. Notably, three of the four participants in Group Two cited the need for immediate or “instant” feedback as a reason for using an eModeration system.
Design Ideas
Two participants extended the features for automatic changes in the mark allocation in the memorandum whenever the mark allocation in the question paper changes and the self-generation of analysis grids. Features related to the creation of an assessment, such as automatic grade distribution changes and self-generation of analysis networks, were not included in the final analysis.
Final Design Decisions
The moderator view will include a list of the schools to which the moderator has been assigned for moderating assessments. The moderator screen included the facility to upload subject-specific documents, templates for the regional moderator to fill in, and functionality to download a report.
Qualitative findings from PD workshops
Additional criteria included in the system quality dimension were the ability to track documents, changes and deadlines. Participants highlighted the help functionality built into the system, the need for fast response time, personalized notifications, platform independence, and the need for multi-entity integration.
Conclusion
Chapter Five presented the qualitative findings from the PD workshops and concluded with a table comparing the pre- and post-PD criteria (Table 5-7) used to create the artifact instances. This chapter presents the results of an online survey conducted after the participants' interaction with the prototype of the eModeration system.
Participant demographics
Of the 40 respondents, 25% participated in their role as an external moderator, 12.5% as an internal moderator, 2.5% as a regional moderator, and 82.5% as a teacher, which is representative of the community.
Survey response rate
Quantitative results from survey
- Effectiveness
- Efficiency
- Satisfaction
- Learnability
- Flexibility
- Information quality
- Relationship between constructs
- User experience
Notably, none of the participants agreed that it was easy to remember how to use the system. Respondents were neutral (37.5%) about the overall flexibility of the eModeration system, with 55% agreeing that the system was flexible.
Qualitative results from online survey
In contrast to some of the responses regarding the selection of roles being a challenge (in question 19.1), some participants indicated that they appreciated being able to choose the appropriate role, as shown in the comments "The drop-down menu for easy selection of user type" and "I especially liked the file organization and the way a moderator could be selected". Responses were categorized as ease of learning, ease of use, ease of access, and ease of file transfers, as evidenced by the comments “The transfer and access to documents is very smooth” and “The ease and speed of document transfer.
Post survey focus group interviews
- Focus Group demographics
- Focus group results
- Ease of use
- Flexibility
- Learnability
- Factors influencing adoption decisions
Many of the original respondents were unable to participate in the focus group interviews due to time constraints. The hedonistic qualities are discussed in terms of aesthetics, collaboration, and the novelty and stimulation of the eModeration system.
