4.2. Research Methodology
4.2.2. Research Design
4.2.2.9. Alignment between DSR and PADRE
Considering the importance of stakeholders in the eModeration context, the inclusion of stakeholders in all cycles of the DSR process was an imperative. To contextualize the problem and to provide a contextualized solution, it was necessary to consider the entry points of stakeholder involvement in the various stages of DSR. The DSR process depicted in Figure 4-1 provides a conceptual representation of the cycles iterated in the DSR process. It was necessary to provide a practical approach to its implementation and to incorporate a reflection of learning at each stage.
ADR is acknowledged as an approach to implementing DSR and, based on the application of PADRE to ADR in involving users in the research process as demonstrated by Haj-Bolouri et al. (2016), the cycles of the PADRE approach were iterated within each cycle of the DSR process (see Figure 4-4) to involve stakeholders and incorporate a reflective process during data collection and evaluation.
The insights gained were documented at each stage of the PADRE process, as recommended by Haj-Bolouri et al. (2016). A core concept of PD (Section 4.2.2.2) is the “genuine participation” of the user in generating design ideas (Cozza et al., 2020, p. 274). Accordingly, PD was confined to data capturing of requirements and design ideas during two PD workshops, and the PADRE approach was used to plan, implement, evaluate, and reflect on the insights gained from the data collected and analyzed during the DSR process.
The components and stages of the PADRE process as applicable to this study are discussed in the following sections. Figures 4-5 to 4-9 have been adapted from the stages of the PADRE process illustrated by Haj-Bolouri et al. (2016).
114
Figure 4-4: Mapping of PADRE to DSR (Drechsler & Hevner, 2016; Haj-Bolouri et al., 2016)
115
Problem formulation
The first component of the PADRE cycle is problem formulation (see Figure 4-5), which occurred in the Change and Impact and Relevance cycles of DSR.
An awareness of the problem was triggered by a literature review pointing to a lack of dedicated eModeration systems in secondary schools. Consequently, during the Problem formulation component, the first stage (Plan) consisted of an articulation for the need to develop an artefact to evaluate an eModeration system (Rajamany, 2020). The second stage (Implement) consisted of a systematic literature review to formulate a problem statement, identify gaps in the literature, gather the requirements of an eModeration system, and determine the user experience constructs of an eModeration system. During this stage, a research article on the requirements of an eModeration system (Rajamany et al., 2020b) and an article on trends regarding the implementation of eModeration systems were published (Rajamany et al., 2021). The third stage (Evaluate) resulted in the synthesis of the eModeration system requirements and user experience constructs from the existing literature. The last stage (Reflect) included the compilation of a list of features of an eModeration system together with user experience constructs from an analysis of technology acceptance models, the field of HCI, and the field of IS success models. A reflection on the identified features and user experience constructs resulted in the identification of features to include in an eModeration system.
Figure 4-5: Problem formulation
116
Building
During the Building component of the PADRE cycle (see Figure 4-6), which occurred during the Design Cycle of DSR, planning consisted of the design of two participatory design workshops to actively engage stakeholders in contributing design requirements and specific functional needs relating to the development of a prototype eModeration system.
Figure 4-6: Building component of PADRE
During the “Implement” stage, participants answered questions on the requirements and design of an eModeration system, based on their knowledge and roles (e.g., teacher, moderator, examiner, assessment body). The articulated requirements and design ideas were documented in a peer reviewed article and implemented through a prototype eModeration system (Rajamany et al., 2022), developed from an analysis of the learning and reflection of the requirements elicited from the systematic literature review as well as the participatory design workshops.
The evaluation stage comprised of an online survey to confirm how well stakeholder requirements were met and to compile additional stakeholder functional specifications. The reflection and learning from the implemented prototype and a peer review of the publication resulted in revised functional specifications based on the user experience and articulated stakeholder needs.
Intervention
117
The Intervention component of the PADRE cycle (see Figure 4-7) occurred during the Design Cycle of DSR. Methods of distributing links to the eModeration prototype to teachers in a private secondary school where the eModeration prototype was implemented were considered during the planning stage.
During the implementation stage, teachers and moderators actively engaged with the eModeration prototype so that the user experience could be evaluated. The evaluation phase comprised of an analysis of the data around stakeholder perceptions of the user experience and functionality of the eModeration prototype. Reflection and learning resulted in conclusions around the quality and user experience of the functionality of the eModeration prototype as well as the effectiveness with which stakeholder needs were addressed. Stakeholders’ evaluation of the eModeration prototype generated learning around the user experience and functionality of the prototype. The revised functional specifications were an outcome of the reflection and learning from stakeholder responses to the survey and a peer review of the publication that documented an analysis of stakeholders’ user experience (Rajamany et al., 2022).
Evaluation
During the evaluation component of the PADRE cycle (see Figure 4-8), which occurred during the Design Cycle of DSR, the planning stage consisted of the development of focus group interviews to corroborate the findings from the online survey.
Figure 4-7: Intervention component of PADRE
118
Figure 4-8: Evaluation component of PADRE
During the Implement stage, stakeholders were interviewed in groups (based on the subjects taught) to elicit domain-specific criteria for the development of an eModeration evaluation framework. The interview results were documented in a journal article and presented for peer review. During the Evaluate stage, the statistical analysis of the survey data, triangulated with an analysis of the focus group interview findings, contributed to the compilation of the components, user experience constructs, and criteria to include in an eModeration evaluation framework. The reflection and learning from the analysis of the data and the peer review of the publication resulted in the development of an initial eModeration evaluation framework.
Formalization of learning
During the Formalization of learning component of the PADRE Evaluation Cycle (see Figure 4-9), which occurred during the Rigor Cycle of DSR, criteria for evaluation of the eModeration framework were extracted from a literature review of criteria commonly used in the evaluation of DSR artefacts was conducted during the planning stage. Semi-structured interview questions were formulated using the criteria for evaluation as a focal point.
119
Figure 4-9: Formalization of learning component of PADRE
During the Implement stage, interviews were conducted with domain experts. During the evaluate stage, responses to the interviews were analyzed to validate the components, criteria, and user experience constructs to include in the final evaluation framework. In collaboration with interviewees, a determination was made that the learning outcome was satisfactory, and that no further iterations were required. The reflection and learning arising from the evaluation of the final criteria resulted in a validated eModeration evaluation framework together with guidelines for the evaluation of an eModeration system, which were documented and communicated in this study.