Notably, no new variables were identified for learnability from the PD workshops. This finding is consistent with the participant demographics (see Table 4-5). All participants were experienced IT teachers and moderators with more than five years of experience. Although participants had initially raised the issue of “teachers’ digital skills”, “user buy-in”, and “technical expertise” as general constraints of implementing an eModeration system, it is evident that this group of users did not classify themselves as lacking technical expertise, as these challenges were not raised again.
In line with the view expressed by Dresch et al. (2015) of the need for DSR to add value to existing theoretical knowledge and improve practical situations in organizations, criteria identified from the PD workshops were mapped to the usability constructs identified in the theoretical framework (see Figure 3-14) to produce new criteria for inclusion in an eModeration evaluation framework.
The criteria from the PD workshops are depicted in Table 5-6.
Table 5-6: Criteria elicited from PD Description
COMPONENT CONSTRUCT CRITERIA
SYSTEM REQUIREMENTS System quality
Audit trail Multi-user authentication
Annotation tools Multi-user technology
Calendar Notifications
Checklist Reliability
Choose moderator Response time
Compatibility Robust hardware specifications
Cross platform Security
Customizable comments Synchronization
Dependability Tracking changes
External communication Tracking deadlines
Flexibility Tracking documents
Infrastructure and Resources Voice-over button Internet connectivity Web-based
Information quality
Accuracy Progress Bar
Centralized data storage Reminders of deadlines
Compatibility Reporting
Data Currency Security of information
File format Timeliness
Instant feedback Versioning
158
Based on their specific requirements, PD workshop participants included criteria that were not evident in the literature reviewed (see Table 5-6). Additional criteria included in the system quality dimension were the facility for tracking documents, changes, and deadlines. Further, criteria not previously identified from literature were included as part of the service quality, user experience and flexibility constructs (indicated in italics in Table 5-7). Participants emphasized the help functionality included in the system, the need for a quick response time, customized notifications, platform independence, and the need for multiple subject integration.
It is significant that these criteria contribute to aspects that domain experts regard as integral to an eModeration system, thus answering Research Question three and adding value to the existing body of knowledge. The specific user experience criteria that add to the net benefits of the use of an eModeration system were identified as customized notifications, an intuitive interface, and the facility for a live video chat with the moderator (see Table 5-7).
Description
COMPONENT CONSTRUCT CRITERIA Service quality
FAQ Technical support
Functional help Ticketing help system Quick response
TASK
Efficiency
Built-in templates Online editing
Cost saving Reduced printing
Database of comments Time saving Environmentally friendly Live video chat Integration with cloud
storage Voice notes
Effectiveness Automatic updates Organized file structure
Collaboration Shared folders
Satisfaction Ease of use Task performance
Satisfaction with specific
functions, e.g., sharing User friendly Flexibility Platform independent Multiple roles
Multiple subject integration Learnability N/A
USER Hedonic qualities Customized notifications Customized settings/user preferences Intuitive interface
159
During the group design discussions, two groups indicated a need for the moderator to “comment on the document”; however, participants did not elaborate on this functionality. While participants discussed functionality to edit documents online and addition of an MS Word plugin to enable them to do so, the inclusion of an annotation tool functionality in the final design was omitted from all discussions. Extant literature provides a convincing argument for the inclusion of annotation tools, resulting in annotation tools being previously included as a user requirement for an eModeration system (see Figure 5-2).
Various statements made by participants indicate the significance of the need to “track changes”,
“track documents”, “track deadlines”, and to “generate a history of the proof of moderation”.
These were coded as tracking and were included as criteria in the Systems Quality construct identified in the theoretical framework (see Figure 3-14). As depicted in Table 5-7, these are context-specific criteria that stakeholders believe should be incorporated in a dedicated eModeration system.
Table 5-7: Comparison pre and post PD Criteria COMPARISON OF PRE AND POST PD CRITERIA
Description Identified from
literature review Identified from PD workshops
COMPONENT CONSTRUCT CRITERIA
System Requirements
System quality Audit trail
Availability
Calendar
Capability
Checklist
Choose moderator
Compatibility
Complexity
Cross platform
Dependability
External communication
Flexibility
Infrastructure and resources
Multi-user authentication
Multi-user technology
Multiple screens
Notifications
OCR
Organized file structure
Reliability
Response time
160
COMPARISON OF PRE AND POST PD CRITERIA
Description Identified from
literature review Identified from PD workshops
COMPONENT CONSTRUCT CRITERIA
Robust hardware specifications
Security
Synchronization
Tracking changes
Web-based
Information
quality Accuracy
Centralized data storage
Completeness
Data currency
Format
Instant feedback
Legibility
Output quality
Progress bar
Reminders of deadlines
Reporting
Security of information
Timeliness
Versioning
Service quality FAQ
Functional help
Technical support
Ticketing help system
TASK REQUIREMENTS
Efficiency Annotation tools
Built-in templates
Database of comments
Cost saving
Integration with cloud storage
Online editing
Reduced printing
Time saving
Tracking deadlines
Effectiveness Availability
Automatic updates
Capability
Collaboration
Dependability
Progress bar
Shared folders
Tracking documents
Voice-over button
Satisfaction Usefulness
Ease of use
Satisfaction with specific functions, e.g., sharing
Task performance
User friendly
Flexibility Platform independent
161
COMPARISON OF PRE AND POST PD CRITERIA
Description Identified from
literature review Identified from PD workshops
COMPONENT CONSTRUCT CRITERIA
Multiple roles
Multiple subject integration
Learnability Digital literacy
Self-efficacy
Training and experience
USER REQUIREMENTS
Hedonic
qualities Aesthetics
Customized notifications
Intuitive interface
Customized settings
Live video chat