88
System quality, information quality, and service quality have a direct impact on the usability of a system (see Figure 3-7). Lew et al. (2010) suggest that information quality, which can be evaluated as an internal or external quality characteristic, should be added as a quality in use characteristic. When designing tasks for quality in use, content and functions are embedded in the task design itself rather than as attributes of the software. Accordingly, in this study, information quality was included as an aspect of system requirements (external quality characteristic) and as part of the task design (internal quality characteristic), which will affect the quality in use (Figure 3-14).
Section 3.8. presents the selection of constructs for the evaluation of an eModeration system
89
using Nielsen's (1993, 1996) description of usability as a measure of how well users can use the system’s functionality (see Table 2-4), Errors is not selected as a construct for further discussion. Reliability incorporates errors and is included as part of system quality (see Table 3-8). Although Flexibility is common to only two of the definitions of Usability in Table 2-4, empirical evidence indicates Flexibility as part of system quality in IS Success models (Petter et al., 2013). Additionally, data from the participatory design workshops indicate the importance of including Flexibility as a construct for an eModeration system.
The usability of an eModeration system can be specified and tested using a collection of core functional constructs reviewed in the literature. These constructs are discussed with reference to how effectively the user’s goals are met (effectiveness); how efficient the eModeration system is in allowing expert users to attain a high level of productivity (efficiency); how easy it is to learn to use the eModeration system (learnability); how pleasant it is to use or subjectively satisfy users of the eModeration system (satisfaction); and how the eModeration system can be adapted for different users, devices, and tasks (flexibility) (Bevan, 2001; Green
& Pearson, 2006; ISO-IEC, 2018; ISO-IEC 25010, 2011; Lee, 1999; Nielsen, 1993, 1996).
Effectiveness can be explained in terms of whether the system does what it is intended to do (Atoum et al., 2014), and is described as the “the extent to which the system adds to the achievement of organizational goals and benefits” (Santa, MacDonald and Ferrer; 2019, p. 41).
Within the context of this study, effectiveness refers to the degree to which users achieve specified goals as measured against the criteria of correctness and completeness (Ferreira et al., 2020). Effectiveness is a critical component in determining the usability of an IS (ISO-IEC, 2018; ISO-IEC 25010, 2011), focussing on completing activities and the quality of the task outcome (ISO-IEC, 2018; Santa et al., 2019).
Efficiency is explained as the resources expended in relation to the accuracy and completeness with which users achieve their goals (ISO-IEC 25010, 2011), and refers to the comparison of what is actually performed with what can be achieved with the same consumption of resources (Atoum et al., 2014). Efficiency is a measure of how productive a system is without any waste of time, financial, and/or human resources (Centobelli et al., 2019). The focus is on performing tasks optimally and satisfying the needs of various types of users (Cioloca et al., 2013; Rai &
90
Selnes, 2019; Santa et al., 2019). Due to its significant impact, user efficiency is often regarded as the most important attribute of usability (Nielsen, 1993).
Satisfaction is defined as the “degree to which user needs are satisfied by using a product or system in a specified context of use” (Ferreira et al., 2020, p. 1; ISO-IEC 25010, 2011). The updated version of the ISO standards defines satisfaction as the “extent to which the user's physical, cognitive and emotional responses that result from the use of a system, product or service meet the user’s needs and expectations” (ISO-IEC, 2018). In contrast to the former definition, which emphasizes the context of use as important in determining the satisfaction that the user derives from using the product, the latter definition highlights the user experience that arises from the ability of the system to meet the user’s needs. Satisfaction is an important measure in evaluating the effectiveness of an eModeration system as users’ cognitive processes, together with emotional and affective elements, influence the intention to use technology (Taherdoost, 2018; Rai & Selnes, 2019).
Learnability refers to the degree of ease with which a user can interact with a new system. As a usability attribute, a system is regarded as easy to learn when users can progress quite quickly from not knowing the system to completing a task (Nielsen, 1993, 1996). The extent of use is explained as the user’s capability of becoming proficient with an application (Weichbroth 2020, p. 55569). Learnability in use is defined as the degree to which specified users can learn efficiently and effectively while achieving specified goals in a specified context of use. This characteristic has become part of the quality in use model to account for the learning process and the importance of context of use during learning (Bevan et al., 2016; Lew et al., 2010;
Weichbroth, 2020). The ability to quickly and easily use an eModeration system will positively influence the user’s experience of the system (Van Staden, 2017). Accordingly, learnability has been included as a quality in use construct (see Figure 3-14) and use was incorporated into learnability.
Flexibility is a quality in use characteristic to evaluate the extent to which a system can be adapted for different types of users, tasks, and contexts to enable users to achieve their goals in contexts “beyond those initially specified in the requirements for the system” (Bevan et al., 2016, p. 275). Flexibility relates to the context of use (Bevan et al., 2016).
91
Based on the attributes of analysis depicted in Table 3-8, the incorporation of constructs related to the IS success models in formulating a theoretical framework for this thesis are discussed in the following paragraphs.
The individual impact of an eModeration system contributes to the overall UX of the individual. The organizational impact is linked with effectiveness and efficiency within the larger context of quality in use. Benefits such as time and cost savings will result from the effectiveness and efficiency with which teachers and moderators complete their tasks (Law et al., 2008; Van Staden, 2017). Thus, net benefit was incorporated into effectiveness and efficiency in the theoretical framework.
Many IT systems are socially constructed and their implementation typically entails considerable learning for them to be implemented and adopted by users (Korpelainen, 2011).
Considering learnability solely as a quality of the system does not include differing use contexts, nor does it incorporate an evaluation of the learning process. Minimizing the amount of time required to learn to use a system depends entirely on who the user is and on the tasks that the user is attempting to complete (Lew et al., 2010). Tools with more functionalities will possibly be harder to use. As users acquire more experience with a system, it becomes easier to use (Ajibade, 2018). Similarly, learnability becomes less important when individuals develop behavioural intentions to use the system once they are accustomed to do so (Venkatesh
& Bala, 2008). Training improves perceived usability, thus resulting in performance impacts (Farhan et al., 2019). It is necessary to recognize the role of training in the adoption of IT systems, as insufficient learning can limit the adoption and use of a potentially valuable system (Lew et al., 2010). The user and task requirements and their effect on learnability are therefore important in evaluating the UX of an eModeration system.
Given that usability is related to satisfying the do goals of the end user (Lew et al., 2010), satisfaction is positioned as part of the usability constructs contributing to the overall UX (see Figure 3-14). Users’ emotions, physical and psychological responses, internal and physical state resulting from previous experiences, and user skills are included as part of the ISO definition (see Table 2-6) of UX (Díaz-Oreiro et al., 2019).
The pragmatist perspective advocates a detailed analysis of UX to offer rich insights into interactions between the user and the system (Law et al., 2007), which aligns with the pragmatic
92
approach underpinning this study. Additionally, in line with Lund's (2001) observation that, while subjective reactions to the usability of a system are most closely bound to user behaviour and purchase decisions, there is a tendency to neglect these aspects of user experience in favour of performance measures; hedonic qualities have been included in the theoretical framework.
3.8.1. Theoretical framework for the evaluation of an eModeration system
The traditional usability framework focuses primarily on user cognition and user performance in human-technology interactions (Law et al., 2009). In contrast, UX emphasizes the non- utilitarian aspects of such exchanges. The focus is thus shifted to user affect, sensation, and meaning, together with the value of such interaction in everyday contexts (Law et al., 2009).
Law et al. (2014) argue that theoretical frameworks should examine the relationship between affect, action, and cognition. Hedonic qualities such as aesthetics and digital literacy have therefore been included in the theoretical framework. The physical technology and task environment are contextual factors that affect the user experience (Law et al., 2009). When designing tasks for quality in use (e.g., for evaluating efficiency and effectiveness in use), content and functions are embedded in the task design itself rather than as attributes of the software, as user experience is related to the usage of a system (Lew et al., 2010).
93
Figure 3-14: Theoretical framework for eModeration
As depicted in Figure 3-14, and explained in Section 2.5.3.6, usability is subsumed by UX.
The following section describes the refinement of the UX requirements outlined in Chapter Two.
3.8.2. Revised UX requirements for an eModeration system based on theoretical framework
The broad constructs pertinent to the evaluation of the UX of an eModeration system, as identified in the theoretical framework, provides a basis for refining the UX requirements previously identified from the literature into criteria for an eModeration system.
The user, task, and system components (see Table 2-9) extrapolated from literature and the requirements (see Table 2-2) to evaluate a potential system were refined and categorized into specific usability and UX criteria, as depicted in Table 3-9.
94
Table 3-9: Refined criteria for evaluating the UX of an eModeration System
COMPONENT CONSTRUCT CRITERIA
System
System quality
Audit trail Multi-user technology
Availability Notifications
Compatibility Organized file structure
Complexity Reliability
Cross platform Robust hardware
External communication Security
Flexibility Synchronization
Infrastructure and
resources Web-based
Information quality
Accuracy Legibility
Compatibility Output quality
Completeness Reporting
Data currency Security of information
Format Timeliness
Service quality Technical support
Task
Efficiency
Availability Quick response
Cost saving Reduced printing
Database of comments Time saving Effectiveness
Capability Usefulness
Dependability Voice-over button
Productivity
User Satisfaction Satisfaction with specific functions
Learnability Ease of use Training and experience
UX Hedonic qualities Aesthetics Self-efficacy
Digital literacy
The identified criteria are used as a basis for the development of an evaluation framework for an eModeration system in Chapter Six.