4.2. Research Methodology
4.2.3. Research Method
4.2.3.3. Data Collection Instruments
An online survey was used as a data collection technique to determine educators’ perceptions of how previously identified constructs contribute to the UX of an eModeration system.
4.2.3.3.1. Questionnaire considered
The following questionnaires were isolated for consideration, based on the usability and user experience focus of this research: System Usability Scale (SUS), User Experience Questionnaire
123
(UEQ), The Usefulness, Satisfaction, and Ease of Use Questionnaire (USE), and the Computer System Usability Questionnaire (CSUQ). A brief overview of each questionnaire follows.
SUS
System Usability Scale (SUS) is a mixed-tone, ten statement questionnaire for subjectively assessing the usability of a system (Lewis & Sauro, 2009; Lewis, 2018). Each question has a five- point scale ranging from Strongly Disagree to Strongly Agree. The SUS is highly reliable (alpha
= 0.91) and can be used by a broad group of usability practitioners to evaluate almost any type of user interface because it is technology agnostic (Bangor et al., 2009). The SUS has excellent reliability (coefficient alpha typically exceeds .90), validity, and sensitivity to a wide variety of independent variables (Sauro & Lewis, 2016).
User Experience Questionnaire UEQ
The User Experience Questionnaire (UEQ) is a widely used questionnaire for measuring users’
subjective impressions of a system (Devy et al., 2017; Hinderks et al., 2018; Schrepp et al., 2017b).
The main goal of the UEQ is a fast and direct measurement of UX (Schrepp et al., 2017a), considering aspects of pragmatic and hedonic quality (Devy et al., 2017; Schrepp et al., 2014).
The questionnaire consists of 26 items grouped into six scales, each of which represents a distinct UX quality aspect (Schrepp et al., 2017a). The reliability (i.e., the consistency of the scales) and validity (i.e., that the scales really measure what they intend to measure) of the UEQ scales were investigated in several usability tests. These studies showed a sufficient reliability of the scales (measured by Cronbach’s Alpha). Additionally, several studies have shown a good construct validity of the scales (Schrepp et al., 2017a).
Usefulness, Satisfaction, and Ease of Use Questionnaire (USE)
The Usefulness, Satisfaction, and Ease of Use Questionnaire (USE) is a valid, reliable instrument and easily accessible questionnaire (Faria et al., 2016) that measures the subjective usability of a system (Lund, 2001). USE can be applied to various scenarios of usability assessment because it is non-proprietary and technology-agnostic (Lund, 2001; Faria et al., 2016; Gao et al., 2018). The immediate consequence of the data to the organization, for instance, “Can users successfully
124
complete the task?” and “What problems did they have?”, means that raw data from even a small sample of typical users could be extremely useful in identifying a need for design changes (Bevan, 1995, p. 120).
The Computer System Usability Questionnaire (CSUQ)
The Computer System Usability Questionnaire (CSUQ) can be used across different user groups and research settings for measuring users’ subjective opinions in a “scenario-based situation”
(Assila et al., 2014, p. 470). CSUQ has been successfully applied in academic and practical contexts, and is relevant and applicable to usability evaluations in various research contexts (Assila et al., 2014). The items produce four scores measuring the user’s overall impression of the system, in addition to the system usefulness, information quality, and interface quality.
4.2.3.3.2. Justification for choice of questionnaires based on study context
The choice of questionnaire depends on the quality aspects to measure. This study investigated UX with a focus on usability. It was therefore necessary to employ questionnaires that focus on both the usability and the UX aspects of an eModeration system. This section presents a rationale for disregarding the SUS questionnaire, followed by a justification for the UEQ and its applicability to the eModeration context. Next, a motivation for using the USE questionnaire is provided. Lastly, a discussion on how questions were adapted from the USE and CSUQ questionnaires in the final questionnaire disseminated to participants is provided.
SUS and CSUQ are widely used questionnaires for assessing perceived usability (Lewis, 2018).
The SUS questionnaire, while easy to use and freely available, has limitations in its scoring method and the results it provides are too general (Devy et al., 2017). For a deep analysis, specific items are needed to measure user experience attributes, making the SUS unsuitable for this study that focuses on measuring user experience with a focus on usability (Santoso et al., 2016). Therefore, the SUS was not considered further.
CSUQ scores are sensitive to independent variables such as experience with the computer system, type of computer used, and range of experience with different computers, and user groups of
125
varying experience (Lewis, 2019). In this study, teachers have varying degrees of experience. The CSUQ would therefore be pertinent.
Over the years, the usability approach has shifted from the testing and evaluation of completed artefacts to incorporating usability evaluation into the design phase of software development. ISO definitions of usability have evolved to accommodate this shift and have been revised to include product quality and usability characteristics (Chung & Sahari, 2015). This move has led to the notion of understanding usability based on user experience, which is the stance taken in this study (see Section 2.5.3.6). The USE questionnaire measures the utilitarian as well as the experiential attributes of a product with a focus on the utilitarian aspect (Chung & Sahari, 2015).
The USE questionnaire analyses and summarizes the usability of a system based on the usefulness, ease of learning, satisfaction, and ease of use constructs of usability (Gao et al., 2018). Anecdotal evidence and a literature review point to the importance of these factors for an eModeration system, which should be used voluntarily, be easy to learn quickly, and be used without negatively impacting on teachers’ time (Van Staden, 2017). Participants are provided with opportunities to make qualitative comments, which is particularly important to provide useful feedback on the UX of the system.
Consistent with Schrepp et al's. (2017a) recommendation that it makes sense to use more than one questionnaire for broader evaluations, this thesis made use of two questionnaires (CSUQ and USE) to evaluate the usability and adopted the UEQ to measure the UX of the prototype eModeration system. In alignment with the theoretical framework (see Figure 3-14), questions from the CSUQ were added to the USE questionnaire to cover information quality, as discussed below.
While the UEQ has the disadvantage of only providing high level detail of the strengths and weaknesses of a product, it is possible to make concrete improvements of the eModeration prototype by using the different scales describing the distinct quality components (Schrepp et al., 2014). The ready availability of the UEQ benchmark and the ease of use of the provided data analysis tool strengthened the motivation for using the UEQ (Santoso et al., 2016; Schrepp &
Thomaschewski, 2020).
126
The time a participant is willing to spend on answering questions for a UX evaluation is typically quite limited (Hinderks et al., 2018). Therefore it is important to use questionnaires with few items that can be answered swiftly. The UEQ consists of 26 items, which requires a total of about 3 to 5 minutes to answer (Hinderks et al., 2018). This is especially convenient, given that personal evidence and literature findings point to teachers being under immense pressure to complete administrative tasks timeously (Chung & Sahari, 2015).
Having established the suitability of the CSUQ, USE, and UEQ questionnaires to the eModeration context, it was necessary to determine their value in terms of the underlying theoretical constructs (see Figure 3-14). In aligning the questions to the theoretical framework and the data that emerged from the literature review and PD workshops, the CSUQ and USE questionnaires were adapted for the purposes of this research, as discussed below.
Quality in use factors were used as a basis for the derivation of constructs in the theoretical framework. In alignment with the theoretical underpinning of this study (see Figure 3-14), information quality from CSUQ was combined with ease of use from the USE questionnaire.
Interface quality from CSUQ was combined with satisfaction from the SUS questionnaire.
Flexibility was not included in the USE questionnaire. Additional questions were thus formulated, based on data that emerged from PD workshops and the literature review. System usefulness constructs were categorized and integrated into satisfaction, efficiency, and learnability. An interrogation of the seminal literature in UX, for instance, Forlizzi & Battarbee (2004), Hassenzahl (2003, 2008), and Hassenzahl and Tractinsky (2006) do not provide evidence of the term user friendly (as discussed in Section 2.5.3.1) being used in the field of HCI. User friendly was thus removed from the questionnaire. The UEQ was used in its entirety to determine the UX of the eModeration prototype.