Instructor and Student Perspectives

(1)

ORIGINAL RESEARCH

User Experience Matters: Does One size Fit all? Evaluation of Learning Management Systems

Fatih Demir¹ · Charmaine Bruce‑Kotey¹ · Fahad Alenezi¹

Accepted: 13 April 2021 / Published online: 17 April 2021

Abstract

Learning Management Systems (LMS) are in use for years and are still crucial assets for both teachers and students for teaching and learning. There is a wide variety of LMS available for institutions to administer learning in and out of the classes. However, deciding to pick an LMS to integrate is a critical decision that affects both instructors and students. On the other hand, although there is previous research LMS from students’ perspective, there is a lack of research in the literature from instructors’ standpoint. This research aims to gain a deeper understanding of how in-service teachers, enrolled as students in the study program of instructional technology, use of LMS in terms of user experience and satisfaction of three well-known LMS, including Canvas, Blackboard, and Moodle. The results show a significant difference between the three LMS, Blackboard, Moodle, and Canvas, in terms of users’ subjective satisfaction, ease of use, and functionality.

Keywords LMS · User Experience · Satisfaction · Canvas · Blackboard · Moodle · Instructor

1 Introduction

In the twenty-first century, education encompasses multiple learning environments, including physical locations, virtual classrooms, contexts, and cultures. Advanced technology and digital media make available effective learning anywhere, anytime for students using virtual 3D classrooms (Khlaisang and Songkram 2019). Additionally, technology development allows teachers to facilitate learning and organize in many ways (Kafai et al. 1998).

The research shows that online learning and personalized learning lead to better success (Kößler and Nitzschner 2015) and increase learner engagement (Fake and Dabbagh 2020). A well established and empirically tested online and blended learning environment have been used effectively for exploration in many educational settings (Jan and Vlacho- poulos 2019). Even the advance of technology allows instructions to automatically label the activities across the online learning systems and prompt informed responses (Pillutla

* Fatih Demir [email protected]

1 Northern Illinois University, 1625 W. Lincoln Hwy, Dekalb, IL 60115, USA

(2)

et al. 2020). Thus, designing an attractive learning environment is vital to achieving learning goals and objectives.

A learning management system (LMS) plays a critical role for teachers for the admin- istration, tracking student records, presenting documents, reporting, and delivering course materials. It is one of the core assets of online education that helps instructors administer course content and meet the expected outcomes. LMSs are used for not only fully online courses but hybrid, blended, or flipped classrooms. The research shows that selection of LMS is crucial that stimulates pleasure of using to increase the course outcomes and sug- gest higher education institutions select an LMS that attracts students (Mpungose and Khoza 2020). However, there is a wide variety of LMSs in the market where there are no indicators to compare each other in terms of user satisfaction, usability, and prior features used by the teachers. In this research, we compared three LMS, Canvas, Blackboard, and Moodle, in terms of satisfaction, ease of use, and functionality features on teachers’ standpoint as moderating the course work.

The main goal of this research is to gain a deeper understanding of teachers’ use of LMSs, how they use it, and what extent the LMS is usable, and the most frequently used feature/functionality of it.

In this paper, we discussed the data collection methods and procedures and compare the given LMSs in terms of functionality, ease of use, and user satisfaction. The results were discussed in terms of Expectancy Discomfort Theory and linked to the literature’s most recent results. Considering the limitations and lessons learned from this research expressed to better design research for those who want to conduct future research on this topic.

1.1 Background

The teaching methods and learning methods have been changing and moving rapidly with the advancements of technology. Students want to learn at a distance, and at a lower cost and quicker (Swan et al. 2000; Ya Ni 2013). Educational instructors want to teach more effectively, with flexibility, and conveniently communicate with students. With the introduction of e-learning and distance learning, educational institutions called for software that aids online learning. Learning management systems play a significant role in online learning. A learning management system (LMS) is an application for delivering educational content, assessment, tracking a student’s advancement, and reporting educational courses and training programs (Watson and Watson 2007). The purpose of the LMSs is to help teachers deliver educational material to the students, govern tests, monitor assignments, and track students’ progress. It allows for teaching and learning actions in a unified environment, with no dependence on time or space (Pinã 2013). It focuses on facilitating two- way interaction between teachers and learners and also amongst learners themselves. Many researchers feel that LMSs are an innovative teaching/ learning system and have essential features (Pinã 2013; Watson and Watson 2007). However, there are many indicators in choosing an LMS such as subjective satisfaction, usability, technical specifications, etc.

(Kasim and Khalid 2016). Moreover, subjective satisfaction with a product is an essential indicator of willingness to use a system (Demir et al. 2012).

Several underlying satisfaction theories address the relationship between satisfaction and dissatisfaction and mainly evaluates the users’ post-experience evaluation scores.

Although there is varying meaning of satisfaction such as adequate, pleased or enough, user experience satisfaction views the concerns with users’ satisfaction level with a product (Demir et al. 2017). The user’s satisfaction focuses more on consumers’ attitudes

(3)

concerning the willingness to continue using a product or buying behavior (Kim 1997).

Expectancy Discomfort Theory (EDT) was developed by Richard L. Oliver (1980) which explains users’ satisfaction level, distinguish the expected and perceived product performance and expectation for the product’s future use. The theory predicts high expectations generate greater user satisfaction. Initially, EDT consists of four constructs: expectations, perceived performance, disconfirmation of beliefs, and satisfaction. This study’s key constructs are identifying the most frequently used tools of the LMSs by the teachers and evaluating the perceived post-satisfaction level with the LMS chosen by the participants.

There is much research conducted evaluating LMS from the perspective of the students.

However, there has been a gap in research regarding teachers being the focal point evaluating user experience and satisfaction with the LMSs. There have been studies to evaluate LMSs with students being the users. However, there has been a lack of satisfaction, ease of use, and functionality testing with instructors, making this topic an important one to discuss. Various researchers in education have conducted studies to compare many LMSs such as Blackboard, Moodle, and Sakai (Al-Ajlan 2012; Cavus and Zabadi 2014; Sabine and Beate 2005). These comparisons focus on the interaction, participation, and shared knowledge among learners (Son et al. 2016), their perspectives (Jung 2011), and satisfaction with LMSs (Naveh et al. 2010; Orfanou et al. 2015). Moreover, the results indicate that the "one-size-fits-all" design is not satisfactory for the students on performance feedback and student motivation, particularly the many factors that impact feedback for those using social comparisons (Tesley 2017).

On the other hand, some studies just focus on understanding instructors’ acceptance to use LMSs (Cigdem and Topcu 2015) instead of their satisfaction and the functionalities they deem important. Most studies were also conducted when LMSs were first developed in the early 2000s. LMSs have grown and changed to adapt to users’ (mainly students) satisfaction levels. With technological advances and the developed features found in LMSs, there is an urge for newer studies on this subject matter. This study also aims to seek research on the teachers’ perspectives and user experience with LMSs, and their satisfaction with the given LMS.

1.2 Functionality

An LMS functionality is critical and should have specific features that appeal to both instructors and students. Instructors would have more confidence to teach a course online if they feel an LMS is a suitable environment for learners and easy to use (Almarashdeh 2016). Pinã (2010, 2013) highlighted that LMS had impacted educational instructors in a very positive way. They are now able to provide a more interactive learning experience. It has brought content creation, communication tools, assessment tools, and administrative tools under a single secure platform. Communication is the primary functionality of LMSs. Through conversation and discussion with peers and the teacher, students grow in knowledge, share opinions, practice skills, and ultimately achieve a learning objective (Ya Ni 2013). Both students and teachers value the communication features of LMSs (Lonn and Teasley 2009). Instructors need to give the student public feedback through LMSs and have private and formative feedback functionality necessary for instructors (Rubin et al. 2010). Cavus and Zabadi (2014) found that Moodle’s features allow teachers to easily present materials to their students using file exchange/ internal mail. Sakai does not have this functionality. Moodle has a real-time chat that allows those on the same course to have synchronous discussions. Sakai offers

(4)

wikis, chats, and blogs. However, by some, LMSs have been viewed as tools that are unsuccessful in providing an individual social presence needed for learning (Brady et al.

2010; Minocha 2009). These studies concentrate on students’ views on the functionality of LMSs. However, West et al. (2006) conducted a study that examined faculty mem- bers’ use of Blackboard and found that many face technical challenges and had difficulty using the tools/ features effectively to produce course content for teaching. This ties into a teacher’s satisfaction with LMSs. If a teacher has the knowledge and skill to build a course on an LMS, they would be more in tune with the features and confident to use an LMS platform as a teaching mechanism.

1.3 Satisfaction

The process of discovering whether or not a user is happy with the product is a funda- mental aspect of evaluating a product. Determining subjective satisfaction is an essential aspect of user experience, likelihood in predicting future use, and willingness to continuously use a product (Demir and Parraci 2018). There are various ways to determine user satisfaction to understand users’ satisfaction levels with a product. The most common method of determining the satisfaction level is user satisfaction surveys. These surveys offer a quantitative value of satisfaction level. However, the researcher needs further evaluation to more resounding to understand design problems, likes or dislikes with the product, and expectations of the users. Furthermore, discovering satisfaction scores is crucial to evaluate a product in user experience studies.

The user experience of a device or interface is critical in obtaining a successful product. User satisfaction refers to a user’s acceptability and contentment with consump- tion and interaction with a product (Demir 2011). User satisfaction would be met and increase once the product meets the requirements of the user. The satisfaction level of an LMS is vital and can affect how often and how in-depth a teacher could use the features associated with an LMS. Evaluating teachers’ satisfaction rather than just accept- ing LMSs provides a better understanding of its usability (Almarashdeh 2016). The higher the teachers’ satisfaction level, the more they would use LMSs (Almarashdeh 2016), and in turn, the more engaged students would be, making them more likely to finish their studies. Almarashdeh (2016). Hsu (2012) found that teachers were satisfied with Moodle and its features, indicating that teachers found that Moodle was cost-effective, provided fewer time constraints, and did not need any technical support.

1.4 Usability Testing

As the usage of LMSs rapidly grows in higher education, there becomes a need to assess the satisfaction and effectiveness of LMSs (Almarashdeh 2016). Understanding usability requirements through the users, tasks, and the environment would ultimately help evaluate and improve the product according to the users’ needs. The users’ expectations can be measured through a usability test (Demir et al. 2012). A usability test is critical as it evaluates the functionalities and satisfaction levels of a product in terms of its effectiveness, efficiency, error tolerance, and learnability (Demir 2011). This study evaluates the "satisfaction, ease of use, and functionality" components of usability testing.

(5)

2 Research Questions

1. To what extent do the three LMSs (Blackboard, Moodle, and Canvas) differ in functionality and ease of use?

2. To what extent do the three LMSs (Blackboard, Moodle, and Canvas) differ users’

subjective satisfaction?

3. Which features of the three LMS are used most frequently by the users?

3 Method

The purpose of this study was to explore how graduate-level in-service teachers perceive the satisfaction, ease of use, and functionality of learning management systems (LMSs), including Canvas, Moodle, and Blackboard, and how satisfied they were with them overall. We surveyed the teachers’ perceived perception of functionality, ease of use, and satisfaction level with LMS systems and investigated their subjective satisfaction with tree LMSs, Canvas, Moodle, and Blackboard.

In-service teachers who are also graduate students at a Midwestern state in the United States (N = 98) were offered a course that they should use an LMS, either one of Can- vas, Blackboard, or Moodle design a course and deliver content to their students fully.

The graduate students who were also in-service teachers participated in a survey study and posted a reflection paper distinguishing their experience with the LMS they used.

In addition to the survey, each participant was asked to submit a reflection paper evaluating the given LMS experience. The participants were not limited to their answers.

They had the freedom to write anything they want to address, including the LMS’s pros and cons, challenges, struggles, design problems, and design recommendations.

4 Research Design and Procedure

The participants were asked to use one of three learning management systems, Canvas Moodle, and Blackboard as an instructor in an eight-week-long graduate course enti- tled “Exploring LMSs”. The participants were in-service teachers pursuing a graduate degree in a mid-western state university in the USA. The participants chose one of the three given LMSs freely to be used with the teacher privileges by the first week of the class and developed a course of their preference. At the end of the course, a survey instrument is distributed to obtain participants’ feedback on the satisfaction, ease of use, and functionality of the LMS that they have experienced.

A mix-method approach is applied with a survey instrument to collect quantitative data by the end of the semester. Participants were also asked to submit their reflections that were not associated with the survey study. It was held in a separate session one week later than the survey study. Participants were informed to write a reflection paper without any open or close-ended questions, but the personal experiences with the LMS they choose likes and dislikes by mostly referring to the factors they satisfied or not.

(6)

The data were collected in the Fall 2016 and Spring 2017 semesters. As the regular semesters were comprised of 16 weeks, this 8-week-long course was offered four times, first-eight-week and second-eight-week, during those two semesters.

Although most LMSs have no version numbers released, all three LMS systems were updated in August 2016 and were kept the same version during the data collection time frame for consistency of the results.

4.1 Participants

The researchers surveyed 98 graduate students at a higher learning institution who were also serving as in-service teachers at the public schools at the same time. The participants took a graduate-level Exploring LMSs course practicing with any of three LMS, including Blackboard, Canvas, and Moodle. The participants’ ages ranged between 23 and 59. The gender of the participants was reported as 71 female and 27 male. All participants were enrolled in an MS program in Instructional Design in the academic year of 2016–2017.

All participants reported that they used technology daily for more than five years. Half of the participants (N = 48) indicated that they have teaching experience less than three years, 25% (N = 24) with 3 to 5 years, and 25% (N = 24) more than five years. Although all participants reported that they used at least one of the LMSs as students, none of them used an LMS with a teacher profile and never designed a course before using an LMS. A total of 40 participants (39%) were teaching English, 26 (25%) Math, 18 (18%) Foreign Languages, 8 (8%) Art, and 6 (6%) Social Science in the public schools.

4.2 Materials

4.2.1 Survey Instruments

Qualtrics is an online surveying platform that distributes questionnaires through email, collecting, saving, and reporting data. Qualtrics allows the anonymization of participants’

personal information by eliminating the IP addresses. The higher education institution that this study carried out has the license of Qualtrics for survey studies. Thus, an online survey with Qualtrics was offered to the users to measure functionality and ease of use, satisfaction, and frequency of the tools they used with each LMS. The Qualtrics survey included a Likert scale type questionnaire, varying from totally agree, agree, natural, disagree, and totally disagree, which was delivered via emails for each participant.

The first section of the questionnaire was designed to understand how the LMS func- tioned well and how satisfactory it was. The questions in this section were listed as (1) The interface presents difficulties; (2) The interface is complicated; (3) There is a need to support the use of the interface; (4) There are difficulties in the use to prepare courses; (5) Tools are hardly or not used; (6) Tools do not fulfill its aim; (7) The form of tool use is not evident; (8) Help menu supports my needs; (9) I do not have any problem using LMS in different devices such as desktop, smartphone, tablets and so on.

The second section of the survey was designed to determine the frequency of use of the available tools. The questions include (1) I frequently used syllabus; (2) I frequently used Announcements; (3) I frequently used the Chat room; (4) I frequently used Discussion and Messaging; (5) I frequently used Assignments; (6) I frequently used Grade book; (7) I frequently used Polls/Tests and Quizzes; (8) I frequently used Communication/Collabora- tion Tools (blogs, wikis, conferencing, videos, etc.); (9) If you have any other tool that you

(7)

used frequently, please indicate (open-ended). The second section of the survey questions derived from the course content that the participants must complete as part of the course requirements.

The last section of the survey included the System Usability Scale (SUS), an industry- standard satisfaction survey tool to determine the participants’ subjective satisfaction level (Sauro and Lewis 2016). The SUS was easy to implement and determine the satisfaction widely deployed in surveying the subjective satisfaction. The questions of SUS includes (1) I think that I would like to use this system frequently; (2) I found the system unnecessarily complex; (3) I thought the system was easy to use; (4) I think that I would need the support of a technical person to be able to use this system; (5) I found the various functions in this system were well integrated; (6) I thought there was too much inconsistency in this system;

(7) I would imagine that most people would learn to use this system very quickly; (8) I found the system very cumbersome to use; (9) I felt very confident using the system; (10) I needed to learn many things before I could get going with this system.

To ensure instrument validity, the researchers tested the survey with a total of 20 in-service teachers using any LMS in their workplace. We asked them via email to complete and submit the survey on Qualtrics, an online survey platform, which is fully anonymized without collecting any personal data nor IP addresses. The researchers received 100% feedback and evaluated the 20 responses to measure the validity and reliability. To ensure instrument reliability, the researchers used the test–retest method. The survey instruments tested for validity and reliability before deploying into the study.

The comments from the responses were used to evaluate the survey instruments. Cron- bach’s alpha was computed to assess the internal consistency reliability of the instrument.

The result showed that the values of Cronbach’s alpha for the variable "Functionality and ease of use" determined 0.81, which is above 0.70 for each variable, which also indicates a good internal consistency.

On the other hand, the System Usability Scale Survey (SUS), a ten-item, Likert scale type, is a standardized industry survey deployed in usability studies for years. The SUS is a reliable and valid survey instrument deployed in many studies and is one of the quickest to converge on the correct conclusion if the sample size is limited (Brooke 1996; Sauro and Lewis 2016).

4.2.2 Reflection Paper

In addition to the survey data, the participants posted a reflection paper describing their experience with the given LMS. Since there were no open or close-ended questions or limited writing, participants freely expressed their thoughts about the LMS they have used throughout the semester. The responses were grouped by LMS as likes and dislike about the LMS. The data were used to understand better the survey results and the participant’s perception of the LMS they experienced.

4.3 Implementation

Three learning management systems were offered to the participants to pick any LMSs to design a course. The content must represent an actual course that the participants were currently teaching or going to teach. Along with the instructor privileges with the selected LMS, the participants were given five demo accounts to test/interact as students with their course content. Each participant developed a course gradually from scratch over the 8-weeks-time

(8)

by adding more content and using features. The participants were free to choose the course content in which they would like to develop. No matter which LMS was chosen, the required activities identified as adding a syllabus, creating assignments, modules with text and multimedia content, quizzes, grading quizzes, discussions, announcements, and grade book as well as emailing individual students, emailing all students, and reporting the statistics for each student. During the 8-week implementation time, participants logged in as instructors to develop and manage the LMS. They logged back in as demo students to investigate, interact with the content, and respond to the assignments and discussion.

At the end of the semester, participants were asked to respond to a survey evaluating the functionality, ease of use, satisfaction, and frequently used tools with the LMS they experienced. The participants also posted a reflection paper describing their experience.

For the following semester, some of the same in-service teachers were registered to the same course to use a different LMS. The re-enrolled participants might have chosen the same content that they developed in the previous semester or a different topic, which is totally up to the participants as long as they completed the required activities mentioned above.

5 Results

A total of 104 graduate students who were also in-service teachers in public schools registered for the Exploring LMSs course in the Fall 2016 and Spring 2017 semesters. However, 98 students participated in the study. The most popular LMS was identified as Canvas (N = 51), followed by Moodle (N = 28), Blackboard (N = 19), and Sakai (N = 6). Only six out of 98 participants took the Sakai LMS training course, so this group was excluded from this analysis due to its small number of participants. Therefore, this study only included responses to the surveys of three LMSs, Blackboard, Moodle, and Canvas.

Data were screened for missing values and outliers. The result of the descriptive statistics showed there were no outliers. A problem, however, was that 30% of the data had missing values. Thus, Hot-deck imputation which was implemented to handle the missing data. Hot- deck imputation is a common technique used to replace missing values with observed values from the sample. There were two dependent variables in the research question; functionality and ease of use, and user satisfaction. Each dependent variable was measured by items on an agreement scale (1 = Strongly Disagree, 2 = Disagree, neither disagree nor agree, 4 = Agree, 5 = Strongly agree). Then, a composite score for each variable was computed with an average score of 3.44 (SD = 0.63) for the functionality and ease-of-use, and an average score of 3.42 (SD = 0.60) for the user satisfaction variable. Table 1 shows the mean and standard deviation for each variable for all three LMSs combined.

Prior to conducting the MANOVA, the researchers assessed the MANOVA’ assumption; Equal sample (cell) sizes, normality, multicollinearity, and linearity between dependent variables. Because the number of participants in each group was not equal (Blackboard = 19, Moodle = 28, Canvas = 51), this assumption has been violated. Shapiro–Wilk (SW) was used to assess the normality of the two dependent variables. As shown in Table 2, the result of

Table 1 Mean and Standard Deviation for Dependent variables, N = 98

Variable Mean SD

Functionality and ease of use 3.44 .63

User satisfaction 3.42 .60

(9)

SW was significant for functionality and ease of use, while it was not significant for teachers’

subjective satisfaction, violating the normality assumption for the first variable. A correlation matrix was used to assess multicollinearity.

A value of correlation coefficient greater than 0 indicates a positive association (Chen et al.

2020). The result (Table 3) shows that the correlation coefficient was less than 0.80, which indicated that the multicollinearity assumption had been met (Farrar and Glauber 1967). A matrix scatters plot analysis revealed that there was a linear relationship between the two dependent variables.

1. Q1. To what extent do the three LMSs (Blackboard, Moodle and Canvas) differ in functionality and ease of use?

A multivariate analysis of variance (MANOVA) was conducted to assess whether the three LMS (Blackboard, Moodle, and Canvas) differed in functionality and ease of use and teachers’ subjective satisfaction. Before the analysis of the MANOVA result, the assumption of homogeneity of covariance was assessed. The result of Box’s Test of Equality of covariance matrices was not significant (p > 0.001), which indicated that the assumption of homogeneity of covariance had been met. Because the assumption of normality and the Equal sample (cell) sizes were violated, Pillai’s Trace statistic was interpreted instead of Wilks’ Lambda.

The MANOVA result indicated significant differences between the three LMS in functionality and ease of use and teachers’ subjective satisfaction: F (4,19) = 4.12, p = 0.003, partial η2 = 0.08. The result of the MANOVA was significant, which suggests that the three LMS significantly differed in functionality and ease of use, and user satisfaction. Because the MANOVA result was significant, and the size of cells not equal, the Two-steps Scheffe post hoc test procedure was carried out to identify where the differences apply. In the first step, using an adjusted alpha level (0.05/3) = 0.016, the result of the post hoc test indicated that Moodle and Canvas differed significantly in functionality and ease of use (p < 0.016). Sim- ilarly, Moodle and Canvas differed significantly in user satisfaction (p < 0.016). In the second step, univariate pairwise comparisons were carried out to evaluate only the pairs groups that differed significantly: Moodle and Canvas. Using a Bonferroni adjusted alpha level (0.05/2) = 0.025, the result of the post hoc test indicated that Moodle and Canvas differed significantly in functionality and ease of use (p < 0.025) and in user satisfaction (p < 0.025).

Q2. To what extent do the three LMSs (Blackboard, Moodle, and Canvas) differ teachers’ subjective satisfaction?

Table 2 Results for SW tests on

two dependent variables Variable WSp

Functionality and ease of use .004

User satisfaction .52

Table 3 Results of Correlation

Matrix between two variables User satisfaction

Functionality and ease of use .64

(10)

Although the SUS score itself did not indicate why users were satisfied or not with the system, it is an industry-accepted and widely used tool to measure users’ satisfaction level with the system where Blackboard, Canvas, and Moodle LMS in this case. It is a fact that raw SUS scores were not percentages of user satisfaction. However, the average SUS score is 68, with a standard deviation of 12.5. An average score of 68 is in the 50th percentile represent the score is higher than 50% of all tested systems and applications (Sauro and Lewis 2016).

Adopting Sauro and Lewis’ (2016) curved grading scale -see Table 4- (Lah and Lewis 2016), satisfaction result results indicated that Blackboard SUS score averaged 65.3 (Table 5). To this end, Blackboard’s curved grading score indicated that participants’ satisfaction was at a C level, and there was room for improvement.. The Moodle SUS score averaged 52.5, which the least satisfied LMS among all three. The curved grading scale for Moodle was at D level. The Canvas SUS score was averaged 70.6, which was also at a C level like Blackboard.

6 Q.3 Which features of the three LMS are used most frequently by the teachers?

Teachers using Blackboard (Fig. 1) strongly agree that they use the system to make announcements (32.55% strong agreement and 23.39% agreement). The teachers also agree that the grading book was used frequently (40.19%), with (21.61%) strongly agreeing to this fact. The teachers rarely use the platform for participating in discussions and as a tool for conducting polls/tests and quizzes gnarring strong disagreements (23.92% and 18.53%, respectively).

Table 4 Satisfaction scores LMS SUS Usability Learnability

Blackboard 65.3 66.5 64.7

Moodle 52.5 51.7 52.3

Canvas 70.6 71.1 65.5

Table 5 Curved grading scale

interpretation of SUS scores Letter grade Numerical score range

A + 84.1–100

A 80.8–84.0

A− 78.9–80.7

B + 77.2–78.8

B 74.1–77.1

B- 72.6–74.0

C + 71.1–72.5

C 65.0–71.0

C− 62.7–64.9

D 51.7–62.6

F 0–51. 6

(11)

With Moodle (Fig. 2), teachers also tend to use the platform to post announcement (32.21% strong agreement and 26.31% agreement), post syllabus (18.62% strongly agree, and 23.11% agree), and post assignments (26.35% strongly agree, and 34.88%

agree). The site was rarely used as a chat room (21.93% disagree, and 12.81% strongly disagree), a platform for discussions (40.95% disagree, and 21.53% strongly disagree).

Teachers also use the application to post polls/tests and quizzes (16.24% strongly agree, and 24.28% agree). However, the system was very rarely used as a communication/collaboration tool (8.62% strongly agree, and 12.81% agree).

Likewise, in their experience with Canvas (Fig. 3), teachers tend to post announcements (28.39% strongly agree, and 29.35% agree) and post syllabus (30.26% strongly agree, and 26.91% agree). Canvas was also used for discussions and messaging (23.54% strongly agree, and 12.25 agree), post assignments (26.35% strongly agree, and 29.65% agree) and conduct polls/tests and quizzes (29.11% strongly agree, and 31.18% agree). The system was rarely used for a chat room (19.25% disagree, and 22.05% strongly disagree) and as a communication/collaboration tool (28.71 disagree, and 24.53% strongly disagree).

It clearly seems that the three different LMS were frequently used for posting announcements, posting assignments, posting syllabus, conducting tests and quizzes, and grade book. The systems were least used as communication/collaboration tool. The results do not indicate that other functionalities of the LMSs were never used but rarely.

For instance, many users indicated that they have used once the ‘adding syllabus’ feature but not visited frequently later on. The results indicate the most frequently used feature as the research question addressed.

50.76 56.45

30.99 34.98

61.04 61.8

40.37

27.07

27.4 19.89

31.95 14.72

17.23 11.59

20.63

22.69

21.5 24.15

37.06

50.3

21.73 26.61

39 50.24

BLACKBOARD

Fig. 1 The frequency of use of different features in Blackboard

(12)

41.73 58.52

30.32

19.31

61.23 57.95

40.25

21.43 26.1

24.15

34.94

18.21

18.92 18.36

19.23

16.25

32.17 17.33

34.74

62.48

19.85 23.69

40.25

62.32

MOODLE

Strongly Agree + Agree Neutral Strongly Disagree + Disagree

Fig. 2 The frequency of use of different features in Moodle

57.17 57.74

31.55 35.79

56

27.51

60.29

25.07

24.32 17.26

27.15 19.32

12.32

11.59

13.63

21.69

18.51 25

41.3 44.89 31.68

60.9

26.08

53.24

CANVAS

Strongly Agree + Agree Neutral Strongly Disagree + Disagree

Fig. 3 The frequency of use of different features in Canvas

(13)

7 Discussion

The study participants were offered to choose the LMS they preferred among three to develop a course as the instructor, whether they were currently teaching or planning to teach in the future. Although there could be many factors that might affect the perception of faculty choosing a particular LMS, such as academic rank, experience, and gender (Al Meajel and Sharadgah 2018), this research discusses the survey results as well as self- reported data on the LMS that participant experienced with.

As Expectancy Discomfort Theory (EDT) suggested, the users’ satisfaction level with the product and its performance would be measured post-experience and predicts the likelihood of future-use-of-the-product. The EDT also expects high expectations with a product generate greater user satisfaction. Keeping this in mind, Canvas was the most popular LMS (N = 51) among others and received the highest satisfaction scores. The results showed that the Canvas reached out to significantly higher scores regarding ease of use and user satisfaction. The results validate the EDT, Canvas, as the most preferred LMS with the highest satisfaction scores. The strengths of Canvas were ease-of-use, notification alerts to the students with the changes, analytics, and personalization as stated in the pseudonymized participants’ reflection below:

Jason: As an instructor, I thought building the class and making assignments in Canvas were fairly quick and easy. The module system was easy to use and upload- ing files was quick and efficient. You can color code all of the course you are teach- ing and it would make it very clear and easy to use if teach multiple courses. Next to simply unpublish to start editing the course is a very nice way of working. You know when you have finished you publish and you can alert the students of changes automatically including grades. The course analytics are very helpful. You can see clearly what is working in real-time. You can also cater to each student’s needs with ease, by custom quizzes, assignments and groups.

Heather: From the get-go, I considered it to be my favorite LMS over Blackboard and significantly surpassing Sakai. I like the usability of the system and the interaction.

It is a plus that it shows student progress in each module and that it limits students’

abilities to access content if the instructor decides to do so. It is customizable too. I came to learn that instructors can make exams available to particular students in the class at a given time. This is unlike other LMSs.

Michael: The biggest strengths that I see with canvas are the ability to have such great control over classes and the ability to alter things to your liking. Canvas definitely offers a lot of personalization compared to other LMS programs. Canvas allows instructors to create an interactive course schedule with assignments, quizzes, and other materials. The module organization system allows the instructor to present course content so that students’ progress through the course in a streamlined way, organizing notes, assignments, quizzes, links, videos, and other media in the same space.

The results show a significant difference between the three LMSs, Blackboard, Moodle, and Canvas, in terms of user satisfaction, ease of use, and functionality. Specifically, this study’s findings indicate that Moodle and Canvas differed significantly in functionality and ease of use, and users’ subjective satisfaction. These results support some studies (Cavus & Zabadi, 2014; West et al. 2006; Hsu 2012). Canvas’ features such as

(14)

building the class, making assignments, and uploading files were distinguished from others. These features made more teachers like Canvas, whereas technical difficulties with Blackboard and Moodle. Notably, technical difficulties in any of the three LMS would delay the effective content delivery while making it hard for users when using the tools and features (West et al. 2006).

As addressed in the literature, there is no "one-size-fits-all" solution for the teachers who use LMS to deliver the course content (Tesley 2017; Khlaisang and Song- kram 2019). There were differences in the system usability and learnability in all three LMSs. The results show that the numerical scores were highest in Canvas, which can be attributed to the learners’ participation and shared knowledge affected by communication among students and interaction with the system (Demir et al. 2017; Son et al.

2016). The study results show that announcements were made more on the Blackboard LMS, unlike the other LMSs. Blackboard had easy functionality for the teachers to use when making the announcements because by using Blackboard more students would be reached by the information that is being announced. This was vital, as teachers could easily accomplish their goals and objectives.

As addressed in the EDT and in the literature, the LMS functionality is highly important to achieve learning objectives and meet the expected satisfaction. Although most LMSs could accommodate the major needs such as discussion boards, gradebooks, announcements, etc., the most frequently used tools must be designed in a usable, user- friendly, and easy-to-use manner. Each LMS in the research has its own user interface and hosts the development specifications differently. Therefore, the research was designed to address the most used features/tools by the teachers to understand how and why people were satisfied or dissatisfied with a particular LMS that they used. To this end, the results show that assignments were the most used function with the Moodle LMS, as it was easiest to use. It was found on the homepage and thus, could be easily accessed by the teachers, and there were no technical problems with how it could be used. It was also found that the grading book was the most frequently used functionality across all three LMS. This is because of the importance of the grading feature to in-service teachers. They would like to know more about how the grading aspect works when grading and evaluating students’ work. Moreover, Canvas LMS was used more by the teachers to post the syllabi. This can be attributed to its high system usability scale and learnability, which led to more user satisfaction.

The results also indicate that participants used Moodle’s announcement feature more often than Blackboard and Canvas. This was because they found Moodle’s announcement feature easy-to-use and visible on the homepage or depending on the different course content they created. The reflections submitted by participants show that there were differences among the participants depending on the course content; for instance, a math teacher would use more structured modules where an English teacher prefers to communicate with students through announcements. Teachers who used Canvas posted the syllabus prior to posting assignments. The participants used Moodle’s technical functionality more than other LMSs. The results with Moodle is also in parallel with the results of Badia et al.

(2019) that shows the assignments, quiz, forum, lesson, and external tools were the most frequently used tools by the teachers in Spain which makes the Moodle significantly higher in perceived learning impact. Each participant used the LMS with the teacher privileges to design a course and obtained fake student accounts on the same LMS to access and manipulate the LMS both in teacher and student profiles. As the research was not designed to measure the communication and collaboration features, it may not represent the actual teaching environment’s frequency of use.

(15)

As indicated in the reflection responses, participants found Moodle’s functionality more user-friendly and easy to use. Participants indicated that they want most likely to use Moo- dle because of its cost-effectiveness. This supports Hsu (2012), who noted that some teachers were satisfied with Moodle and its features, which was an indication that Moodle was cost-effective and consumed less time when in use, as well as not needing technical support when being used.

It is important to note that all participants with different majors chose the LMS and course that they developed freely depending on the content they would teach. Thus, different course designs may have impacted the participants’ subjective satisfaction and an impact on choosing different tools to use more frequently. Both UX survey instruments and SUS were subjective rather than objective measurements. On the other hand, some participants completed some instructional design classes previously that also may impact how they perceive satisfaction, ease of use, and functionality of LMSs.

As expectancy discomfort theory suggests, perceived usefulness and confirmation to satisfaction lead significantly to the prospective reuse of a product (Cathy et al. 2005) as well as the developed loyalty to the product (Lin et al. 2009). Although it was not measured within this study, the intention of reuse with the most satisfied LMS would be higher in the future. However, choosing an LMS would be a part of the administrative responsibility of education institutions. Thus, most schools offer only one LMS to the teachers to deliver the course, and teachers may not have an option to choose.

8 Limitations

The participants of this study chose an LMS among three to create a course in their area of expertise. Most participants used only one of the three LMSs, and the content they set up was varying from Spanish to Math and Science education. They did not test all of the three LMS. Neither developed a course. Additionally, they did not get a chance to interact with the actual students but used dummy student accounts to manipulate the course content artificially. For instance, there is no way to test the full functionality of the gradebook until actual students submit assignments. The results would be more reliable if the data were collected through an actual course with real students communicating with the LMS. This simulation is expected to represent the real-world practices; however, the artificial setting of the development and implementation might not represent the teachers’ practices in their actual classroom settings. Therefore, the research setting may limit the generalizability of research findings.

The participants indicated that they used the various LMS as a student during their college education, but they were the first-time course developers using LMSs that never got any LMS course development training before the course. The previous research indicates that the majority of LMS may have long-term implications on users (Kevan and Ryan 2016). The previous experience with LMS may influence the perception of the participants of this study. Some participants have more experience and familiarity with particular LMSs; however, the data was not captured.

Most usability studies include one-on-one sessions to measure time on task, success rates, and observe the interaction. However, the participants were graduate students taking the graduate course in a fully online format. Therefore, the data collection was limited to the survey responses rather than moderated usability testing.

(16)

The LMS was not randomly assigned to the participants because they were expected to use it in their schools. However, all participants applied the same requirements on the LMS that is chosen. A research design with randomly assigned LMS would be a better design for future studies.

The frequency of use data is self-reported, and the term "frequency’ is not defined as the use of a daily or weekly basis. All participants were answered the frequency questions subjectively. The lack of learning analytics generated from the LMSs is a severe limitation of this study, which would be a valuable asset to interpret the results.

The research design and development of the scale and analysis methods carry out some weaknesses, such as missing the evaluation of each LMS’s mean score and measuring functionality and ease of use as one variable.

Another limitation of the study is 30% of the missing values in survey data. The imputation methods as well as the high volume of missing data could have a serious effect on the results.

9 Conclusion

In the education community, there are many LMSs where some educators may have limited or no idea about their features, pros, and cons of each system in terms of user experience, ease-of-use, user-friendliness, and user satisfaction. In this research, three LMSs were compared, including Canvas, Blackboard, and Moodle, in terms of satisfaction, ease of use, and functionality features from teachers’ standpoint as moderating the course work.

This user experience research was implemented with in-service teachers who are also graduate students at a mid-western university in the US required to use an LMS, either Canvas, Blackboard, or Moodle, to design a course and deliver content to their students. A survey study was conducted following a semester-long course inquiring into the experience with the LMS they used. The participants also posted a reflection paper distinguishing their experience in the later phase of the study.

As suggested by the expectancy disconfirmation theory, the post-use satisfaction level for each LMS is higher accordingly with the preferred LMS to use. The subjective satisfaction level is parallel with the participants’ choice of LMS for Canvas, and Blackboard is identified at C level, where Moodle ended up at D level. These results indicate that all LMSs have room for further development.

The course’s goal for each participant’s development was varying depending on the course content, area of mastery, and rationale. However, the participants were required to use particular tools/features, including adding syllabus, creating assignments, modules with text and multimedia content, quizzes, grading quizzes, discussions, announcements, and grade book as well as emailing individual students, emailing all students, and reporting the statistics for each student. The most common tools/features were identified as assignments and grade book for Blackboard users; assignments for Moodle users, and polls, tests, and quizzes for Canvas users. However, the most used tools do not necessarily represent the importance of the tools used over non-used; again, each participant designed a course in his area of expertise with varying purposes. Some of the unused features would be in place when offered e with actual students in school settings. Regarding the preferences in choosing an LMS, there is a significant difference between the three LMSs regarding user satisfaction, ease of use, and functionality. Specifically, this study’s findings indicated

(17)

that Moodle and Canvas differed significantly in functionality and ease of use, and user satisfaction.

Future researchers should consider measuring objective outcomes such as efficiency and effectiveness with pre-defined tasks as well as think-aloud sessions and interviews for a better understanding of how instructors interact with the LMSs. Moreover, this study focused on satisfaction, ease of use, and functionality of the given LMSs. Future researchers should also consider investigating the pedagogical aspects of LMSs while addressing the user experience, design issues, and satisfaction. Additionally, in this research, as a limitation, both LMS and the course topic was chosen by the participants as all participants completed a set of the same series of tasks regardless of the chosen LMS and the topic i.e., STEM, language, history, etc. However, the chosen topic was not tracked, which may indicate the relationship between the course topic and satisfaction rates. This would be a good addition for future researchers.

References

Al Meajel, T. M., & Sharadgah, T. A. (2018). Barriers to using the blackboard system in teaching and learning: Faculty perceptions. Tech Know Learn, 23, 351–366

Al-Ajlan, A. S. (2012). A comparative study of e-learning features, methodologies, tools, and new develop- ments for e-learning. In E. Pontes (Ed.), InTech

Almarashedeh, I. (2016).Sharing instructors’ experience of learning management system: A technology perspective of user satisfaction in distance learning course. Computers in Human Behavior, 63, 249–255.

https:// doi. org/ 10. 1016/j. chb. 2016. 05. 013

Badia, A., Martín, D., & Gómez, M. (2019). Teachers’ perceptions of the use of moodle activities and their learning impact in secondary education. Tech Know Learn, 24, 483–499

Brady, Κ, Holcomb, L., & Smith, B. (2010). The use of alternative social networking sites in higher educational settings: A case study of the e-learning benefits of Ning in education. Journal of Interactive Online Learning, 9(2), 151–170

Brooke, J. (1996). SUS: A quick and dirty usability scale. In P. Jordan, B. A. Weerdmeester, & McClelland, I. L. (Eds.), Usability evaluation in industry (pp. 189–194). London: Taylor and Francis Ltd.

Cathy, S. L., Sheng, W., & Ray, J. T. (2005). Integrating perceived playfulness into expectation-confirmation model for web portal context. Information and Management, 42, 683–693

Cavus, N., & Zabadi, T. (2014). A Comparison of open source learning management systems. Social and Behavioral Sciences, 143, 521–526

Chen, L., Inoue, K., Goda, Y., Okubo, F., Taniguchi, Y., Oi, M., Konomi, S., Ogata, H., & Yamada, M.

(2020). Exploring factors that influence collaborative problem solving awareness in science education.

Tech Know Learn, 25, 337–366

Cigdem, H., & Topcu, A. (2015). Predictors of instructors’ behavioral intention to use a learning management system: a Turkish vocational college example. Computers in Human Behavior, 52, 22–28 Demir, F. (2011). Technology use in community policing: usability evaluation by eye tracking method. LAP

LAMBERT Academic Publishing.

Demir, F., Ahmad, S., Calyam, P., Jiang, D., Huang, R., & Jahnke, J. (2017). A next-generation augmented reality platform for mass casualty incidents (MCI). Journal of Usability Studies, 12(4), 193–214 Demir, F., Karakaya, M., & Tosun, H. (2012). Research methods in usability and interaction design: Evalu-

ations and case studies. (2nd ed.). LAP LAMBERT Academic Publishing.

Demir, F., & Parraci, W. (2018). The more complex the less success in online library services: Evaluat- ing the user experience for international students. Issues and Trends in Educational Technology, 6(2), 50–64

Fake, H., & Dabbagh, N. (2020). Personalized learning within online workforce learning environments:

Exploring implementations, obstacles, opportunities, and perspectives of workforce leaders. Tech Know Learn. https:// doi. org/ 10. 1007/ s10758- 020- 09441-x

Farrar, D., & Glauber, R. (1967). Multicollinearity in regression analysis: the problem revisited. The Review of Economics and Statistics, 49(1), 92–107. https:// doi. org/ 10. 2307/ 19378 87

(18)

Hsu, H. H. (2012). The acceptance of Moodle: An empirical study based on UTAUT. Creative Education, 3, 44–46

Jan, S. K., & Vlachopoulos, P. (2019). Social network analysis: A framework for identifying communities in higher education online learning. Tech Know Learn, 24, 621–639. https:// doi. org/ 10. 1007/

s10758- 018- 9375-y

Jung, I. (2011). The dimensions of e-learning quality: The learner’s perspective. Educational Technology Research and Development, 59(4), 445–464

Kafai, Y., Franke, M., Ching, C., et al. (1998). Game design as an interactive learning environment for fostering students’ and teachers’ mathematical inquiry. International Journal of Computers for Math- ematical Learning, 3, 149–184. https:// doi. org/ 10. 1023/A: 10097 77905 226

Kasim, N. N. M., & Khalid, F. (2016). Choosing the right Learning Management System (lms) for the higher education institution context: a systematic review. International Journal of Emerging Technolo- gies, 11(6), 55–61

Kevan, J. M., & Ryan, P. R. (2016). Experience API: flexible, decentralized and activity-centric data collection. Tech Know Learn, 21, 143–149

Khlaisang, J., & Songkram, N. (2019). Designing a virtual learning environment system for teaching twenty-first century skills to higher education students in ASEAN. Tech Know Learn, 24, 41–63 Kim, S. H. (1997), Modeling residents satisfaction: Comparison of the Francescato and Fishbein-Ajzen

Model Department of Urban and Regional Planning (Unpiblished doctoral dissertation). University of Illinois at Urbana-Champaign.

Kößler, F. J., & Nitzschner, M. M. (2015). Learning online: A comparison of different media types. Tech Know Learn, 20, 133–146

Lah, U., & Lewis, J. R. (2016). How expertise affects a digital-rights-management-sharing application’s usability. IEEE Software, 33(3), 76–82. https:// doi. org/ 10. 1109/ MS. 2015. 104

Lin, C., Tsai, Y. H., & Chiu, C. (2009). Modeling customer loyalty from an integrative perspective of self- determination theory and expectation-confirmation theory. Journal of Business and Psychology, 24, 315–326

Lonn, S., & Teasley, S. D. (2009). Saving time or innovating practice: Investigating perceptions and uses of learning management systems. Computers & Education, 53(3), 686–694

Minocha, S. (2009). Role of social software tools in education: A literature review. Education and Training, 51(516), 353–436

Mpungose, C. B., & Khoza, S. B. (2020). Postgraduate students’ Experiences on the use of moodle and canvas learning management system. Tech Know Learn. https:// doi. org/ 10. 1007/ s10758- 020- 09475-1 Naveh, G., Tubin, D., & Pliskin, N. (2010). Student LMS use and satisfaction in academic institutions: The

organizational perspective. The Internet and Higher Education, 13(3), 127–133

Oliver, R. L. (1980). A cognitive model of the antecedents and consequences of satisfaction decisions. Jour- nal of Marketing Research, 17(11), 460–469

Orfanou, K., Tselios, N., & Katsanos, C. (2015). Perceived usability evaluation of learning management systems: Empirical evaluation of the system usability scale. International Review of Research in Open and Distributed Learning, 16(2), 227–246

Pillutla, V. S., Tawfik, A. A., & Giabbanelli, P. J. (2020). Detecting the depth and progression of learning in massive open online courses by mining discussion data. Tech Know Learn. https:// doi. org/ 10. 1007/

s10758- 020- 09434-w

Pinã, A. A. (2010). An overview of learning management systems. In Y. Kats (Ed.), Learning management systems technologies and software solutions for online teaching: Tools and applications. (pp. 1–19).

IGI Global.

Pinã, A. A. (2013). Learning management systems: A look at the big picture. In Kats, Y. (Ed.), Learning management systems and instructional design: Best practices in online education (pp. 1–19). USA:

IGI Global

Rubin, B., Fernandes, R., Avgerinou, M. D., & Moore, J. (2010). The effect of learning management systems on student and faculty outcomes. Internet and Higher Education, 13(2), 82–23

Sabine, G., & Beate, L., (2005). An evaluation of open source e-learning platforms stressing adaptation issues. In Proceedings of 5th International Conference on Advanced Learning Technologies (pp. 163–

165). DC, USA: Washington.

Sauro, J., Lewis, J. (2016). Quantifying the user experience: Practical statistics for user research. Amster- dam; Waltham, MA: Elsevier/Morgan Kaufmann.

Son, J., Kim, J., Na, H., & Baik, D. (2016). A social learning management system supporting feedback for incorrect answers based on social network services. Journal of Educational Technology & Society, 19(2), 245–257

(19)

Swan, K., Shea, P., Frederickson, E., Pickett, A., Pelz, W., & Maher, G. (2000). Building knowledge building communities: Consistency, contact, and communication in the virtual classroom. Journal of Edu- cational Computing Research, 23(4), 359–383. https:// doi. org/ 10. 2190/ 2FW4G6- HY52- 57P1- PPNE Teasley, S. D. (2017). Student facing dashboards: one size fits all? Tech Know Learn, 22, 377–384

Watson, W. R., & Watson, S. L. (2007). An argument for clarity: What are learning management systems, what are they not, and what should they become? TechTrends, 51(2), 28–34

West, R. E., Waddoups, G., & Graham, C. R. (2006). Understanding the experiences of instructors as they adopt a course management system. Educational Technology Research and Development, 55(1), 1–26 Ya Ni, A. (2013). Comparing the effectiveness of classroom and online learning: Teaching research meth-

ods. Journal of Public Affairs Education, 19(2), 199–215

Publisher’s Note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.