Journal of Information Technology and Computer Science Volume 7, Number 3, December 2022, pp. 196-208
Journal Homepage: www.jitecs.ub.ac.id
User Experience in Learning Management System:
Edmodo versus Google Classroom
Hanifah Muslimah Az-zahra*1, Dini Nurhayati2, Admaja Dwi Herlambang3
1,2,3Brawijaya University, Malang
1[email protected], 2[email protected], 3[email protected]
*Corresponding Author
Received 01 February 2022; accepted 18 December 2022 Abstract. Nowadays, Learning Management System (LMS) is widely used to support the learning process. LMS becomes a medium of interaction between students and teachers. Poor usability and user experience (UX) of an LMS can make students spend more time learning the system than learning the educational content. Therefore, usability and user experience evaluation are important for e- learning. Edmodo and Google Classroom are two widely used LMS. This study involved students as inspectors to compare the user experience of Edmodo versus Google Classroom using Technique for User Experience Evaluation in E-learning (TUXEL). Inspectors divided into two groups: one group inspected Edmodo and another group inspected Google Classroom. After completing the inspection, the inspectors were asked to fill three questionnaires: usability inspection, general LMS / pedagogical usability inspection, and user experience. The data then processed and analyzed per category per dimension. The results of the overall general usability and pedagogical usability inspections show fewer problematic aspects in Edmodo (22 items) than Google Classroom (30 items). However, Google Classroom seems to be preferred by students because it is simpler, practical and, more comfortable. Edmodo is recommended for distance learning since it has more features. While Google Classroom is more suitable to be used as a support or complement for a learning course.
1 Introduction
The use of information technology in education emerged the e-learning (electronic learning) term. It covers a broad spectrum of activities, from its utilization for supporting learning to fully online learning [1]. E-learning is defined as instructions delivered through digital devices such as computers or mobile devices to support learning [2]. E-learning includes the use of the internet and other important technologies to produce learning materials, teach students, and also organize courses [3]. An E-learning system that is specially designed to provide educational content, activities, and support for managing learning programs is called Learning Management Systems (LMS) [4].
Today's LMS leads to social and collaborative learning. The collaborative method emphasizes social learning and involves students sharing knowledge and doing tasks collaboratively [5]. Social learning creates informal learning: students can build a network, share, collaborate, and exchange ideas to solve a problem [6]. Edmodo and Google Classroom are the LMS that implement collaborative methods. Even though Edmodo and Google Classroom do not explicitly mention themselves as LMS, based
Hanifah Muslimah et al. , User Experience in Learning Management System...197 on the characteristics of their services, many observers and researchers [7–10]
categorize them as LMS.
Edmodo is one of the social learning networks to connect students with instructors (teachers). Edmodo’s network allows teachers to share content, distribute quizzes, assignments, and manage communication with students, colleagues, and parents. In Edmodo, instructors can discuss in online classes, provide polls to check students’
understanding, and give badges to students based on behavior or performance. Edmodo offers features that are: accelerating learning goals, activating free admin accounts, badge awards for individual students, building community networks, making polls for students, measuring student progress, connecting students, administrators/teachers, and parents, Online class discussions, personalize with the application.
Google Classroom is an online collaboration platform for teachers and students designed to enhance the learning experience. Teachers can create online classrooms, ask students to join classes, and create and give assignments. Students and teachers can communicate about assignments, teachers can also monitor the progress of their students. The features offered by Google Classroom are providing classrooms without paper, access to other Google products such as Gmail, Google Drive, Google Form, and collaboration between teachers and students outside the classroom, easy to manage, save money, and for teachers to better track student progress.
LMS becomes a medium of interaction between students and teachers so their interaction will be easy and intuitive [11]. The low quality of interaction in e-learning will hamper the development of effective learning with technology [12]. Poor usability and user experience (UX) of an LMS can make students spend more time learning the system than learning the educational content [11]. Therefore, usability and user experience evaluation is important for e-learning.
E-learning is very different from other interactive systems. The meaning of usability and user experience, in general, is not sufficient to cover the user experience in the e- learning context [13]. It is important to evaluate the pedagogical design of e-learning systems [14][11]. So, the user experience evaluation method, in general, has not been able to evaluate e-learning. A special method for evaluating user experience on e- learning is needed. Technique for User Experience Evaluation in E-learning (TUXEL) is a technique provided to evaluate the usability and user experience of an LMS [15].
TUXEL is questionnaire-based so that it is not only easily implemented at a low cost but also allows collecting perceptions from the user's point of view [15]. It has come with a developed version: TUXEL 2.0, which is a guided inspection approach. TUXEL 2.0 has three evaluation stages: (1) usability inspection, (2) pedagogical usability inspection, and (3) user experience evaluation.
The study was conducted at a vocational high school in Malang Indonesia as there was a specific need to use the LMS on online learning during the COVID-19 pandemic and industrial work practice. In their second year, students are required to carry out industrial work practices for six months, however, learning in schools continues in parallel. The duty of work practices limits the student’s active hours of learning at school. These time limitations raise the need for teachers and students towards LMS to communicate and interact outside the physical classroom.
This study evaluates students' experiences of their interactions with Edmodo and Google Classroom. Based on the results of the evaluation, it will be concluded which LMS is felt most positive by the students. The results of the evaluation can be useful for teachers or the school for consideration in adopting which LMS is the most appropriate to be applied to support learning
198 JITeCS Volume 7, Number 3, December 2022, pp 196-208
2 Literature Study
Technique for User eXperience Evaluation in e-Learning (TUXEL) 2.0
The development of TUXEL begins with defining three dimensions for TUXEL:
usability, pedagogical usability, and user experience. The study of the initial version of TUXEL evidenced problems regarding the difficulty of students in filling out the questionnaire. The participants had trouble remembering the function, aspect, or element of the LMS. Based on those problems, TUXEL 2.0 was developed. It is designed to gather deeper usability and user experience problems of LMS based on students' viewpoints [15]. It is questionnaire-based so that it is easily implemented at a low cost and fast. It also does not require prior user testing and is more intuitive for inspectors in finding problems [15].
The TUXEL 2.0 has a preliminary stage and three main stages. The preliminary stages are task definition and checklist matching. The three main stages, which include three dimensions of TUXEL, are (1) usability inspection, (2) general LMS / pedagogical usability inspection, and (3) user experience evaluation. The usability dimension is divided into 4 categories: (1) login, (2) general interface, (3) assignment, (4) assessment. Questionnaires were given to the participants regarding these 4 categories. The "login" category questions are related to the clarity of instructions and feedback while users are logging in. The questions of the "general interface" category, related to the usefulness of the user interface, such as the navigation structure, the terminology used, and the information setups. The questions of the "assignments"
category are intended to evaluate specific aspects related to the task, such as the clarity of instructions, the level of user control, and the feedback from the LMS. The
"assessments" category poses questions related to the instructional feedback and goal.
The usability dimension is divided into 4 categories: (1) login, (2) general interface, (3) assignment, (4) assessment. Questionnaires were given to the participants regarding these 4 categories. The "login" category questions are related to the clarity of instructions and feedback while users are logging in. The questions of the "general interface" category, related to the usefulness of the user interface, such as the navigation structure, the terminology used, and the information setups. The questions of the
"assignments" category are intended to evaluate specific aspects related to the task, such as the clarity of instructions, the level of user control, and the feedback from the LMS. The "assessments" category poses questions related to the instructional feedback and goal.
The pedagogical usability dimension is also divided into 4 categories: (1) help and documentation, (2) LMS learnability, (3) learning through the LMS, (4) LMS flexibility. The first category, “help and documentation” relates to whether the LMS provides assistance and whether the learning material is comprehensive and forward- looking. The second category, “LMS learnability” is related to the ease of the system to learn. The third category, “learning through the LMS” is about evaluating the extent to which students can learn through the. The fourth category, “LMS flexibility” is to evaluate the extent to which students can choose the learning material they want.
Meanwhile, the user experience dimension adapts the User Experience Questionnaire (UEQ) [16] technique because of its viability, ease of use, and validity [15]. The UEQ is a semantic differential scale that allows users to rate their experience based on pairs of adjectives. The technique uses 26 adjectives to evaluate 6 factors: (1) attractiveness, (2) perspicuity, (3) efficiency, (4) dependability, (5) stimulation, and (6) novelty. TUXEL 2.0 adapts 12 of the 26 UEQ [16] pairs of adjectives.
Hanifah Muslimah et al. , User Experience in Learning Management System...199
3 Methodology
The study begins with the preliminary stages: defining inspection tasks and matching the tasks with TUXEL 2.0 categories. Furthermore, the evaluation involved students as inspectors. The inspectors were vocational high school students in Malang, Indonesia who had been carrying out industrial work practices. The inspectors was asked to perform determined tasks. There were two groups of inspectors, group one inspected Edmodo and another group inspected Google Classroom. There are 12 people in each group, so there are 24 evaluators in total. Groups of ten to twelve participants gave statistically significant results in user experience evaluation [17]. The data was gathered through questionnaires. There were three questionnaires: usability inspection, general LMS / pedagogical usability inspection, and user experience evaluation. The data are then processed and analyzed per category per dimension.
3.1 Task Definition and Checklist Matching
The preliminary stage was determining the tasks that must be carried out by students as inspectors. The tasks were determined by analyzing features on Edmodo and Google Classroom so that the comparison will be equal. The task determination also refers to the four categories of TUXEL 2.0: general interface, user login, assignments, and assessments. Each task should represent TUXEL 2.0 categories as can be seen in table 1.
Table 1. The tasks for Edmodo and Google Classroom inspection.
No Task Description TUXEL Categories
1 Log in Login, General Interface
2 Join the class with provided code Login, General Interface 3 Study the posted material General Interface, Assignments 4 Add a comment to the post General Interface
5 Submit an assignment as instructed General Interface, Assignments, Assessments
6 Do the available quiz General Interface, Assessments
7 Log out Login, General Interface
3.2 General Usability Inspection
The general usability inspection is analyzing the usability of the LMS. Students as inspectors performed the tasks given and inspected each task. Based on the problems list provided by TUXEL 2.0 (can be read further in reference [18]), the inspectors checked if there was any problem that occurred with each task. When the inspectors found a problem, they write down the problem ID, specify the location or type of task, and define the recurrence level of the problem (the problem is repeating, repeating throughout the system, or repeating on several tasks on the report table provided by TUXEL 2.0 (see the “Usability Inspection – Problem Reporting Table” on reference [18]). If the problem repeats for several tasks, the task must also be defined.
3.3 Pedagogical Usability Inspection
Pedagogical usability is the usability that is related to the type of learning material that leads users to reach their learning achievements, so it will have implications for the learning platform [19]. At this stage, it is expected that an LMS could support the delivery of learning material as well as facilitate the learning process. TUXEL 2.0
200 JITeCS Volume 7, Number 3, December 2022, pp 196-208 provides the “General LMS Evaluation Checklist” (see on reference [18]) to evaluate the pedagogical usability. The checklist was adapted into Bahasa Indonesia and filled by inspectors. Inspectors marked items that correspond to problems encountered when operating the LMS and explained the details of the problem in the comments section.
3.4 User Experience Evaluation
The last step is evaluating the user experience of the LMS using a seven-point semantic differential scale questionnaire. The aim is to gather student perceptions regarding student experiences when using LMS. TUXEL 2.0 adapts 12 of the 26 UEQ [16] pairs of adjectives: (1) unlikable/pleasing, (2) uncomfortable/comfortable, (3) does not meet expectations/meets expectations, (4) obstructive/supportive, (5) inefficient/efficient, (6) impractical/practical, (7) conventional/innovative, (8) uncreative/creative, (9) complicated/easy, (10) confusing /clear, (11) not interesting/interesting, (12) demotivating/motivating.
The inspectors rated their experience on LMS by choosing one of seven-point that comes closest to the adjective that portrayed their feelings. After filling out the scale, the inspector gave responses to criticism or advice. Those seven-point semantic differential scale data were converted to a score range of -3 to +3, thereafter the median score was calculated.
3.5 Analysis
Data from the inspectors were collected, processed, and analyzed. The general usability inspection data were analyzed by counting the number of problems per task and item ID, examining the problem description, and checking whether the inspector assessed the problem as recurring. The pedagogical usability inspection data were analyzed for each aspect by counting the problems that had been check-marked and looking at the keywords of the comments given. Meanwhile, the data collected from the user experience form were processed by calculating the median for each pair of adjectives and examining the keywords of the comments given. For each dimension (the general usability, pedagogical usability, and user experience), the calculation results were linked with the keywords of the comments given. Thereafter the problem concluded from each dimension. Finally, the problems of each dimension were linked to drawing a complete conclusion related to the learning experience of the LMSs.
4 Results
Table 2 shows the number of problematic items and table 3 shows descriptions of the problematic items found on the inspection of the general usability and pedagogical usability dimensions.
Table 2. Number of problematic items found on General Usability and Pedagogical Usability Dimension
Dimension Category
No. of Item per Category
No. of problematic item Edmodo Google
Classroom General
Usability Login 4 1 3
General Interface 12 6 5
Assignments 7 2 2
Hanifah Muslimah et al. , User Experience in Learning Management System...201
Dimension Category No. of Item
per Category No. of problematic item Edmodo Google
Classroom
Assesments 3 0 2
Total problematic items on General Usability 9 12 Pedagogical
Usability Help and
Documentation 3 3 3
LMS Learnability 5 5 3
Learning through the
LMS 6 2 6
LMS Flexibility 7 3 6
Total problematic items on Pedagogical Usability 13 24
Total 22 30
Table 3. Problems Classification and Description per Item’s ID Dimension Category Item’s
ID Item’s description Problems found Edmodo Google
Classroom General
Usability Login LF2 LMS should inform required field to be filled not informed
Yes No
LF4 Information of how to fill
a field not available No Yes LF3 Information of not-
completed-field not available
No Yes
LF1 Feedback an action not
provided No Yes
General
Interface VD5 Low color contrast No No N1 Navigation options are
not visible Yes Yes
C1 Interface design is not
consistent No No
C2 Terminology is not
consistent No No
L4 Icon, button, label, and link not functional or not understandable
Yes Yes
VD2 Unnecessary content and/or information
Yes No
VD3 Important information
not visible Yes No
N4 Content/information not well-organized
No No
LC3 System exit option not
available No Yes
N6 Search engine for information finding not available
No Yes
202 JITeCS Volume 7, Number 3, December 2022, pp 196-208
Dimension Category Item’s
ID Item’s description Problems found Edmodo Google
Classroom N5 Visited links not different
to unvisited ones
Yes No
N2 Breadcrumb navigation
not available Yes Yes
Assignme nts
L2 Instructions are limited or unclear
Yes No
LF4 Information about how to fill-in the field correctly is not available
No Yes
LC2 User cannot cancel an
action at any time Yes No
LF6 User progress of an
action is not showed. No No LC1 User cannot undo/redo an
action easily No No
LF5 Confirmation of important action not required
No No
LF1 LMS does not provide feedback when performing an action
No Yes
Assessme
nts OO1 Score when performing
activities not provided No No IF1 Immediate feedback is
not provided when answering
No Yes
IF2 Feedback is not
relevant/appropriate No Yes Pedagogical
Usability
Help and Document ation
HD1 Manual or online help is not provided
Yes Yes
HD2 Manual or online help is
hard to find Yes Yes
HD3 Instruction provided is
complex Yes Yes
LMS Learnabili ty
L1 LMS is not intuitive so it
is difficult to operate Yes Yes L3 User need others people
help to complete task
Yes Yes
L6 User will forget how to operate LMS after a long time
Yes Yes
L7 User cannot perform the
task quickly Yes No
N3 LMS cannot be operated easily
Yes No
Hanifah Muslimah et al. , User Experience in Learning Management System...203
Dimension Category Item’s
ID Item’s description Problems found Edmodo Google
Classroom Learning
through the LMS
CL1 Group activities cannot be carried out through LMS
No Yes
CL2 Colleagues
communication is not allowed through LMS
Yes Yes
CL3 User cannot see other colleagues activities
Yes Yes
CL4 File sharing is not
provided No Yes
CL5 User cannot post comments or answer question from others
No Yes
OO2 User cannot track their progress in the course
No Yes
LMS
Flexibility FL1 User cannot choose course, order and pace to study
Yes Yes
FL2 User cannot consult
difficulties in learning Yes Yes FI3 User cannot seek
additional feedback with instructor
No Yes
P1 User cannot customize
the interface Yes Yes
P2 Shortcut keys not
provided No Yes
P3 Language option not provided
No No
AI1 Self-assessment not
provided No Yes
Total 22 30
4.1 General Usability of Edmodo versus Google Classroom
On the overall general usability dimension, Edmodo has fewer problematic aspects (9 items) than Google Classroom (12 items). In both LMS, problems are found in the login, general interface, assignments, and assessments categories. Out of the four items in the login category, one problematic item was found in Edmodo, and three problems were found in Google Classroom. Out of the 12 items in the general interface category, six problematic items were found in Edmodo, and five problems were found in Google Classroom. Out of the seven items in the assignment category, two problematic items were found in both LMSs. Out of the three items in the assessment category, no problematic items were found in Edmodo and two problems were found in Google Classroom.
The problem with logging in to Edmodo was discovered by students who felt
204 JITeCS Volume 7, Number 3, December 2022, pp 196-208 confused when choosing which column is required because there was no sign for the required column (LF2). Meanwhile, the problem with logging in to Google Classroom is that when logging in, students were redirected to the Google Account login page which is considered confusing by students (LF4, LF3, and LF1). In the General Interface category, students reported difficulties in both LMSs to find navigation, menus, and options (N1, N2). Students were also struggling to understand icons, buttons, and labels (L4). In Edmodo, students found unnecessary content and/or information (VD2), yet important information is not visible (VD3).
The problem in the assignment category is that students felt that the instructions or instructions are not clear (L2) on assignments in Edmodo and there is no information on how to fill in the correct way (LF4) on assignments in Google Classroom. Students also found that Edmodo does not accommodate action cancellation, while Google Classroom does not give success or failure messages when submitting assignments (LF1). In the assessment category, there is no problematic item was found in Edmodo, whereas there are two problems were found in Google Classroom are related to no immediate feedback provided for a quiz (IF1) and no additional material or link when the student finished the task (IF2).
4.2 Pedagogical Usability of Edmodo versus Google Classroom
On the overall pedagogical usability dimension, Edmodo has fewer problematic aspects (13 items) than Google Classroom (24 items). In both LMS, problems are found in the help and documentation, LMS learnability, learning through the LMS, and LMS flexibility categories. Out of the three items in the help and documentation category, all items were found problematic in both LMSs. Out of the five items in the LMS learnability category, all items are problematic in Edmodo and only three problematic items were found in Google Classroom. Out of the six items in the learning through the LMS category, two problems were found in Edmodo but all items were found problematic in Google Classroom. Out of the seven items in the assessment category, three problematic items were found in Edmodo and six problems were found in Google Classroom.
The problem with the help and documentation category in both LMSs are manual or online help is not provided (HD1); manual or online help is hard to find (HD2); and instruction provided is complex (HD3). In the LMS learnability category, problems were found related to the ease with which LMS was learned by the user. Both LMS were reported that not intuitive so students are difficult to operate them (L1), students need other help to complete tasks (L3) and students will forget how to operate LMS after a long time (L6). However, Google Classroom is considered easier and quicker (L7 and N3) to perform tasks rather than Edmodo.
The problem with the learning through the LMS category is both LMSs do not provide features for students to communicate with students or teachers (CL2), and view the activities and grades of other students (CL3) on the LMS. On top of that, Google Classroom does not provide services to carry out group activities (CL1), file sharing from students to other students (CL4), no question and answer forum between students (CL5) and no progress tracking in the completion of the course (OO2). On the LMS flexibility category, both LMSs reported that students are unable to choose a course, order and pace to study (FL1), consult difficulties in learning (FL2) and customize the interfaces of the LMSs (P1). On top of that, Google Classroom does not provide additional feedback with the teacher/instructor (FI3), shortcuts to increase productivity
Hanifah Muslimah et al. , User Experience in Learning Management System...205 (P2), and self-assessment for students (AI1).
4.3 User Experience of Edmodo versus Google Classroom
Figure 1 shows the overall median calculation for user experience in both LMSs which tend to be neutral and positive (scores 0 to 3). Almost all of the keywords show Google Classroom has more positive scores than Edmodo, except for the last keyword i.e.
“creative”.
Fig. 1. Comparison of median scores for user experience of Edmodo versus Google Classroom Edmodo has a neutral median score (0) of the “pleasant” keyword whereas Google Classroom has a positive score (2). In the "comfortable” keyword, the median score for Edmodo is quite positive (1) while Google Classroom score is positive (2). In the
"efficient" keyword, the median score of Edmodo and Google Classroom are both positive (2). In “practical” keywords, Edmodo's median score is between positive/very positive (2.5), slightly below Google Classroom which is very positive (3). Followed by the "easy" keyword, Edmodo is rated between neutral-fairly positive (0.5), far below Google Classroom which is positive (2). In the "event" keyword, the median score for Edmodo is neutral (0), almost the same as Google Classroom which is between neutral/fairly positive (0.5). Furthermore, the “meets expectations” keyword on Edmodo is neutral (0) while Google Classroom is quite positive (1). In the “supportive”
keyword, the median score of both Edmodo and Google Classroom is positive (2). The
"interesting" keyword, Edmodo has a median between quite positive/positive (1.5) and Google Classroom is positive (2). In the "motivate to learn" keyword, the median score
206 JITeCS Volume 7, Number 3, December 2022, pp 196-208 of Edmodo is quite positive/positive (1.5) while the median of Google Classroom is positive (2). The “inventive” keyword shows the similarity of the median score between Edmodo and Google Classroom, which is positive (2). For the "creative" keyword, the median score of Edmodo is very positive (3), above the Google Classroom score which is positive (2).
5 Conclusion and Recommendation
The results of the overall general usability and pedagogical usability inspections show fewer problematic aspects in Edmodo (22 problematic items) than Google Classroom (30 problematic items). This is because Edmodo provides more complete features than Google Classroom. Edmodo features are more complete for the assessment, learning through the LMS and LMS flexibility categories. Edmodo provides scoring along with immediate and relevant feedback on students' assessments. Edmodo also provides space for group activities and discussion forums between students and teachers. However, the simplicity of Google Classroom seems to be preferred by students. Although it does not provide as many features as Edmodo, students find Google Classroom more fun, comfortable, practical, interesting and motivating to learn and fulfill students' expectations of an LMS, rather than Edmodo.
The problems encountered in both LMS are related to the ease of use. On the general interface category, students reported difficulty finding navigation, menus or options on the LMS. The user experience measurement also shows that the “confusing/evident”
keyword is calculated as neutral. Joyce and Nielsen [20] argue that teenagers, of which high school students are included, are over-confidence, but they perform worse than adults in performance on the web. Due to their lower reading levels, impatience, and underdeveloped research skills, thereby reducing their successes in doing the tasks.
Therefore, usability is thus as important for students as well as for any other user group.
Edmodo claims to be an LMS with a variety of complete questions/evaluations.
Google Classroom has the advantage of integrating GDrive, GMail, and other technologies. In addition, Google Classroom provides features that provide freedom to fail during exploration. So that, Edmodo is recommended for distance learning since it has more complete features, especially for assessment. Online learning with Edmodo allows students to receive feedback from learning activities and communicate to students or teachers. While Google Classroom is more suitable to be used as a support or complement for a learning course as it is simpler and more practical. Google Classroom makes it easy for students to get learning material from the teacher or submit assignments.
References
1. Ssekakubo, G., Suleman, H., Marsden, G.: A streamlined mobile user-interface for improved access to LMS services. eLmL - International Conference on Mobile, Hybrid, and On-line Learning. 92–101. 2014.
2. Clark, R.C., Mayer, R.E.: e-Learning and the Science of Instruction: Proven Guidelines for Consumers and Designers of Multimedia Learning. Wiley. 2011.
3. Fry, K.: E-learning markets and providers: some issues and prospects. Education + Training. 43, 233–239. 2001. https://doi.org/10.1108/eum0000000005484.
Hanifah Muslimah et al. , User Experience in Learning Management System...207 4. Dubost, A., Klein, M.R., Dang, J.: Building interoperability between LMS and Brokerage Platforms. In: Proceedings of the 2004 3rd International Conference on E-Activities. 2004.
5. Ghirardini, B.: E-learning Methodologies: A Guide for Designing and Developing E-learning Courses. Food and Agriculture Organization of the United Nations.
2011.
6. John Laskaris: Social learning: the revolution in eLearning, https://www.talentlms.com/blog/social-learning-in-elearning/, last accessed 2019/03/26.
7. Francom, G.M., Schwan, A., Nuatomue, J.N.: Comparing Google Classroom and D2L Brightspace Using the Technology Acceptance Model. TechTrends. 65, 111–
119. 2021. https://doi.org/10.1007/s11528-020-00533-0.
8. Kado, K., Dem, N., Yonten, S.: Effectiveness of Google Classroom as Online Learning Management System in the Wake of COVID-19: Bhutanese Students’
Perceptions. Presented at the October 7. 2020.
9. Moca, M.: The Students’ Perception on Incorporating Google Classroom in Learning Economic Disciplines. In: Fotea, S.L., Fotea, I.Ş., and Văduva, S. (eds.) Post-Pandemic Realities and Growth in Eastern Europe. pp. 85–100. Springer International Publishing, Cham. 2022.
10. Taufik, M., Rijal, A.S., Dahniar, D., Apriani, E.: The Effectiveness of Learning English Using LMS Google Classroom during Covid-19 Pandemic. AL-ISHLAH:
Jurnal Pendidikan. 13, 960–970. 2021.
https://doi.org/10.35445/ALISHLAH.V13I2.706.
11. Ardito, C., Costabile, M., De Marsico, M., Lanzilotti, R., Levialdi, S., Roselli, T., Rossano, V.: An approach to usability evaluation of e-learning applications.
Univers Access Inf Soc. 4, 270–283. 2006. https://doi.org/10.1007/s10209-005- 0008-6.
12. Kakasevski, G., Mihajlov, M., Arsenovski, S., Chungurski, S.: Evaluating usability in learning management system moodle. In: ITI 2008 - 30th International Conference on Information Technology Interfaces. pp. 613–618. 2008.
https://doi.org/10.1109/ITI.2008.4588480.
13. Reeves, T., Benson, L., Elliott, D., Grant, M., Holschuh, D., Kim, B., Kim, H., Lauber, E., Loh, C.: Usability and Instructional Design Heuristics for E-Learning Evaluation. 2002.
14. Kirsi, S., Tervakari, A.: An Evaluation of the Usefulness of Web-based Learning Environments.The Evaluation Tool into the Portal of Finnish Virtual University..
2003.
15. Nakamura, W., Oliveira, E., Conte, T.: TUXEL: A Technique for User eXperience Evaluation in e-Learning. In: Anais dos Workshops do VII Congresso Brasileiro de Informática na Educação (CBIE 2018). Brazilian Computer Society (Sociedade
Brasileira de Computação - {SBC}). 2018.
https://doi.org/10.5753/cbie.wcbie.2018.52.
16. Laugwitz, B., Held, T., Schrepp, M.: Construction and Evaluation of a User Experience Questionnaire. In: Holzinger, A. (ed.) HCI and Usability for Education and Work. pp. 63–76. Springer Berlin Heidelberg, Berlin, Heidelberg. 2008.
17. Macefield, R.: How to Specify the Participant Group Size for Usability Studies: A Practitioner’s Guide. J. Usability Studies. 5, 34–45. 2009.
18. Nakamura, W., de Oliveira, E., Conte, T.: Support material for evaluating the usability and User eXperience of Learning Managemement Systems. 2018.
208 JITeCS Volume 7, Number 3, December 2022, pp 196-208 19. Nokelainen, P.: An empirical assessment of pedagogical usability criteria for
digital learning material with elementary school students. Educational Technology
& Society. 9, 178–197. 2006.
20. Joyce, Alita; Nielsen, J.: Teenager’s UX: Designing for Teens, https://www.nngroup.com/articles/usability-of-websites-for-teenagers/, last accessed 2020/07/20.
