Log Data Indicators for Identifying Learner Engagement in MOOCs
Mohamad Shafri Amir Mohd Sharif 1*, Prasanna Ramakrisnan1
1 School of Computing Sciences, College of Computing, Informatics and Media, Universiti Teknologi Mara, Shah Alam, Malaysia
*Corresponding Author: [email protected], [email protected] Accepted: 15 February 2023 | Published: 1 March 2023
DOI:https://doi.org/10.55057/ijares.2023.5.1.5
__________________________________________________________________________________________
Abstract: Engagement during learning is crucial as the instructor and even management can see how well and understood is learner about the topic or course they are up to. As Engagement happens in Massive Open Online Courses (MOOCs), a lot of research papers discuss it and it is more interesting as the pandemic happens and most students must adapt to online learning.
Log data in MOOCs have recorded engagement indicators and it is called log data indicators.
However, it comes to the question of what is considered engagement in MOOCs as engagement has three parts which are affective, cognitive, and behavioral. A Systematic Literature Review (SLR) was used in this study to find out what engagement indicators for MOOCs can be captured. The result showed for this study is the indicators that can be used to analyze to explore student engagement which include the description of the indicators accordingly. From an online education perspective and improvement of the MOOCs platform, The engagement indicators are really necessary as it is part of learning. Able to identify the engagement and view the level of engagement for every participant in the course. The exploration of the engagement itself makes the instructor well knowledge of what to improve in their courses materials.
Keywords: MOOCs, education, engagement indicator, log data indicator
___________________________________________________________________________
1. Introduction
Massive Open Online Courses (MOOCs) platform is where students can learn anywhere, they like as long as have access to the internet. Students can learn and register for any courses provided on the MOOCs platform. MOOCs have been touted as game-changing in the field of education (Sooryanarayan & Gupta, 2015). Online learning such as MOOCs is different from traditional learning. A few research has found that interactions in online learning, particularly MOOCs, have substantial relationships and positive consequences (Stracke et al., 2018). This proves that engagement happens in MOOCs gives a positive outcome to the learner.
Engagement in MOOCs is different from engagement in a physical class, where students can engage with educators directly. According to Stracke et al., (2018), because MOOCs frequently do not allow for such direct contact, there is no comprehensive research on interactions with MOOC facilitators. Indirect contact interaction with educators gives MOOCs invisible to engagement. Even though engagement happens in MOOCs, educators are unable to view how and what type of engagement. It is because engagement is really important in learning. In an online learning environment, engagement is a critical indicator of a user's learning experience (Huang et al., 2019). Therefore, the metric to show the engagement level in MOOCs is
necessary to understand how to engage the learner in courses currently learning and make educator, and owner of the course make the course-related decision to improve the course and increase the engagement in MOOCs. Engagement data are being recorded in logs. MOOCs can record and process massive amounts of information and save it into logs. Logs data summaries of massive quantities of MOOC activity that are both understandable and actionable (Geigle et al., 2017). Utilizing log data for the purpose to enhance MOOCs has been widely used by researchers in their research. For example, (Kőrösi & Farkas, 2020), use log data to predict student performance at the end of the MOOC course. As log data is still raw and required some sort of features framework and algorithm to process to show the expected outcome. The same goes for engagement data being recorded in log data is still raw and required a special process or algorithm to be more readable to educators. (Moubayed et al., 2020) retrieve a new dataset from log data that represented the necessary engagement metrics using MATLAB.
Learner engagement is described as a student's cognitive and emotional energy being invested in completing a learning assignment (Halverson & CR Graham, 2019). Engagement is necessary for educators to improve ways of teaching and materials for learning. Attending courses (behavioral engagement), asking questions (cognitive engagement), and/or expressing enthusiasm toward course events or teachers are all examples of student involvement (Hew et al., 2018). In the traditional way of teaching, the learner is present physically in a room and the educator will be able to teach directly. In this way, engagement happens in class, where the educator can see how active the student is and how bored the student is with the current learning topic.
How students appreciated the online recorded lectures, which improved their involvement and helped them form a rapport with the instructor (Clark et al., 2017). However, learning through the MOOCs platform, the instructor can’t see how the engagement happens and how good or bad is the course for a learner which might contribute dropout rate. Learners that show a low degree of engagement are more likely to fail (Swinnerton et al., 2017). From a management view, the dropout rate will be affecting the MOOCs business as it is indicated that the course they register for might be boring or engaging while learning the offered course is not so hype for a learner. The high drop-out rates of MOOCs raised the issue of their quality, which is currently being debated (Stracke, 2017). Understanding the level of engagement in MOOCs able to give judgment to improve the quality of courses offered in MOOCs.
According to (Tseng et al., 2016), We still don't have a good knowledge of how students engage in such courses. Therefore, this paper will have an expert to validate the engagement indicator that happens in MOOCs. The level or degree of a student's involvement in a learning activity is known as student engagement (KF Hew, 2015). Student engagement is divided into three parts which are behavioral, emotional, and cognitive (Henrie et al., 2015). The actions that are required for academic performance are referred to as behavioral engagement (Henrie et al., 2015). While for emotional engagement according to (K. F. Hew, 2016), students' affective responses or feelings toward teachers, peers, the course, and learning are referred to as emotional engagement. Next, Students' motivational goals and self-regulated learning capabilities are commonly referred to as cognitive engagement (Joksimović et al., 2018). In terms of engagement in MOOCs, SLR methodology is required to answer the research question for this paper which is shown in Table 1.
Table 1: Research questions, description, and motivation.
Research Question Description and Motivation
What engagement indicators can be captured in log data from MOOCs platform?
The question is focused on getting the engagement indicator, which can be captured from the log data in MOOCs platform.
The indicators are gathered using SLR.
2. Motivation
The primary driving force behind conducting this study was because nature of engagement itself in online learning as it highly trending. Two factors are driving this rising scholarly interest in MOOC participation (Deng et al., 2020). Online learning platforms which are called MOOCs offer a lot of courses that attract anyone of any age to learn. Not like other normal physical classes, classes happening on MOOCs platform are hard to see the engagement. For example, in physical classes can see there is engagement happening between students and teacher where they can interact with each other in a real-time situation. While classes in MOOCs based on students’ availability which is means students can learn and open the course in their free time and the instructor doesn’t have to wait for them to complete a course.
Instructors in MOOCs act as a teacher who is expert in particular offered courses and their task is only to upload their course teaching materials and prepared the assessments for their registered students. Due to that, an instructor is limited in their knowledge which is the instructor can’t see the engagement happening in their course and does not know what real list of indicators are used to count as engagement in MOOCs. Engagement in learning is related to student learning performance. (Hussain et al., 2018)in their studies, to determine the impact of engagement on student performance, they employed machine learning (ML) algorithms to detect low-engagement students in a social science course at the Open University (OU).
Knowing the indicator of engagement happening in MOOCs helps the instructor to improve their course’s material and led to better student performance.
3. Related work
A lot of researchers have been relating engagement in MOOCs. Most of the research articles only discuss engagement or a few and it is depending on the purpose of the studies. For example, the discussion on engagement on how it is affecting the dropout or completion of the courses in MOOCs was discussed by Wang et al., (2018). Wang et al., (2018), researched the effects of social-interactive engagement on learning development and dropout rates in MOOCs.
(Wang et al., 2018), in their studies only focus on social-interactive engagement. Estrada- Molina & Fuentes-Cancell, (2022), use the studied engagement to lower the MOOC attrition rate. To be specific, the studies use the systematic review to find out which engagement can reduce the dropout. Next, academic engagements used by Guajardo Leal et al., (2019) in their article, use a method called systematic mapping of literature, which aims to identify the characteristics of production in each field, to systematically explore the literature that has been produced between the years of 2015 and 2018 about the concept of academic engagement in online, massive, and open learning courses. Another engagement which is learner engagement is discussed in an article research by Deng et al., (2020). their study adopts a multi-dimensional, person-cantered approach to assess learner engagement in MOOCs, expanding the scope of other studies. Romero-Rodriguez & MS Ramirez-Montoya, (2019) also use learner engagement to explore how the implementation of gamification techniques in MOOCs on energy sustainability impacts participants' engagement and aims to determine which interactive
gamification media are most effective in inspiring students' interest and drive. In addition, the systematic approaches done by Paton et al., (2018) is a systematic review to evaluate learner engagement. A research paper from (Carlon et al., 2020), discusses how to increase engagement in MOOCs by evaluating the speaking rate in videos. (Mongkhonvanit et al., 2019) account for student participation by incorporating course interaction covariates in a Deep Knowledge Tracing methodology. Student engagement in the courses is also related to self- regulated learning (SRL). Min & Foon, (2019), details the unique behavioral, emotional, and cognitive engagement characteristics that participants exhibit when they self-regulate their learning in MOOCs. As shown in Table 2, the present literature on engagement is divided into categories.
Table 2: Literature on engagement
No Title Research
Method Year Discussed
Engagement Description
1. Effects of social-interactive engagement on the dropout ratio in online learning:
insights from MOOC
Research Article
2018 Social- interactive
research the effects of social interaction on learning development and dropout rates
2. Engagement and desertion in MOOCs: Systematic review
Systematic review
2022 Generic Engagement
confirmed that tailored instruction, interaction, and feedback pose the biggest obstacles to ensuring MOOC participation.
3. Engagement and retention in VET MOOCs and online courses: A systematic review of literature from 2013 to 2017
Systematic Review
2018 Generic Engagement
Did systematic evaluation of literature from 2013 to 2017 examined learner engagement and retention in university MOOCs and online courses.
4. Gamification in MOOCs:
Engagement application test in energy sustainability courses
Research article
2019 Generic Engagement
explores how the implementation of gamification techniques in MOOCs on energy sustainability impacts participants' engagement and aims to determine which interactive gamification media are most effective in inspiring students' interest and drive.
5. Linking learner factors, teaching context, and engagement patterns with MOOC learning outcomes
Research article
2020 Generic Engagement
demonstrates substantial differences between the three MOOC cohorts in terms of student characteristics (gender, origin, motivation), instructional setting (course level, course duration,
manner of evaluation), and learning results (course completion, perceived quality of instruction)
6. Systematic mapping study of academic engagement in MOOC
Systematic Literature
Review
2019 Academic Engagement
comprehensively examines the literature that has been published between the years of 2015 and 2018 about the concept of academic engagement in online, massive, and open learning courses.
7. From the learner's perspective: A systematic review of MOOC learner experiences (2008–2021)
Systematic Review
2022 Generic Engagement
gives insightful information about how to use student experiences to inform course design and evaluation.
8. Supporting Self-Regulated Learning in Online Learning Environments and MOOCs:
A Systematic Review
Systematic Review
2019 Generic Engagement
gives a thorough assessment of research on methods for promoting self-regulated learning in various online learning settings and how these methods consider human aspects.
9. Video and Learning: A Systematic Review
Systematic Review
2018 Generic Engagement
enumerate the outcomes of modifying the video's appearance, substance, and activities assessing learning outcomes, including academic, transfer, and recall achievement is one of many (Poquet et al., 2018)
10. Refocusing the Lens on Engagement in MOOCs
Research Article
2018 Student Engagement
propose a multilevel definition of students' participation in MOOCs and investigate the relationships between participation and students' actions across five different courses (Crues et al., 2018).
11. Content Type Distribution and Readability of MOOCs
Short Paper
2020 Generic Engagement
Analysis of content distribution enables comparison of MOOCs across all subject areas.
12. Deep Knowledge Tracing and Engagement with
Short Paper
2019 Student Engagement
a student's subsequent item response with greater than 88%
MOOCs accuracy. These forecasts can be used to provide students with customised interventions and improve courses in a targeted manner.
13. Self-Regulated Learning Process in MOOCs:
Examining the Indicators of Behavioral, Emotional, and Cognitive Engagement
Research Article
2019 Behavioral, Emotional, and cognitive
engagement
examined the self-regulated learning procedures used by the MOOC participants in the three areas of involvement
4. Systematic Literature Review
Our comprehensive literature study was motivated by Liao et al., (2020). A systematic literature review is chosen to gather log data indicators that might be able to refer to student engagement in MOOCs. The methodology itself helps the researcher to answer academic inquiries. SLR methodology is a methodology to review previous research studies systematically based on what the researcher wants in the first place. Information retrieved from the literature review can be a basis for further research efforts (Collins et al., 2021). To identify the student engagement indicator in MOOCs, the researcher must know what the MOOCs are and what engagement is. By doing a SLR, the researcher was able to identify the log data indicators for engagement in MOOCs. The steps provided in this methodology are shown in Figure 1, as a guide to the researcher to implement it. A brief explanation of the step is explained below.
Figure 1: Step of Systematic Literature Review
5. Planning the Review
The first step for conducting SLR is planning the review. The flow chart explains the Planning review is shown in Figure 2. The objective as per described in Table 2 must be fully understood by the researcher of this study as it is needed to keep track of answering the requestion questions. To be clear, the objective is to find out what engagement indicator can be easily captured by log data in MOOCs. Log data in MOOCs are quite a lot as it keeps increasing while up and running. The job to utilize the log data are belonging to any person in charge who knows what the indicator engagement is that is present in log data. Thus, the researcher for this study is responsible for answering the research question by doing the systematic literature review and
must align with the research question and the objective of this research. As these are already described on the understanding of the objective and identified the research question presented in Table 1, Next is to develop the protocol for the researcher to follow and guide on which papers are eligible for this study.
Figure 2: Flow chart for the planning review.
5.1 Element of Review
The researcher developed the protocol based on the scope of this study. The protocol is shown in Figure 3. The protocol included the elements of review, aims of the study, research question to be answered, inclusion criteria, search strategies, quality assessment criteria, and screening procedures (Xiao et al., 2019). As per describe, by Xiau et al., (2019), the elements of review in this research focus on a paper that discusses MOOCs and engagement happens in MOOCs platform is the main thing for this study. As engagement is divided into three parts which are behavioral, cognitive, and emotional engagement defined by Deng et al., (2020). Engagement in MOOCs is also being call learner engagement, student engagement, and academic engagement. Therefore, the keyword for the search is “MOOCs”, “Engagement”, “Student Engagement”, “Learner Engagement”, “Academic Engagement”, “Behavioral Engagement”,
“Cognitive Engagement”, “Emotional Engagement” and “dropout rate”. Dropout rate is also being used as a keyword for searching the paper, due to dropout rate is related to engagement in MOOCs. Estrada-Molina & Fuentes-Cancell, (2022), in their studies, mentioned that engagement is a viable way to lower dropout rates. It is challenging for students to stay engaged in the course when instructors do not design engaging learning materials, activities, and tactics that promote learners' interaction, hence a high dropout rate may result (Alemayehu & Chen, 2021). To obtain engagement information, it is a good idea to include the term dropout.
Figure 3: The Protocol
5.2 Aims of the study
The aims of this study as simple as its title are to find the indicator for engagement in log data in MOOCs. The list of engagements indicator which can be captured in log data should be present in form of a table. To achieve the aims, the research question should arise and be answered in these studies. An inquiry that a research endeavor seeks to answer is known as a research question (Mattick et al., 2018). The research question is shown in table 1 and is described briefly there.
5.3 Search keywords, Inclusion and Exclusion Criteria
Table 3 displays a list of all the different keywords and short descriptions created for search purposes, as a result of the related work in these studies. In terms of inclusion criteria, the researcher decided to make sure the paper must mention the keywords as presented in Table 3.
the researcher strictly only reviews papers from 2018 until 2022. The resulting paper from searching using the keywords must be published in the English language to better understand the content However, the review material is only limited to research articles, book chapters, and publications that were pertinent to our major subject. In addition, a paper written other than in English is excluded from the review. The abstract of the paper dint discusses engagement in MOOCs also excluded from the review. It is because to align the protocol decided in Figure 2.
The inclusion and exclusion criteria are presented in Table 4.
Table 3: Keywords and description
Keywords Description
“Engagement” or “Student Engagement” or
“Learner Engagement” or “Academic Engagement” or “Behavioral Engagement”, or “Cognitive Engagement” or “Emotional Engagement” or “dropout rate”
To find the engagement indicator being used by the researcher in this field, all related engagement must be used for searching the research article.
“MOOCs” Engagement can happen anywhere, and this study only focuses on engagement that happened in MOOCs.
Table 4: Inclusion and Exclusion criteria
Inclusion Criteria Exclusion Criteria
1. Only the paper is written in the English language
2. One of the keywords presented in Table 3 is mentioned in the paper.
3. The Research paper is strictly published from 2018 until 2022.
4. The material papers are from research articles, book chapters, and publications related to these studies
1. Papers written in other than English is excluded.
2. Exclude the paper which did not discuss MOOCs and related to Engagement in the abstract.
5.4 Search Strategies
Search strategies are also needed for SLR to search for literature material. The search strategies, in this case, are researcher must form a multiple-word string for searching by combining the keywords provided in Table 3. The keyword “MOOCs” must come together with the keyword
“Engagement”. The examples of formed strings for searching are shown in Table 5. The combined wording string used was “AND” as all keywords are needed and required to find related and suggested research papers. The formed string as shown in Table 5 are used to search for the research paper at online database sources. The reliable online database sources for science computer research papers are shown in Table 6. All keywords are used in those database sources for searching. Any research paper that is already being recorded and found to exist in another search result in other databases source is ignored as it is already being looked up. Before being recorded into Mendeley software for reference storage and future analysis, the researcher's job is to follow the protocol to filter research papers that are already defined and need to be passing the screening procedure.
Table 5: Keyword and Formed Search String
Table 6: Online Database Source
Database Source Website Link
ACM Digital Library https://dl.acm.org
IEEE Xplore https://ieeexplore.ieee.org/Xplore/home.jsp
Science Direct https://www.sciencedirect.com
Web of Science https://clarivate.com/webofsciencegroup
Scopus https://www.scopus.com
5.5 Screening Procedures
The list of research papers that come out as result using the formed search string will be screened one by one by the researcher as it is part of the screening procedure. The procedure at this phase is shown in Figure 4. The first thing first to do is to check the title of the research paper. It is to make sure that the title is still in the context which is mean the title is related to engagement and MOOCs. Next is to read the abstract of the selected research paper. The
Keywords Formed Search String
MOOCs 1. “MOOCs AND Engagement”
2. “MOOCs AND Engagement AND Drop Out”
3. “MOOCs AND Cognitive Engagement”
4. “MOOCs AND Cognitive Engagement AND Dropout”
5. “MOOCs AND Emotional Engagement”
6. “MOOCs AND Emotional Engagement AND Dropout Rate”
7. “MOOCs AND Student Engagement”,
8. “MOOCs AND Student Engagement AND dropout rate”
9. “MOOCs AND Learner Engagement”,
10. “MOOCs AND Learner Engagement AND dropout rate”
11. “MOOCs AND Academic Engagement”, 12. “MOOCs AND Academic Engagement AND
dropout rate”
13. “MOOCs AND Behavioral Engagement”
14. “MOOCs AND Behavioral Engagement AND dropout rate”
Engagement or Student Engagement or Learner Engagement or Academic Engagement or Behavioral Engagement, or Cognitive Engagement or Emotional Engagement or dropout rate
purpose of it is to see the relevance of this research and can find out the answer to the research question. The screening procedure in this study is to make sure that the research paper is good enough to do a quality assessment.
Figure 4: The Screening Procedure
5.6 Quality Assessment
Quality assessment criteria are for the reviewer such as the researcher in this study to execute after the selection of the research paper. As said by Baethge et al., (2019), a quality assessment tool to aid editors in making decisions regarding manuscripts as well as reviewers, readers, and authors in evaluating articles and composing narrative reviews. The quality of the selected research paper must be answered by the quality assessment questions which have been shown in Table 7. Four questions are prepared for the researcher to answer when reviewing the selected research paper. Is also able to make sure researchers know what to do and keep aligned with the objective of this research. Following the sequence, as shown in Table 7, will lead to able to extract the answers from the selected research paper. The final step in preparing the pool of research for data extraction and synthesis is quality assessment, which serves as a fine sieve to polish the full-text articles Xiao & M Watson, (2019).
Table 7: Quality Assessment Questions
Question Index Questions
Q1 Are the main purpose of the study stated clearly?
Q2 Is the paper discussing MOOCs and Engagement?
Q3 Is the research paper answer the research question as shown in Table 1?
Q4 Is the research paper mentioned the engagement indicator in MOOCs?
6. Identify Potential Studies
After the successfully defined protocol, the next step is to Identify the potentially related studies by strictly following the protocol prepared as mentioned above. The search for potential studies can be done using reliable sources, which are electronic, backward searching, and forward- searching. The electronic sources mainly used in this research are shown in Table 6. The total research paper result of searching using formed string as shown in Table 5 is 1352. The detailed total search result is shown in Table 8. Since there is a lot of research papers available over the past 5 years, The inclusion and exclusion criteria take place to narrow down the search result.
There is a lot of duplication since some of the research papers exist in other databases and are presented in the result search list. Those duplications were removed and as a result, it is getting a smaller number of results for the screening procedure. The title was first to be screened and some of the research paper got removed as it out of context and irrelevant to these studies. The abstract of the selected research paper was read to understand better what the study is about and if it really can answer this research question. The result of the screening procedure was then moved to the quality assessment. Narrowing down the search for a research paper is necessary to avoid reading a lot of research papers does not meet the requirement of this
systematic literature review. The summary of the identify potential studies are shown in Figure 5.
Table 8: Total Search Results
Database Source Total Result
ACM Digital Library 145
IEEE Xplore 70
Science Direct 681
Web of Science 30
Scopus 426
Total 1352
Figure 5: Process to identify potential studies
7. Analysing and Synthesizing Selection Research Paper
Quality assessment criteria are executed when analyzing the list of selected research papers. A list of questions needs to be answered for a researcher to synthesize the information. The questions are shown in Table 7. All questions must be answered to qualify for synthesizing information. The first question is to make sure that the aims of the study are mentioned and understood in the selection paper. It is because the aims will tell the objectives of the research and predict what the paper is all about. By reading the full text of the research paper as the aims are clear to researchers, can answer research questions second, third and fourth. By discussing and mentioning MOOCs and engagement the paper are able to answer the second question. It is to make sure the selection paper themes are MOOCs and engagement. Understanding the research questions of this paper is required to answer the third quality assessment question.
The last question for quality assessment is to find out the engagement indicator mentioned and explained in the selected research paper. The result from analysing the list of the selected research paper is shown in Figure 6.
Figure 6: Result from quality assessment
8. Result
The outcome of SLR is a lot of researchers researching about engagement in MOOCs and how to measure the engagement. The purpose of this study is to gather the log data indicator for MOOCs and directly answer the research questions for this research. The result of SLR is presented in form of the table in Table 2. Four main sections are concluded for engagement indicator in MOOCs, which is videos, discussion forum, learning activity, and social learning.
Most of the research paper mention watching videos as part of the engagement in MOOCs. It is because videos in MOOCs are an important part of learning as they can demonstrate to students the knowledge currently being taught. Basic methods for involving students in MOOCs include using visual elements like videos and discussion boards (Bonafini et al., 2017). The discussion board also known as a discussion forum in MOOCs is the basic feature available in all MOOCs platforms. The interaction between peers and instructor are happen here in terms of posting questions and replying to answer questions. Addition from Galikyan et al., (2021), said that a discussion forum offers unique information on the content created by students within a MOOC and is the sole area where students can interact with one another through text. MOOCs are also occupied with the features to upload and complete assignments.
According to the results of SLR, submitting the assignment is part of the engagement.
Moubayed et al., (2020), in their research paper, mentioned that the length of time it takes students to submit assignments is the best indicator of their degree of engagement. Quiz attempts and crossword puzzles are part of the learning activity offered in MOOCs. The MOOCs system will count how many attempts have been made by the learner to achieve all correct for the specific number of questions. It depends on the instructor to put how many questions and also how many attempts can be made to achieve all correct. The last section is social learning, the engagement indicator in this part is commenting on every page available such as videos, lecture notes page, and assessment. The commenting features give the learner the ability to give an opinion and that proves students are engaged with material studies prepared by the instructor.
Table 9: Result engagement indicator in MOOC, synthesis from Literature Review
Engagement Indicator Description Source
Video The total duration of watched videos.
Indicate how long students watch the video by the hour, minute, and seconds
1. Hardt (2022) 2. Ng & Lei (2022) 3. Thi et al. (2022) 4. Tang et al. (n.d.)
The number of videos viewed/watch
Indicate the total number of videos that students already watched
5. Gledson et al. (2021) 6. Deng, R., et al., (2020) 7. Doherty & Doherty
(2019)
8. Deng, R. et al. (2019).
9. Romero-Rodriguez, L.
et al. (2019)
10. Zhang, G et al. (2018).
11. Gledson et al. (2018) 12. Thomas et al. (2018) Discussion
forum
Students reply to the discussion post
To express opinions and answer the questions
1. Almukhaylid &
Suleman (2020) 2. Farrow et al. (2020) 3. Liu et al. (2021) 4. Khalil, M et al. (2016).
5. Zhang, G et al. (2018).
6. Yan et al. (2018) 7. Deng, R. et al. (2019).
8. Nurhadi, N. A et al.
(2021)
9. Yan et al. (2019) 10. Farrow et al. (2019) Posted question Students posted a comment
Learning Activity
Submitted assignment.
Submitted the assessment before the due date.
1. Deng, R. et al. (2019) 2. Soffer & Cohen (2019) 3. Romero-Rodriguez, L.
et al. (2019)
4. Deng, R. et al. (2020) 5. Moubayed et al.
(2020)
6. Ding & Zhao (2020) 7. Belinskiy, A. (2021) Quiz attempts the total sum of attempts for ten quizzes
Crossword puzzles.
Social Learning
Commenting on every learning page
Pages created in MOOC to include lecture notes, video lectures, learning activities, or assessment
1. Shukor, N. A. et al.
(2019)
9. Discussion
As a result, from the SLR shown in table 1.1, the log data indicator in MOOCs mainly has 4 sections, which are video, discussion forum, learning activities, and social learning. The indicators are used for engagement measurement in MOOCs as it is happening on online platforms. Videos are the first section of log data indicators as videos mainly appear as educational material for MOOCs. Most online educational materials are distributed through videos (Cohen et al., 2018). Counting the duration of watched videos and the total how many videos that have been viewed are part of the indicators for engagement. It is because there is engagement and activity when learners keep viewing the same videos which are writing down important notes or key points. To better understand the content and context of the studies is better to repeat the same videos provided in MOOCs. When there is misunderstanding or confusion from the videos, the discussion forum came to play the part. A discussion forum is the second section for log data indicators. Learner post questions to a discussion forum where everybody who registered for the same courses can see and instructors in charge can see how good and how well the understanding and interactions between learners are. Replying to the posted question in a discussion forum, prove that the learners are eager to help others and themself as being monitored by instructors. From that activity itself, the engagement already happen which is post question and got the interaction between other learner and instructors
where the debate between them might be happening. To better understand the teaching materials which are videos and to avoid keep questioning the same issue and having to answer the same question in a discussion forum. It is better to do learning activities to be clear about what has been thought from the videos. Instructors in MOOCs will give the assignment to learners and submit the assignment before the due date is counted as engagement in MOOCs.
This shows that the learner did their homework and submit on time. In addition, quiz given to the learner are an engagement when the count of attempts to correct all prepared questions. The activity for the quiz to be corrected is to make sure knew the wrong and have tried to fix it.
Engagement happens when they try to understand the quiz question and study the courses.
Finally, social learning is the last section for log data indicators. For the social learning section, according to Shukor & Abdullah, (2019). commenting on every learning page can be counted as engagement as learners express their opinion and suggestions from there.
10. Conclusion
The systematic literature review was used to gather log data indicators for identifying learner engagement in MOOCs. The methodology helps researchers to identify the activity happening in MOOCs referring to engagement. MOOCs is different from physical class, where educators can see the engagement happen directly and will directly do everything, they could do to improve teaching based on the engagement while MOOCs is an online platform and the instructor is limited in their power to see the engagement. The result from SLR helps educators to measure and identify the level of engagement happening in their courses.
Reference
Alemayehu, L., & Chen, H. L. (2021). Learner and instructor-related challenges for learners’
engagement in MOOCs: a review of 2014–2020 publications in selected SSCI indexed journals.
Almukhaylid, M., & Suleman, H. (2020). Socially-Motivated Discussion Forum Models for Learning Management Systems. ACM International Conference Proceeding Series, 1–11.
Baethge, C., Goldbeck-Wood, S., & Mertens, S. (2019). SANRA—a scale for the quality assessment of narrative review articles. Research Integrity and Peer Review 2019 4:1, 4(1), 1–7.
Bonafini, F., Bonafini, F., Chae, C., Park, E., & Jablokow, K. (2017). How Much Does Student Engagement with Videos and Forums in a MOOC Affect... Online Learning Journal, 21(4).
Carlon, M. K. J., Keerativoranan, N., & Cross, J. S. (2020). Content Type Distribution and Readability of MOOCs. L@S 2020 - Proceedings of the 7th ACM Conference on Learning @ Scale, 401–404.
Clark, K. R., Vealé, B. L., Watts, L. K., Kr, C., Bl, V., & Watts, L. A. (2017). A review of the use of massive open online courses (MOOCs) in medical imaging education.
Nsuworks.Nova.Edu, 15(2).
Cohen, S. S., Madsen, J., Touchan, G., Robles, D., Lima, S. F. A., Henin, S., & Parra, L. C.
(2018). Neural engagement with online educational videos predicts learning performance for individual students. Neurobiology of Learning and Memory, 155, 60–64.
Collins, C., Dennehy, D., Conboy, K., & Mikalef, P. (2021). Artificial intelligence in information systems research: A systematic literature review and research agenda.
International Journal of Information Management, 60, 102383.
Crues, R. W., Bosch, N., Perry, M., Angrave, L., Shaik, N., & Bhat, S. (2018). Refocusing the Lens on Engagement in MOOCs. Proceedings of the Fifth Annual ACM Conference on Learning at Scale.
Deng, R., Benckendorff, P., & Gannaway, D. (2019). Learner engagement in MOOCs: Scale development and validation. British Journal of Educational Technology, 51(1), 245–
262.
Deng, R., Benckendorff, P., & Gannaway, D. (2020). Linking learner factors, teaching context, and engagement patterns with MOOC learning outcomes. Journal of Computer Assisted Learning, 36(5), 688–708.
Ding, Y., & Zhao, T. (2020). Emotions, engagement, and self-perceived achievement in a small private online course. Journal of Computer Assisted Learning, 36(4), 449–457.
Doherty, K., & Doherty, G. (2019). Engagement in HCI: Conception, theory and measurement.
ACM Computing Surveys, 51(5).
Estrada-Molina, O., & Fuentes-Cancell. (2022). Engagement and Desertion in MOOCs:
Systematic Review. ERIC, 70, 2022.
Farrow, E., Moore, J., & Gašević, D. (2019). Analysing discussion forum data: A replication study avoiding data contamination. ACM International Conference Proceeding Series, 170–179.
Farrow, E., Moore, J., & Gaševic, D. (2020). Dialogue attributes that inform depth and quality of participation in course discussion forums. ACM International Conference Proceeding Series, 129–134.
Galikyan, I., Admiraal, W., & Kester, L. (2021). MOOC discussion forums: The interplay of the cognitive and the social. Computers & Education, 165, 104133.
Geigle, C., conference, C. Z.-P. of the fourth (2017) A., & 2017. (2017). Modeling MOOC student behavior with two-layer hidden Markov models. Dl.Acm.Org, 205–208.
Gledson, A., Apaolaza, A., Barthold, S., Günther, F., Yu, H., & Vigo, M. (2018). Instructor perspectives on comparative heatmap visualizations of student engagement with lecture video. SIGCSE 2018 - Proceedings of the 49th ACM Technical Symposium on Computer Science Education, 2018-January, 251–256.
Gledson, A., Apaolaza, A., Barthold, S., Günther, F., Yu, H., & Vigo, M. (2021).
Characterising student engagement modes through low-level activity patterns.
UMAP 2021 - Proceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization, 88–97.
Guajardo Leal, B. E. , Navarro-Corona, C. , & Valenzuela González, J. R. (2019). Systematic mapping study of academic engagement in MOOC. Erudit.Org, 20(2).
Halverson, L., & CR Graham. (2019). Learner engagement in blended learning environments:
A conceptual framework. ERIC, 16(1).
Hardt, R. (2022). A System to Motivate Sustained Lecture Video Engagement in Small Private Online Courses. Koli Calling ’22: 22nd Koli Calling International Conference on Computing Education Research, 1.
Henrie, C., Halverson, L., & CR Graham. (2015). Measuring student engagement in technology-mediated learning: A review. Elsevier.
Hew, K. F. (2016). Promoting engagement in online courses: What strategies can we learn from three highly rated MOOCS. British Journal of Educational Technology, 47(2), 320–341.
Hew, K., Qiao, C., & Y Tang. (2018). Understanding student engagement in large-scale open online courses: A machine learning facilitated analysis of student’s reflections in 18 highly rated MOOCs. Erudit.Org.
Huang, T., Mei, Y., Zhang, H., Liu, S., & Yang, H. (2019). Fine-grained engagement recognition in online learning environment. ICEIEC 2019 - Proceedings of 2019
IEEE 9th International Conference on Electronics Information and Emergency Communication, 338–341.
Hussain, M., Zhu, W., Zhang, W., & Abidi, S. M. R. (2018). Student Engagement Predictions in an e-Learning System and Their Impact on Student Course Assessment Scores.
Computational Intelligence and Neuroscience, 2018.
Joksimović, S., Poquet, O., Kovanović, V., Dowell, N., Mills, C., Gašević, D., Dawson, S., Graesser, A. C., & Brooks, C. (2018). How do we model learning at scale? A systematic review of research on moocs. Review of Educational Research, 88(1), 43–86.
KF Hew. (2015). Towards a model of engaging online students: lessons from MOOCs and four policy documents. Ijiet.Org.
Kőrösi, G., & Farkas, R. (2020). MOOC Performance Prediction by Deep Learning from Raw Clickstream Data. Communications in Computer and Information Science, 1244 CCIS, 474–485.
Liao, B., Ali, Y., Nazir, S., He, L., & Khan, H. U. (2020). Security analysis of IoT devices by using mobile computing: a systematic literature review. Ieeexplore.Ieee.Org.
Liu, Z., Mu, R., Liu, S., Peng, X., & Liu, S. (2021). Modeling Temporal Association of Cognition-Topic in MOOC Discussion to Track Learners’ Cognitive Engagement Dynamics. Conference of the South African Institute of Computer Scientists and Information Technologists 2020.
Mattick, K., Johnston, J., & de la Croix, A. (2018). How to write a good research question.
Wiley Online Library, 15(2), 104–108.
Min, L., & Foon, H. K. (2019). Self-Regulated Learning Process in MOOCs: Examining the Indicators of Behavioral, Emotional, and Cognitive Engagement. Proceedings of the 2019 4th International Conference on Distance Education and Learning.
Mongkhonvanit, K., Kanopka, K., & Lang, D. (2019). Deep knowledge tracing and engagement with MOOCs. ACM International Conference Proceeding Series, 340–
342.
Moubayed, A., Injadat, M., Shami, A., & Lutfiyya, H. (2020). Student engagement level in an e-learning environment: Clustering using k-means. Taylor & Francis, 34(2), 137–
156.
Ng, K., & Lei, P. (2022). A Lightweight Method using LightGBM Model with Optuna in MOOCs Dropout Prediction. 2022 6th International Conference on Education and Multimedia Technology, 1, 53–59.
Paton, R. M., Fluck, A. E., & Scanlan, J. D. (2018). Engagement and retention in VET MOOCs and online courses: A systematic review of literature from 2013 to 2017. Computers
& Education, 125, 191–201.
Poquet, O., Lim, L., Mirriahi, N., & Dawson, S. (2018). Video and learning: A systematic review (2007-2017). ACM International Conference Proceeding Series, 20(6), 151–
160.
Romero-Rodriguez, L., & MS Ramirez-Montoya. (2019). Gamification in MOOCs:
Engagement application test in energy sustainability courses. Ieeexplore.Ieee.Org.
Shukor, N. A., & Abdullah, Z. (2019). Using learning analytics to improve MOOC instructional design. Learntechlib.Org. https://doi.org/10.3991/ijet.v14i24.12185 Soffer, T., & Cohen, A. (2019). Students’ engagement characteristics predict success and
completion of online courses. Journal of Computer Assisted Learning, 35(3), 378–
389.
Sooryanarayan, D. G., & Gupta, D. (2015). Impact of learner motivation on MOOC preferences: Transfer vs. made MOOCs. 2015 International Conference on Advances in Computing, Communications and Informatics, ICACCI 2015, 929–934.
Stracke, C. M. (2017). Why We Need High Drop-Out Rates in MOOCs: New Evaluation and Personalization Strategies for the Quality of Open Education. Proceedings - IEEE 17th International Conference on Advanced Learning Technologies, ICALT 2017, 13–15.
Stracke, C. M., Tan, E., Moreira Texeira, A., Texeira Pinto, M. D. C., Vassiliadis, B., Kameas, A., & Sgouropoulou, C. (2018). Gap between MOOC Designers’ and MOOC Learners’ Perspectives on Interaction and Experiences in MOOCs: Findings from the Global MOOC Quality Survey. Proceedings - IEEE 18th International Conference on Advanced Learning Technologies, ICALT 2018, 1–5.
Swinnerton, B., Hotchkiss, S., & Morris, N. P. (2017). Comments in MOOCs: who is doing the talking and does it help? Journal of Computer Assisted Learning, 33(1), 51–64.
Tang, W., Chen, Y., & Guan, B. (n.d.). An Analysis of the Characteristics of Teacher Talk on MOOCs Based on Interpersonal Function Take the “National Excellent Course” of English Writing as an Example KEYWORDS. 2022 6th International Conference on Education and Multimedia Technology, 1.
Thi, N., Ho, T., Thi, A., & Pham, V. (2022). Student Perceptions towards Positive and Negative Aspects of MOOC-based Blended Learning. 2022 6th International Conference on Education and Multimedia Technology, 1, 20–26.
Thomas, C., Nair, N., & Jayagopi, D. B. (2018). Predicting Engagement Intensity in the Wild Using Temporal Convolutional Network. 604–610.
Tseng, S. F., Tsao, Y. W., Yu, L. C., Chan, C. L., & Lai, K. R. (2016). Who will pass?
Analyzing learner behaviors in MOOCs. Research and Practice in Technology Enhanced Learning, 11(1), 1–11.
Vellukunnel, M., Buffum, P., Boyer, K. E., Forbes, J., Heckman, S., & Mayer-Patel, K. (2017).
Deconstructing the discussion forum: Student questions and computer science learning. Proceedings of the Conference on Integrating Technology into Computer Science Education, ITiCSE, 603–608.
Wang, W., Guo, L., He, L., & Wu, Y. J. (2018). Effects of social-interactive engagement on the dropout ratio in online learning: insights from MOOC.
Xiao, Y., and, M. W.-J. of P. E., & 2019, undefined. (2019). Guidance on conducting a systematic literature review. Journals.Sagepub.Com, 39(1), 93–112.
Xiao, Y., & M Watson. (2019). Guidance on conducting a systematic literature review.
Journals.Sagepub.Com, 39(1), 93–112.
Yan, W., Dowell, N., Holman, C., Welsh, S. S., Choi, H., & Brooks, C. (2018). Unpacking the relationship between discussion forum participation and learning in MOOCs:
Content is key. ACM International Conference Proceeding Series, 330–339.
Yan, W., Welsh, S. S., Dowell, N., Choi, H., Holman, C., & Brooks, C. (2019). Exploring learner engagement patterns in teach-outs. ACM International Conference Proceeding Series, 180–184.
