Evaluating learning management systems
Adoption of hexagonal e-learning assessment model in higher education
Sevgi Ozkan
Brunel University, Uxbridge, UK and
Informatics Institute, Middle East Technical University, Ankara, Turkey, and
Refika Koseler and Nazife Baykal
Informatics Institute, Middle East Technical University, Ankara, Turkey
Abstract
Purpose– The purpose of this paper is to demonstrate the importance of undertaking a systemic view of learning management systems (LMSs) evaluation addressing the conceptualization and measurement of e-learning systems success in higher education.
Design/methodology/approach– The paper adopts a quantitative case perspective and derives a conceptual model for e-learning assessment (Hexagonal e-learning assessment model – HELAM). The model is empirically tested for validity and reliability in the university setting.
Findings– Qualitative and quantitative findings have been presented, which will be valuable for academics and practitioners doing research in e-learning evaluation. The findings support the flexibility and relevance of HELAM as an e-learning assessment model. It highlights a number of success measures which are grouped under six dimensions.
Research limitations/implications– Further research efforts should explore new dimensions or test the causal relationships among proposed dimensions within the boundary of e-learning. In that, the paper is limited contextually where attention should be made not to generalize the findings beyond the empirical findings within the case analysis.
Practical implications– The paper supports a practitioner perspective through a consideration of a holistic approach to e-learning assessment. E-learning system developers may find the findings useful when designing and implementing the LMS.
Originality/value– The paper is original as the conceptual model has been derived through both theoretical constructs and empirical analysis. It provides an innovative approach to e-learning assessment.
KeywordsE-learning, Information systems, Statistical analysis Paper typeResearch paper
1. Introduction
E-learning has become one of the most significant developments in the information systems (IS) industry (Wang, 2003). According to New Report by Global Industry Analysts, Inc., it has been claimed that e-learning has emerged as an imperative tool to
The current issue and full text archive of this journal is available at www.emeraldinsight.com/1750-6166.htm
This paper is based on the submitted paper by the authors for European and Mediterranean Conference on Information Systems EMCIS 2008, 25-26 May, Al Bostan Rotana, Dubai, UAE, ISBN: 978-1-902316-58-1. For more details about EMCIS please visit: www.emcis.org
Learning management systems
111
Received 25 January 2008 Revised 10 May 2008, 19 December 2008 Accepted 17 January 2009
Transforming Government: People, Process and Policy Vol. 3 No. 2, 2009 pp. 111-130 qEmerald Group Publishing Limited 1750-6166 DOI 10.1108/17506160910960522
impart knowledge in the academic as well as corporate sectors. Since e-learning has several advantages in terms of cost reduction, simplified training programs, flexibility, and convenience; it is poised to become an integral component of information dissemination, and emerges as the new paradigm of modern education. Backed by several favourable trends, the world e-learning market is projected to exceed US$52.6 billion by 2010 (Global Industry Analysts, 2008). The role of e-learning and information technologies (IT) in education continues to expand in scope and complexity. Moreover, the increasing usage of internet motivates many researchers to improve internet technologies as well as web-based applications. However, the development of e-learning systems is quite a challenging issue for both schools and industry. Both IS researchers and education professionals face numerous difficulties in theoretical and methodological concepts. Little is known about why many users stop their online learning after their initial experience (Sunet al., 2008). As a consequence, the need to develop theories and criteria for judging e-learning success becomes essential to achieve efficient systems.
Increasing effectiveness of the e-learning systems has become one of the most practically and theoretically important research areas in both educational engineering and IS fields (Lee and Lee, 2008). Additionally, the importance of measuring IS success in e-learning applications increases as the investments in e-learning systems increase. Before heading an e-learning system investment, there is an indispensible need for assessing the success of the e-learning systems.
For e-learning applications to be used efficiently for educational purposes; there is an important need to measure the success and effectiveness of the e-learning system systematically (Global Industry Analysts, 2008). Since integration of the Information Communication Technologies into the teaching and learning process has become inevitable especially by the higher education institutions, and assessing the e-learning systems is the only way to ensure that higher education programs delivered via technology are of high quality, developing theories and criteria for judging e-learning success become essential.
Although, there is an important necessity for such an evaluation of every e-learning system, past studies show that there is a gap in evaluation of e-learning environments’
effectiveness in between theoretical level and application level. While a considerable amount of research has been conducted on IS success models (DeLone and McLean, 1992, 2003; Raiet al., 2002; Seddon, 1997), little research has been carried out to address the conceptualization and measurement of e-learning systems success within educational organizations. Traditional IS success models extended to assessing e-learning systems success is rarely addressed (Wanget al., 2007).
This research aims to enhance understanding in defining, evaluating, and promoting e-learning success from an IS perspective. The paper proposes a new e-learning effectiveness evaluation model developed from several existing IS and e-learning evaluation methods (Section 2). In that regard, the objective of this study is to develop a conceptual model to systematically evaluate the success of LMS with respect to learners’ perceived satisfaction. In addition, another objective of this research is to contribute in the ongoing research of online education.
Based on the objectives, in this research study, a number of e-learning success factors and criteria have been identified. Each significant e-learning success assessment aspect and criteria have been combined in this e-learning effectiveness evaluation tool, which altogether have formed a comprehensive e-learning success assessment model. In order
TG 3,2
112
to validate the proposed model, a number of validity tests have been applied. The model has been applied to the LMS (namely NET-Class) used in a higher education institution in Turkey – Middle East Technical University (METU). The case study was undertaken in the context of the various courses under the Informatics Online (ION) programme currently being offered at METU Informatics Institute at the postgraduate level.
The structure of the paper is as follows: Section 2 reports background information.
Section 3 focuses on conceptual model of e-Learning systems evaluation, and Section 4 comprises research methodology of this research, Section 5 presents the findings and discussions and conclusions are presented in Section 6.
2. Background and literature review
This section provides a review of the pertinent literature and in three sub-sections (Sections 2.1-2.3).
2.1 Terminology
In this paper; e-learning refers to the use of electronic devices for learning, including the delivery of content via electronic media such as internet/intranet/extranet, audio or video tape, satellite broadcast, interactive TV, CD-ROM, and so on (Kaplan-Leiserson, 2000).
Today many writers use the terms “LMS” “e-learning,” “online learning,” and
“web-based learning (WBL)” interchangeably, and that approach will be taken in this paper (Hayenet al., 2004).
Asynchronous learning refers to learn in which interaction between instructors and learners occurs intermittently with a time delay. Examples are self-paced courses taken via the internet or CD-ROM, Q&A mentoring, online discussion groups, and email (Kaplan-Leiserson, 2000).
Synchronous learning refers to a real-time, instructor-led online learning event in which all participants are logged on at the same time and communicate directly with each other. In this virtual classroom setting, the instructor maintains control of the class, with the ability to “call on” participants. In most platforms, learners and teachers can use a whiteboard to see work in progress and share knowledge. Interaction may also occur via audio- or video-conferencing, internet telephony, or two-way live broadcasts (Kaplan-Leiserson, 2000).
Blended learning refers to learn events that combine aspects of online and face-to-face instruction. (Kaplan-Leiserson, 2000).
Educational engineering refers to full educational services with strategic consulting, curriculum and coursework design, and custom courseware development.
Learning management system (LMS) is a broad term used for a wide range of systems that organize and provide access to online learning services for learners, teachers, and administrators (MIT-OKI Open Knowledge Initiatives, 2003).
2.2 IS success models
E-learning system is a special type of IS (Wanget al., 2007). In this section a review of the IS success and IS measurement literature has been presented. In regard to IS success, the DeLone and McLean IS Success Models are the most cited IS success models (Gableet al., 2003; Myerset al., 1998; Heo and Han, 2003).
Learning
management
systems
113
In 1992, DeLone and McLean presented an IS Success Model as a framework and model for measuring the complex dependent variable in IS research. After ten years (2002), they published a paper and discussed many of the important IS success research contribution, they propose enhancements to their original 1992 IS Success Model. They made a series of recommendations regarding current and future measurement of IS success. Their IS Success Model become very effective and it is used not only in the IS success but also other related fields like e-learning assessment models.
2.3 Prior studies of e-learning assessment
2.3.1 E-learning success models by Holsapple and Lee-Post. Holsapple and Lee-Post’s e-learning success model is adapted from DeLone and McLean’s (2003) updated IS Success Model which, in turn, is an extension of their original model (DeLone and McLean, 1992). DeLone and McLean identified six dimensions of success factors: system quality, information quality, use, user satisfaction, individual impact, and organizational impact. These were incorporated into their original overall success model shown in Figure 1.
Holsapple and Lee-Post the e-learning Success Model makes the process approach explicit to measure and assess success. Their model also includes success metrics developed specifically for the e-learning context being investigated. They use the process approach to posit that the overall success of e-learning initiatives depends on the attainment of success at each of the three stages of e-learning systems development:
design, delivery, and outcome analysis. Success of the design stage is evaluated along three success factor dimensions: system quality, information quality, and service quality.
Success of the delivery stage is evaluated along two success factor dimensions: use and user satisfaction. Finally, success of the outcome stage is evaluated along the net benefits dimension. The arrows shown in the Figure 2 depict the interdependences within the three stages of success assessment. Success of system design is essential to the success of system delivery, which, in turn, affects the success of system outcome.
The success of system outcome, however, has an impact on the success of subsequent system delivery, as indicated by the double arrow linking system delivery and outcome stages (Holsapple and Lee-Post, 2006).
2.3.2 E-learning critical success factors: an exploratory investigation of learner perceptions. According to Selim (2007), e-learning can be integrated into many university programmes. There are several factors that need to be considered to identify and measure e-learning applications’ critical success factors (CSFs) from learner perceptions.
Four CSFs were identified and measured, namely, instructor characteristics, learner characteristics, technology infrastructure and university support. They tested learner’s attitude towards using e-learning. A sample of 37 class sections with 538 responses was
Figure 1.
DeLone and McLean’s IS success model
Information quality
System quality
Service quality
Source: DeLone and McLean (2003)
Intention to use/use
User satisfaction
Net benefits
TG 3,2
114
used to validate the proposed e-learning CSFs. The results revealed that learners perceived instructor characteristics as the most critical factor in e-learning success, followed by IT infrastructure and university support. Learner characteristics were perceived as the least critical factor to the success of e-learning (Selim, 2007) (Figure 3).
2.3.3 An empirical examination of factors contributing to the creation of successful e-learning environments. According to Johnson et al. (2008) most of e-learning assessment models do not take into account the importance of social presence. Briefly, their study extends previous research by developing a model of e-learning effectiveness which adds social presence.
2.3.4 Surveying instructor and learner attitudes toward e-learning. According to Liaw et al. (2007), there is minimal research on instructors’ and learners’ attitudes toward in e-learning environments. The purpose of their study is to explore instructors’
and learners’ attitudes toward e-learning usage. Their research results instructors have very positive perceptions toward using e-learning as a teaching assisted tool.
Figure 2.
The e-learning success model and sample metrics System design
System quality 1. Easy-to-use 2. User friendly 3. Stable 4. Secure 5. Fast 6. Responsive Information quality 1. Well organized 2. Effectively presented 3. Of the right length 4. Clearly written 5. Useful 6. Up-to-date Service quality 1. Prompt 2. Responsive 3. Fair 4. Knowledgable 5. Available
System outcome Net benefits Positive aspects 1. Enhanced learning 2. Empowered 3. Time saving 4. Academic success Negative aspects 1. Lack of content 2. Isolation 3. Quality concerns 4. Technology dependence System delivery
Use
1. Powerpoint slides 2. Audio
3. Script
4. Discussion board 5. Case study 6. Practice problems 7. Exceltutorials 8. Assignments 9. Practice exam
User satisfaction 1. Overall satisfaction 2. Enjoyable experience 3. Overall success 4. Recommend to others
Source: Holsapple and Lee-Post (2006)
Figure 3.
E-learning CSFs:
an exploratory investigation of learner perceptions Critical success
factors
Instructor characteristics Student
characteristics
Technology infrastructure
University support Source: Selim (2007)
Learning
management
systems
115
Furthermore, behavioral intention to use e-learning is influenced by perceived usefulness and self-efficacy. Regarding to learners’ attitudes, self-paced, teacher-led, and multimedia instruction are major factors to affect learners’ attitudes toward e-learning as an effective learning tool. Additionally, personal attitudes are a major factor to affect individual usage of IT. In other words, understanding users’ attitudes toward e-learning facilitates the creation of appropriate e-learning environments for teaching and learning. Essentially, methods of assessing e-learning cannot be evaluated using a single linear methodology. Thus, there is a need to build a multidisciplinary approach to survey individual attitudes toward e-learning (Wang, 2003) (Figure 4).
2.3.5 An empirical examination of factors contributing to the creation of successful e-learning environments by Richard D. Johnson, Steven Hornik, and Eduardo Salas.
According to Liawet al.(2007), in order to design effective e-learning environments, three constraints have to be provided: learner characteristics, instructional structure, and interaction. They also stated that in developing e-learning, it is necessary to understand target group. First, learner characteristics, such as attitudes, motivation, belief, and confidence need to be identified (Passerini and Granger, 2000). Essentially, e-learning signifies autonomous learning environments. In other words, users have more opportunities for self-directed learning in e learning environments. As for instructional structure, multimedia instruction enables learners to develop complex cognitive skills, such as understanding important elements of conceptual complexity, ability to use acquired concepts for reasoning and inference, and competence to apply conceptual knowledge to novel situations with flexibility (Spiro et al., 1995). Finally, e-learning environments offer group interaction, such as learners to learners, or learners to instructors. Group interaction is a kind of cooperative learning that helps learners to make progress through their zone of proximal development by the activities in which they engage (Vygotsky, 1978). When learners increase their interaction with instructors and learners, they in turn raise their chances of building their own knowledge because much of learning inevitably takes place within a social context, and the process includes the mutual construction of understanding (Bruner, 1971).
Figure 4.
E-learning effectiveness model
Human dimension
Design dimension
E-learning effectiveness
• Course instrumentality
• Course performance
• Course satisfaction AS-CSE-Application-specific
computer self-efficacy
Social presence Interaction
Perceived usefulness AS-CSE
Source: Johnson et al. (2008)
TG 3,2
116
3. Conceptual model of e-learning systems evaluation 3.1 Hexagonal e-learning assessment model (HELAM)
Hexagonal e-learning assessment model (HELAM) is a conceptual e-learning success evaluation model for assessing learner satisfaction with both internet-based LMSs and blended learning. HELAM has been developed for assessing the e-learning effectiveness according to six-dimensions of e-learning:
(1) Technical issues:system quality.
(2) Technical issues:service quality.
(3) Technical issues:content quality.
(4) Social issues:learner perspective.
(5) Social issues:instructor attitudes.
(6) Supporting issues (Figure 5).
User satisfaction is the main point of the evaluation.
Although there are other e-learning success assessment models in the literature, most of them do not take into account the importance of every aspects of an e-learning systems as a whole. For instance, one of the e-learning success assessment models focus on the learner perception only, the other one pays attention to evaluate system quality of the e-learning, another one is interested in evaluation of instructor attitudes, or others give importance to information quality in e-learning success assessment.
In contrast to existing models, this research study aims to combine all these important aspects of e-learning success assessment models, and develop a comprehensive e-learning success assessment model.
Figure 5.
Hexagonal e-learning assessment model (HELAM)
D. Technical issues: system quality D1. Easy to use D6. User friendly
D2. Security D7. Well organized
D3. Stable D8. Fast
D4. Help option available D5. Support educational tools
E. Technical issues: information (Content) quality E1. Curriculum management
E2. Course flexibility E3. Interactivity
E9. Assignments
E4. Practice exam
E10. Tutorial
E5. Case studies
E11. Up-to-date
E6. Clearly written
E12. Exams
E7. Slides
E13. Right length
E8. Content quality
E14. Well organized B. Social issues: learner perspective
B1. Learner attitudes toward LMS B2. Learner’s computer anxiety B3. Self efficiency B4. Enjoyable experience
B5. Interaction with other students and teacher
F. Technical issues: service quality F1. Student tracking
F2. Course/Instruction authorization F3. Instructional design tools F4. Course management F5. Knowledgeable F6. Security
A. Supportive issues A1. Environment
A2. Trends A3. Ethical issues A4. Personalization
C. Social issues: instructor attitudes C1. Responsiveness
C2. Enjoyment C3. Availability C4. Self-efficiency C5. Promptness C6. Usefulness C7. Fairness
C8. Communication ability
C9. Encouraging interaction between students
E-Learning Hexagonal model To evaluate modern
training systems
Learning
management
systems
117
3.2 Categories of HELAM
There are many ways to classify the measures available for evaluating an e-learning system. The proposed model constitutes 46 LMS success criteria for evaluation. The criteria are grouped into six main categories in accordance with each criterion correlation (Figure 5).
These six main categories are as follows:
(1) Technical issues:system quality (eight criteria).
(2) Technical issues:service quality (six criteria).
(3) Technical issues:content quality (14 criteria).
(4) Social issues:learner perspective (five criteria).
(5) Social issues:instructor attitudes (nine criteria).
(6) Supporting issues (four criteria).
Similar to the model proposed by Holsappleet al.(2006); the overall success of e-learning initiatives depends on the attainment of success at each of these stages of e-learning systems development. In this research, what is trying to be done is to underline the importance of each aspect of an LMS which directly effects the overall success and learner’s satisfaction. Namely, in this model, LMS is divided into subcategories and these subcategories are evaluated individually; and the overall LMS success can be defined as the cumulative sum of success levels of all these individual parts.
Although, 46 criteria are grouped into 6 categories in HELAM; similar to Holsapple et al.(2006), the model can also be divided into two main process levels. Success of the design stage (I) and success of the delivery stage (II). Success of the design stage is then to be evaluated along two success factor dimensions: system quality and information quality. Success of the delivery stage is evaluated along three success factor dimensions: service quality, learner’s satisfaction and instructor’s satisfaction. In addition to these, some supportive issues also effects the user perception of LMSs such as trends or certification/diploma.
The details of six categories and 46 criteria of the HELAM are as shown in Figure 5 – HELAM. This Hexagonal e-learning Success Model proposed here has been adapted from different IS and e-learning success evaluation systems models. Table I presents a list of the models that have been briefly described in this section, which have formed the theoretical basis for HELAM. This model is not LMS specific and is applicable to various e-learning IS.
4. Research methodology 4.1 Data collection
This research study is based on the conceptual model, HELAM, focusing on understanding the learners’ perceived satisfaction from an e-learning environment.
Data has been gathered from survey questions, informal interviews with learners (i.e. students).
The survey instrument has been used to collect data from learners about their perceptions of the impact of the online learning environment in regards to their benefits and satisfaction level. The questionnaire has been developed from the literature after a comprehensive survey of various validated e-learning effectiveness models as depicted in Table I. In the survey, there are 73 questions, which were
TG 3,2
118
partitioned into meaningful small units. The survey has two main parts: first part aims to gather generic data about the learners, and the second part about learners’
METU-Online LMS experiences. The second part is divided into six sections each of which corresponds to HELAM’s categories. The questions that belong to the first five categories were five point likert-type scale items. Each question anchored from 1 to 5, where 1 indicates strong disagreement and 5 indicates strong agreement.
The questions that belong to the sixth category, namely supportive issues, were open-ended questions asked during one-to-one and focus group interviews. The information gathered from these interviews constituted the qualitative findings of the study. All responses were guaranteed confidentiality. The questionnaire is presented in the Appendix.
4.2 Participants
A total of 42 individuals participated in the study (90 percent of the class). Out of those who participated in the study, usable data was obtained from 40. The sample consisted of 31 males and nine females. The average age was 26.7 (SD146.7). All of those participating in the course indicated that they had previous computer and Internet experience, with over 90 percent indicating that they had high levels of experience in both. Geographical diversity has been found to be not an indicator to prefer distance education. It is been found that nearly half of the learners (47.8 percent) live near METU Campus. The other half is living far away from METU.
Learners approximately spend their 32.6 h on computers per week. These 32.6 h includes mail checking, chatting, surfing on the internet, etc. Learners approximately spend 5.2 h on computers for educational purposes and they only spend their 1.5 h on LMS, per week.
4.3 Data analysis
To examine the data, statistical and qualitative analyses methods have been used.
Descriptive statistics were run to analyze the collected data. The responses to the questionnaire were analyzed using the Statistical Package for the Social Sciences 15.0 Windows software program.
References Dimensions
E-learning success models by Holsapple and Lee-Post (2006)
System quality, system usage, services, content quality
Liaw (2007) Instructor and learner attitudes toward e-learning
Wang and Liao (2007) Technical quality, system related hypothesis Islas (2007), three-dimensional model to evaluate
modern training systems
Service quality, content quality and system quality
Johnsonet al.(2008) Factors contributing to the creation of successful e-learning environments
E-learning CSFs Selim (2007) Learner attitudes toward e-learning systems Measuring e-learning systems success in an
organizational context: scale development and validation, Wanget al.(2007)
Service quality, content quality, user satisfaction, and system quality
Teaching effectiveness in technology-mediated distance learning, Webster and Hackley (1997)
System quality, learners’ perspective, information quality, instructor quality
Table I.
Reference models and their associated dimension(s) with HELAM
Learning
management
systems
119
4.4 Measures
In order to validate HELAM, a number of validity and reliability tests have been applied. First, content validity was applied to be sure about the survey’s ability to adequately measure or represents the content of the property that the researcher wishes to measure. Second, factor analysis of the survey was identified. Factor analysis is a statistical technique used to determine the number of components in a set of data.
These components are then named according to their characteristics. Finally, reliability test were applied to the survey to check overall consistency.
4.4.1 Content validity. The researchers of the study referred to content validity when discussing the validity of the questions within the evaluation tool. The aim was to conduct a content validity based on the extent to which the measurement reflects the specific intended domain of content (Carmines and Zeller, 1994) A total number of ten experts in the field of IS and educational technology have been asked to assess whether each dimension in the model is “essential,” “useful but not essential,” or “not necessary”. Four of the experts are from the Information Systems Evaluation and Integration Group (ISEing), Brunel University, London, UK; two from Learning and Teaching Development Unit, Brunel University, London, UK; two from the METU, Informatics Institute, Ankara, Turkey, and two from the Computer Education and Instructional Technology Department, METU, Ankara. HELAM had been revised and developed iteratively according to the feedback gained from this panel of individual experts before it reached to the stage to be used in this study.
4.4.2 Identifying factor structure. Before the factor analysis; from the correlation we scanned for the significance correlation values and looked for any item for which the majority of the values are greater than theavalue 0.05. Then the researchers searched for any value among the correlation coefficients that are greater than 0.9 to avoid any possible multicollinearity problem. Checking the determinant of the correlation matrix is another way of detecting multicollinearity problem. In our case the majority of the correlation coefficients are significant along with values smaller than 0.9 and a determinant greater than the necessary value of 0.00001, therefore all the questions left[1] in our questionnaire correlate reasonably well and none of them is particularly large not leading to multicollinearity or singularity.
Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy and Bartlett’s test of Sphericity was conducted and a KMO value of 0.81 was found. A value close to 1 indicates that of correlations are relatively compact and so factor analysis should yield distinct and reliable factors (Kaiser, 1974). Therefore, we can make an inference such that there are separate differences between factors’ correlations indicating having an adequate sample along with distinct reliable factors. In addition we have a significant test with x2value of 806.57 with 17 degrees of freedom and a significance value of 0.000 therefore we can conclude there are non-zero correlations between variables hence factors exist.
Explanatory factor analysis was conducted to primarily establish the factor structure of the model. In order to investigate the internal consistency of the subscales of the survey Croanbachacoefficient was examined. Descriptive statistics were used to present central tendency and variability. In order to decide the number of factors we used Screen plot and Eigenvalues greater than 1 criteria (Tabachnick and Fidell, 2001).
It was found that items were loaded on expected factors named as: instructor attitude (Factor 1): information content quality (Factor 2), system quality (Factor 3) – and
TG 3,2
120
service quality (Factor 4), learners’ attitude (Factor 5). Total explained variance by running the data was 69.3 percent. This percentage is high enough to consider HELAM questionnaire as successful.
Rotation optimizes the factor structure as a result the relative importance of the five factors is equalized and construction Rotated Component Matrix we have reached more homogenously distributed factor loadings of items among five factors (Table II).
4.4.3 Reliability. In order to determine the reliabilities of the factors and to assess the internal consistency of the factors; we used Cronbach’sa. All the factors have high values of Cronbach’sathat can be seen from Table II, all of which are around 0.8 being close to 1. Since Cronbach’saevaluates how well the items of a factor measure a single unidimensional latent construct; a high value closer to 1 indicates that the items consisting the factor can measure the same underlying structure meaning they form a reliable factor and they are consistent in between the other items in the factor.
5. Findings and discussion
Descriptive statistics of each HELAM category are depicted in Table III. These include mean, maximum, minimum values and standard deviations. Based on the descriptive
Factors Item
Factor loadings
Factor reliability
% Total variance explained
Factor 1: instructor quality F1.1 C.3 0.898 0.8568 28.644
F1.2 C.2 0.810
F1.3 C.4 0.780
F1.4 C.5 0.633
F1.5 C.11 0.683
F1.6 C.6 0.449
Factor 2: information content quality F2.1 E.14 0.840 0.9068 17.727
F2.2 E.2 0.736
F2.3 E.16 0.858
F2.4 E.9 0.705
F2.5 E.6 0.703
F2.6 E.13 0.765
F1.7 E.3 0.554
F1.8 E.15 0.463
Factor 3: system quality F3.1 D.14 0.830 0.6216 8.491
F3.2 D.4 0.717
F3.3 D.13 0.783
F3.4 D.10 0.783
F3.5 D.12 0.784
Factor 4: service quality F4.1 F.9 0.705 0.8496 7.859
F4.2 F.6 0.819
F4.3 F.3 0.718
F4.4 F.2 0.755
F4.5 F.5 0.666
F4.6 F.8 0.518
Factor 5: learners’ attitude F5.1 B.3 0.693 0.8545 6.579
F5.2 B.4 0.646
F5.3 B.11 0.641
Total 69.3
Table II.
Factor analysis of HELAM questionnaire
Learning
management
systems
121
statistical data presented in Table III, just by considering the total e-learning environment quality results, (mean value 3.69), learners’ perceived satisfaction from METU-Online LMS is high. The data in this table additionally proves that learners’
perceived satisfactions from METU-Online LMS for each HELAM category are high.
The findings presented below are a blend of descriptive statistics individual question percentages, focus group interviews and one-to-one interviews with learners supported by literature.
In the interviews with learners, one of the questions was: “if e-learning applications were much more popular than today’s popularity, would you prefer continue your undergraduate education with distance education?” More than half of the responded positively express that popularity was an important factor to preferences of an online learning program:
Finding 1. User is positively influenced by the popularity (trend) of LMS and distance learning application in the environments.
Learners’ opinions regarding technology and distance learning may have significant effects on the success of the LMS (Webster and Hackley, 1997). Researchers have generally argued that the successful implementation of any new technology depends on factors related to users’ attitudes and opinions (Davis et al., 1989). Therefore, we suggest that attitudes and the perceived usefulness toward an LMS should be included as important success factors for distance learning. Furthermore, it is also important to include learners’ perceptions of the relative advantage (Moore and Benbasat, 1991) for evaluating distance learning success. When learners exhibit more positive attitudes toward e-learning, then they have more behavioral intentions to use it (Liaw et al., 2007). Thus, as individuals’ attitudes on e-learning become more positive, they will have greater behavioral intention to use it. The concept of attitude towards computers has gained recognition as a critical determinant in the use and acceptance of computer technology (Liaw, 2002; Smith et al., 2000). Learners’ perceptions of usefulness are related to effective reactions to the learning environment (Johnsonet al., 2008).
According to descriptive statistics of this study, learners have highly positive attitudes (Table III: learner attitudes, mean: 3.56) toward e-learning that included perceived self-efficacy, enjoyment, usefulness, and behavioral intention of use.
Learners found METU-Online LMS tool efficient overall:
Finding 2. Learner attitudes toward computers will positively influence perceived e-learner satisfaction from LMS.
Finding 3. User satisfaction will positively influence perceived net benefits in the LMS context.
Dimensions Minimum Maximum Mean SD
Information content quality 1.25 4.63 3.5162 0.61188
Instructor quality 1.83 5.00 3.8643 0.65278
System quality 2.20 4.20 3.5784 0.38358
Learners’ attitudes 1.00 5.00 3.5636 0.67420
Service quality 1.00 4.50 3.7000 0.58128
Total e-learning environment quality 2.87 4.18 3.6964 0.2526
Table III.
Descriptive statistics:
mean, standard deviation, maximum and minimum values for HELAM dimensions
TG 3,2
122
Finding 4. The quality and manipulation of e-learning system is positively related with users’ perceived enjoyment toward e-learning system.
According to the reinforcement theory developed by Scott (1959) and Kohlberg (1966), an attitude may be regarded, like a habit, as an implicit anticipatory response which mediates overt behaviours, and arises out of them through response reinforcement. So reinforcement is positively and highly related with attitude. In distance learning, the most valuable reinforcement component will be a valid certificate or diploma. METU is one of the most prestigious universities in Turkey. Therefore, its diploma is very effective and valid, which effects learner’s attitudes to distance learning program positively:
Finding 5. Providing a valid/authoritative certification/diploma, influence learner positively.
According to the statistical results, the program is preferred by 56.5 percent of the learners. These learners define themselves as self-disciplined and self-motivated learner. A total of 30.4 percent of these learners think that face-to-face education is better than distance education in learning process. It can be concluded that self-disciplined learner’s prefer distance education:
Finding 6. Learner’s study habits are positively related with learner’s perceived effectiveness from an LMS.
Collis (1995) remarked that the instructor plays a central role in the effectiveness of online delivery: “It is not the technology but the instructional implementation of the technology that determines the effects on learning”. Webster and Hackley, 1997 suggested three instructor characteristics:
(1) attitude towards technology;
(2) teaching style; and (3) control of the technology.
That influence the learning objectives of distance education program. According to our descriptive statistical information, learners have highly positive attitudes (Table III:
instructional quality, mean: 3.86) toward instructors. Instructors respond to them rapidly, teaching style is good enough, explanations are clear and, etc. Additionally, nearly all of the e-learners (95.7 percent) satisfied from instructors’ self-efficiency and attitudes. This affects the overall success of the LMS positively:
Finding 7. Learners’ perceived satisfaction toward e-learning is positively related to instructor’s rapid responses to learner’s needs.
Finding 8. Learners’ perceived satisfaction is positively related with instructors self-efficiency toward content and LMS.
Serwatka states that an important aspect of e-learning effectiveness is the emergence of a shared learning space where learners perceive that they are part of a learning community. This is an important indicator of an effective e-learning environment.
In a distance environment, learners often feel isolated since they do not have the classroom environment in which to interact with the instructor. To overcome this
Learning
management
systems
123
feeling, instructors can provide various forms of interactive components and methods of contacts for the learners. Most importantly, the instructor should exhibit interactive teaching styles, encouraging interaction between the learners and with the instructor (Volery and Lord, 2000). In METU-Online LMS, most of distance learner program learners are satisfied from communication tools (more than 70 percent) Moreover, learners thought that instructors and assistants encourage them to participate in forums, e-mail lists, etc. Learners only complain about one of the communication tools within METU-Online: the chat room. This tool is not available within the LMS due to technical difficulties:
Finding 9. Learners’ perceived satisfaction is positively related to instructors’
communication abilities.
Previous research indicates that instructors’ timely response significantly influences learners’ satisfaction positively (Arbaugh and Duray, 2002; Arbaugh, 2007;
Thurmond et al., 2002). Instructor response timeliness is defined as whether learners perceive that instructors responded promptly to their problems (Sun et al., 2008). According to this study’s survey results, distance learning program learners’
perceived effectiveness from instructors’ attitudes are very high. Most of them stated that instructors are considerate and that they follow up learner problems and try to find solutions:
Finding 10. Learners’ perceived satisfaction is positively related to the capability of instructors to follow up learner problems and solution to learners problems.
Many IS-related studies have pointed out that the user interface is an area where a high level of interaction takes place; a well-designed, user-friendly learner interface therefore becomes one of the critical factors in determining whether learners will enjoy using the METU-Online (Wang et al., 2007). According to more than half of the e-learners (62 percent) METU-Online graphical user interface is suitable for education.
Navigation is easy and system is error free.
Regarding the criteria of system dimension, the stability of the learner interfaces has a great emphasis on the learner. One of the key issues for the learners is to be able to easily access shared data. When it comes to system content, learners care most about whether they find it useful. As to personalization, the results reflect that the learners’ most important requirement is being able to control their learning progress (Wang et al., 2007). When we look at all the criteria with respect to the success of the system, e-learners’ habits become important. “Easy navigation,”
“easy to find the required information,” and “available help option” are important aspects of a system. According to this study’s survey results, ION learners’ perceived effectiveness from METU-Online LMS’s system quality is above average. 60 percent of the learners have found the required information easily, 55 percent of the learners stated that they were able to navigate easily, but with no help option available in the system.
In addition, METU-Online system accessibility has not been found to be efficient.
Although it is a web-based application, due to some technical errors, the system occasionally can become disconnected. This is an important reliability problem, which has a negative impact on the learners’ satisfaction.
TG 3,2
124
According to the descriptive statistics, overall learners have positive attitudes toward METU-Online system (Table III: system quality, mean: 3.57):
Finding 11. System quality will positively influence perceived e-learner satisfaction with e-Learning.
Finding 12. E-learning system’s reliability is positively related with learners’
perceived effectiveness.
Course flexibility and quality are both proven to be significant in this research.
Flexibility of an e-Learning course is a strong indication of learner satisfaction (Sun et al., 2008). According to the descriptive statistics, ION learners’ perceived effectiveness from METU-Online LMS’s content quality has a mean value of 3.51 (Table III) which shows learners are satisfied from METU-Online content:
Finding 13. Information quality will positively influence perceived e-learner satisfaction with e-Learning.
Learners in distance learning courses often face technical problems. It is therefore crucial that each online program has to have a supportive technical staff who has a good control of the technology and is able to perform basic troubleshooting tasks (e.g.
adding a learner at the last minute, modifying learners’ passwords, changing the course settings) (Volery and Lord, 2000). Organisation skills go hand in hand with control of technology. (Hayneset al., 1997) remarked that supportive staffs is essential for overall coordination and that, as the development of an online course is labour intensive, both faculty and technical resources must be identified and committed to the schedule at an early stage. (Volery and Lord, 2000).
According to descriptive statistics, ION learners’ perceived effectiveness of the LMS services quality is (Table III: service quality, mean: 3.70) high. Learners are highly satisfied with the assistants’ attitudes, and with the services provided by the administrative staff:
Finding 14. Service quality will positively influence perceived e-learner satisfaction with e-Learning.
6. Conclusions
This research has been motivated by the desire to gain a better understanding the assessment of e-learning applications. Primary contribution of this research study is to find out how to define, assess, and promote e-learning success. In this context, this study achieved significant progress towards developing a conceptual model HELAM for measuring learner satisfaction with e-learning systems. HELAM identifies success factors influencing e-learners’ satisfaction within six different categories (technical issues: system quality; technical issues: service quality; technical issues:
content quality; social issues: learner perspective; social issues: instructor attitudes;
supporting issues). In these six different categories there are 46 success factors to evaluate e-learning effectiveness. Each of them is quantified by a survey question. The overall success of e-learning can be evaluated as a cumulative sum of all these. The proposed assessment model in this study is independent from any human or country specific features. This makes it applicable to any LMS that is in use within a distance education program at any higher education institution in the world.
Learning
management
systems
125
The proposed model is a new model that validation has become essential.
Therefore, a number of validity and reliability tests have been applied (i.e. content validity, factor analysis, reliability). According to the results of these validity tests, the model has been iteratively modified. Since it is a developing model, it can evolve in parallel with technological improvements and other studies in e-learning. In this context, future studies may explore other dimensions or criteria for measuring the effectiveness of e-learning systems. In this paper, the effectiveness of an LMS (i.e. METU-Online) has been evaluated with both quantitative and qualitative analysis methods. Survey method has been used to obtain relevant data from learners about their perceptions of the impact of the blended learning environment in terms of their benefits and satisfaction level. One-to-one and focus group interviews have enriched the survey’s quantitative findings. According to the survey results, overall learners’ satisfaction is high. Since this paper is in progress, more research is needed to explore the applicability of the HELAM success model to other LMSs. For that, different surveys need to be applied to different LMS users, and this data is to be processed for assessing e-learning environments by using HELAM. Further application of the proposed model in various settings will be beneficial to the continued growth of this important research area. The findings presented in this paper will be valuable for future researchers who would like to do research in e-learning evaluation.
Note
1. All the steps explained through factor analysis contains repeated controls for omitting inappropriate items from the factors in according to the factor analysis rules explained throughout this part. The process is defined over the final version of the factors.
References
Arbaugh, J.B. and Duray, R. (2002), “Echnological and structural characteristics, student learning and satisfaction with web-based course: an exploratory study of two on-line MBA programs”,Management Learning, Vol. 33 No. 3, pp. 331-47.
Bruner, J.S. (1971),The Relevance of Education, Harvard University, Cambridge, MA.
Carmines, E.G. and Zeller, R.A. (1994), “Reliability and validity assessment”, in Lewis-Beck, M. (Ed.),Basic Measurement, Sage, London.
Collis, B. (1995), “Anticipating the impact of multimedia in education: lessons from the literature”,Computers in Adult Education & Training, Vol. 2 No. 2, pp. 136-49.
Davis, F.D., Bagozzi, R.P. and Warshaw, P.R. (1989), “User acceptance of computer technology:
a comparison of two theoretical models”,Management Science, Vol. 35, pp. 982-1003.
DeLone, W.H. and McLean, E.R. (1992), “Information systems success: the quest for the dependent variable”,Information Systems Research, Vol. 3 No. 1, pp. 60-86.
DeLone, W.H. and McLean, E.R. (2003), “The Delone and Mclean model of information systems success: a ten-year update”,Journal of Management Information System, Vol. 19 No. 4, pp. 9-30.
Gable, G.G., Sedera, D. and Chan, T. (2003), “Enterprise systems success: a measurement model”, Proceedings of the 24 International Conference on Information Systems, Association for Information Systems, Seattle, WA, pp. 576-91.
Global Industry Analysts, Inc. (2008), “e-learning: a global strategic business report”, available at: www.strategyr.com/pressMCP-4107.asp (accessed April 2008).
TG 3,2
126
Hayen, J., Cappel, J. and Roger, L. (2004), “Evaluating e-learning: a case study”,Journal of Computer Information Systems, Vol. 44 No. 4, pp. 49-56.
Haynes, R.M., Pouraghabagher, R. and Seu, A. (1997), “Interactive distance education alliance (IDEA): collaborative model delivers on demand”,The Journal: Technological Horizons in Education, Vol. 24 No. 8, pp. 60-3.
Heo, J. and Han, I. (2003), “Performance measure of information systems (IS) in evolving computing environments: an empirical investigation”, Information and Management, Vol. 40 No. 4, pp. 243-56.
Holsapple, C.W. and Lee-Post, A. (2006), “Defining, assessing, and promoting e-learning success:
an information systems perspective”,Decision Sciences Journal of Innovative Education, Vol. 4 No. 1, pp. 67-85.
Initiatives, MIT-OKI Open Knowledge (2003), “MIT-OKI open knowledge initiatives”, available at: web.mit.edu/oki/learn/gloss.html (accessed April 2008).
Islas, E., Pe´rez, M., Rodriguez, G. and Ivonne, I.P. (2007), “E-learning tools evaluation and roadmap development for an electrical utility”, Journal of Theoretical and Applied Electronic Commerce Research, Vol. 2 No. 1, pp. 63-75.
Johnson, R.D., Hornik, S. and Salas, E. (2008), “An empirical examination of factors contributing to the creation of successful e-learning environments”, International Journal of Human-Computer Studies, Vol. 66 No. 5, pp. 356-69.
Kaiser, H.F. (1974), “An index of factorial simplicity”,Psychometrika, Vol. 39, pp. 31-6.
Kaplan-Leiserson, E. (2008), “e-learning glossary”, available at: www.learningcircuits.org/
glossary.html (accessed April 2008).
Kohlberg, L. (1966), “A cognitive-developmental analysis of children’s sex-role concepts and attitudes”, in Maccoby, E. E. (Ed.), The Development of Sex Differences, Stanford University Press, Berkeley, CA, pp. 82-173.
Lee, J.-K. and Lee, W.-K. (2008), “The relationship of e-learner’s self-regulatory efficacy and perception of e-learning environmental quality”,Computers in Human Behavior, Vol. 24 No. 1, pp. 32-47.
Liaw, S.S. (2002), “An internet survey for perceptions of computer and world wide web:
relationship, prediction, and difference”,Computers in Human Behavior, Vol. 18 No. 1, pp. 17-35.
Liaw, S.-S., Huang, H.-M. and Chen, G.-D. (2007), “Surveying instructor and learner attitudes toward e-learning”,Computers & Education, Vol. 49 No. 4, pp. 1066-80.
Moore, G.C. and Benbasat, I. (1991), “Development of an instrument to measure the perceptions of adopting an information technology innovation”,Information Systems Research, Vol. 2, pp. 192-222.
Myers, B.L., Kappelman, L.A. and Prybutok, V.R. (1998), “A comprehensive model for assessing the quality and productivity of the information systems function: toward a theory for information systems assessment”, Information Systems Success Measurement, IGI Publishing, Hershey, PA, pp. 94-121.
Passerini, K. and Granger, M.J. (2000), “A development model for distance learning using the internet”,Computers & Education, Vol. 34 No. 1, pp. 1-15.
Rai, A., Lang, S.S. and Welker, R.B. (2002), “Assessing the validity of is success models: an empirical test and theoretical analysis”,Information systems Research, Vol. 13 No. 1, pp. 50-69.
Scott, A. (1959), “Attitude change by response reinforcement: replication and extension”,William Sociometry, Vol. 22 No. 4, pp. 328-35.
Learning
management
systems
127
Seddon, P.B. (1997), “A respecification and extension of the DeLone and McLean model of IS success”,Information Systems Research, Vol. 8 No. 3, pp. 240-53.
Selim, H.M. (2007), “Critical success factors for e-learning acceptance: confirmatory factor models”,International Journal of Technology Marketing, Vol. 2 No. 2, pp. 157-82.
Smith, B., Caputi, P. and Rawstorne, P. (2000), “Differentiating computer experience and attitudes toward computers: an empirical investigation”,Computers in Human Behavior, Vol. 16, pp. 59-81.
Spiro, R.J., Feltovich, P.J., Jacobson, M.J. and Coulson, D.K. (1995), “Cognitive flexibility, constructivism, and hypertext: random access instruction for advanced knowledge acquisition in ill-structured domains”,Educational Technology, Vol. 31 No. 5, pp. 24-33.
Sun, P.C., Tsai, R.S., Finger, G., Chen, Y.-Y. and Yeh, D. (2008), “What drives a successful e-learning? An empirical investigation of the critical factors influencing learner satisfaction”,Computers & Education, Vol. 50 No. 4, pp. 1183-202.
Tabachnick, B.G. and Fidell, L.S. (2001),Using Multivariate Statistics, 4th ed., Allyn & Bacon, Needham Heights, MA, p. 55.
Thurmond, V.A., Wambach, K. and Connors, H.R. (2002), “Evaluation of student satisfaction:
determining the impact of a web-based environment by controlling for student characteristics”,The American Journal of Distance Education, Vol. 16 No. 3, pp. 169-89.
Volery, T. and Lord, D. (2000), “Critical success factors in online education”,The International Journal of Educational Management, Vol. 14 No. 5, pp. 216-23.
Vygotsky, L. (1978),Mind in Society, Harvard University, Cambridge, MA.
Wang, Y-S. (2003), “Assessment of learner satisfaction with asynchronous electronic learning”, Information and Management Journal, Vol. 41 No. 1, pp. 75-86.
Wang, Y.-S. and Liao, Y.-W. (2008), “Assessing eGovernment systems success: a validation of the DeLone and McLean model of IS success”, Government Information Quarterly, Vol. 25, pp. 717-33.
Wang, Y-S., Wang, H-Y. and Shee, D.Y. (2007), “Measuring e-learning systems success in an organizational context: scale development and validation”,Computers in Human Behavior, Vol. 23 No. 1, pp. 1792-808.
Webster, J. and Hackley, P. (1997), “Teaching effectiveness in technology-mediated distance learning”,Academy of Management Journal, Vol. 6 No. 40, pp. 1282-309.
Further reading
Arbaugh, J.B. and Benbunan-Fich, R. (2007), “The importance of participant interaction in online environments”,Decision Support Systems, Vol. 43 No. 3, pp. 853-65.
TG 3,2
128
Appendix
Learner attitudes
Corresponding HELAM dimension: B. Social issues: learner perspective 1. Face-to-face education is better than distance education in learning process
2. I can manage my “study time” effectively and easily complete assignments on time by using METU-Online
3. I enjoy attending to the ION program overall
4. METU-Online LMS improves my success in ION program 5. I find all my educational need from METU-Online LMS
6. METU-Online makes the communication easier with instructor and other class mates for me 7. ION program’s content is English. This affects my academic success negatively
8. In my studies, I am self-disciplined and find it easy to set aside reading and homework time 9. My learning habits affect my decision to attend an online graduate program
10. I believe that METU-Online is a very efficient educational tool
11. I would recommend my friends to attend an online graduate education program
12. I would recommend my friends to attend specifically METU-Online graduate education program Instructor perspective
Corresponding HELAM dimension: C. Social issues: instructor attitudes 1. Instructor clearly informs the students about grading policy 2. The instructor returns e-mails/posts within 24 h
3. The instructor follows up student problems and tries to find out solution
4. Instructor frequently updates lecture notes and fixes all the errors and mistakes in the documents 5. The instructor responds promptly to questions and concerns
6. The instructor is proficient with all the content used in the course
7. The instructor created an online environment conducive and enjoyable for learning 8. The instructor and the assistant are good at communication with students 9. I think communicating with the instructor or the assistant is worthless 10. It is very hard for me to reach the instructor or the assistant via phone 11. Exam results are announced on time
12. Instructors and assistants are encouraging students to interact with other students by using METU-Online interactive tools
System quality
Corresponding HELAM dimension: D. System quality: technical quality 1. METU-Online’s graphical user interface is suitable for e-learning systems 2. The program directions and navigations are clear
3. METU-Online supports interactivity between learners and system by chat, forums, discussions and etc
4. There are several system errors on METU-Online
5. When I counter an error in the system, I can get immediate feedback by e-mail and telephone 6. Navigation is very easy
7. I can find required information easily
8. In the METU-Online system I can easily navigate where I want 9. METU Online is easily accessible via internet
10. METU-Online is a good educational portal and improve my learning 11. Help option is available on the system
12. METU-Online is accessible seven days 24 h
13. I am informed about all the course announcements METU-Online by using announcements 14. Fonts (style, color, and saturation) are easy to read in both on-screen and in printed versions Content quality
Corresponding HELAM dimension: E. System quality: information (content) quality 1. Lecture notes are the core learning materials in METU-ION program
2. Course content and presentation gain attention
(continued)
Table AI.
Questionnaire
Learning
management
systems
129
Corresponding author
Sevgi Ozkan can be contacted at: [email protected]
3. Course content and presentation are long enough to cover all content 4. The course content is covered to an appropriate degree of breadth 5. The content is up-to-date
6. It is hard to understand and follow the content in lecture notes
7. Lecture notes are not supported by multimedia tools (Flash animations, simulations, videos, audios, and, etc.)
8. There is not interactivity in lecture notes 9. Content is integral
10. Abstract concepts (principles, formulas, rules, etc.) are illustrated with concrete, specific examples 11. Lecture notes are very boring
12. Exam Questions are clearly explained 13. Exam durations are well organized
14. Given examples are up-to-date, real-life examples, they improve my learning 15. Vocabulary and terminology used are appropriate for the learners
16. The objectives of the course are stated clearly Service quality
Corresponding HELAM dimension: F. System quality: service quality 1. Each course has one assistant support during whole semester 2. Assistants’ attitudes are good to learners
3. Assistants’ attitudes are friendly to learners 4. Assistants are knowledgeable enough about content 5. The service supported by institute is good enough 6. I can contact with assistant via mail or phone or fax
7. I do not encounter any problems during communicating with institute administration, secretaries and assistants
8. I do not experience any problems during registrations
9. I can easily solve when I encounter a problem during admission to a course in registrations 10. The institution provides admission requirements information on-line
Supportive issues
Corresponding HELAM dimension: A. Supportive issues
1. METU-ION programs lecture notes are prepared by obeying the ethical and legal issues 2. The course provides any ethics policies that outline rules, regulations, guidelines, and prohibitions 3. I obey the ethical and legal issues in the course related testing, assignments and projects and am
knowledgeable about the consequences of any forms of plagiarism Table AI.
TG 3,2
130
To purchase reprints of this article please e-mail:[email protected] Or visit our web site for further details:www.emeraldinsight.com/reprints
