24
User Experience Analysis of E-Learning UIN Malang with User Experience Questionnaire (UEQ) Method:
Case Study of Students of Library and Information Science Departmen
Alimatul Listiyah
1and Fatimahtus Zahro N.W.
21,2 Department of Library and Information Science, Universitas Islam Negeri (UIN) Maulana Malik Ibrahim Malang, Indonesia E-mail: 1 [email protected]
Abstract-The rapid advancement of information technology has an impact on all aspects of human life. In the field of education, this can be seen through the number of universities that use E- Learning (Electronic Learning) as an alternative media of learning between lecturers and students, especially during this pandemic, where learning is done by online. One of the universities that utilizes E-Learning as media of learning is UIN Malang. Even though it is used as an alternative media of learning, there is still some troubles that experienced by students.
According to a survey we conducted on students from the Department of Library and Information Science of UIN Malang, it was found that the use of E-Learning that owned by the campus was still low. The data shows that 53% of the 60 respondents do not use the application frequently. This study use a quantitative typeof research with asurvey approach. The population in this study is the students of Library and Information Science Department. The sample is part of the entire population that can represent the entire population. To determine the size of the sample to be taken, we used the Slovin formula with a confidence value of 90% and a tolerance value of 10%. In addition, based on a survey conducted using the User Experience Questionnaire (UEQ) method, it was found that in terms of attractiveness, perspicuity, efficiency, dependability, stimulation and novelty, are in a bad position. From the explanation above, it can be concluded that, E-Learning UIN Malang is still not optimally used and still needs a lot of development.
Keywords—E-learning; User Experience; User Experience Questionnaire (UEQ)
I. INTRODUCTION
In digital era, information technology is developing more rapidly and contributes a lot on people's daily lives.
Technological advances that occur, especially in the internet (Interconnection Networking) greatly affect various fields such as business, education, entertainment, and even industry, which can enable humans to be more efficient in communicating, exchanging data, and finding information. In the field of education itself, the development of learning
media can be seen through the number of universities that use E-Learning (Electronic Learning) as a media of learning between lecturers and students [1]. E-Learningcan be used as an alternative solution for teaching at Indonesian universities, especially during a pandemic like this, where learning is done by online.
One of the universities that use E-Learning as an alternative media of learning is the State Islamic University (UIN) of Malang. Through this web-based application, lecturers can share lecture material as well as a media for collecting assignments. In addition, this application can save data quota when compared to other learning media such as video conferencing.
Even though it is used as an alternative learning media, there are still some troubles that experienced by students. According to a survey that we conducted on students of the Department of Library and Information Science of UIN
Breakdown Voltage (kV)
100 101 102
0.2 0.1 2 20 70 90 98 99.9
50
Weibull Breakdown Probability (%)
30
10 5
1 0.5
25
Malang, it was found that the use of E-Learning at UIN Malang was low. The data shows that 53% of the 62 respondents, do not use the application often. The confusing layout and the absence of notifications are the reason they rarely use it. Seeing the phenomena and conditions based on the survey results, this study tries to find out and analyze the lack of student interest in the use of E-Learning at UIN Malang by using the rules contained in the theory user experience.
The main focus of user experience (UX) is the experience of individual subjects regarding the interaction between them and the system. This approach focuses on the individual experience itself as a subjective phenomenon that is constantly being interpreted and reinterpreted by them [2]. In several UX measurement methods, some of which are Software Usability Measurement Inventory (SUMI), System Usability Scale (SUS), User Experience Questionnare, and Questionnaire for User Interaction Satisfaction (QUIS) [3].
In this research, the chosen method is UEQ, which can enable rapid assessment of the user experience of interactive products. In addition, the questionnaire scale is designed for a comprehensive user experience impression. Not only that, the questionnaire format is also structured so that it can support responses user to immediately express feelings, impressions, and attitudes that arise when using a product [4].
Compared to others, UEQ is one of the UX measurement methodswhich is believed to provide more benefits. This is because UEQ can provide comprehensive measurement results on user experience.
Based on the survey conducted, it shows that the level of use of E-Learning at UIN Malang by students of the Department of Library and Information Science of UIN Malang is still low. Therefore, the analysis of E-Learning UIN Malang by using the rules in UX The UEQ scale will be carried out to find out the reason why the level of use is still relatively low. Based on the background above, the formulation of the problem that can be taken in this study is:
Why is E-Learning UIN Malang not very attractive to students by analyzing the user experience of the application. Based on the formulation of the problem above, the purpose of this study is to find out the root cause of why E-Learning UIN Malang is not very attractive to students by testing the user experience application's.
II. METHODS A. Design Research
Design is a step carried out by researchers to provide illustrations and ease in carrying out research. The research design is as follows:
Fig. 1. Research Design
B. Types and Research Approach
In this study, the researcher used a quantitative research type with a survey approach. Quantitative research is research that aims to reveal symptoms as a whole in a context through collecting data from the field [5].
Meanwhile, the survey approach method is research by collecting data from a sample by giving various questions through questionnaires, so that it can provide an overview of various aspects of the population. The survey also has 5 advantages as follows; (1) Only requires low cost, (2) relatively short time, (3) more detailed data, (4) more accurate, (5) data can be focused on a problem.
C. Research
Instruments Data collection tools in this study are in the form of a questionnaire or questionnaire. Questionnaire or questionnaire is a list of questions accompanied by a checklist with a scale [5]. In this study using a web-based questionnaire or using google forms as a tool. There are 26 questions or statements in this questionnaire developed based on EUQ theory instudies user experience that are relevant to research variables, including; (1) Efficiency (Efficiency), (2) attractiveness (attractiveness), (3) Perspicuity (Clarity /Easy), (4) Stimulation (stimulation), (5) Dependability (Reliability), and (6) Novelty (Novelty /Innovative).
26
D. Population and Sample
Population is the entire element or research subject to be studied based on the problems and objectives in the research [6].The population in this study were students majoring in Library and Information Science UIN Malang with a total of 163 students obtained from the documentation of incoming and outgoing students from 2018 to 2021 by the Department of Library and Information Science of UIN Malang. The sample is part of the entire population that can represent the entire population. To determine the size of the sample to be taken, the researcher used the Slovin formula in Firdaus [7]
with a confidence value of 90% and a tolerance value of 10%.
The sample calculation with the Slovin formula is as follows:
Fig. 2 Slovin Formula
Where:
n : Number of Samples N : Number of Populations e : Error Limit (tolerance) of 10%
so:
= 61,977 or equal to 62, it can be concluded, the sample in this study used 62 respondents.
In taking the sample there is a sampling technique which is divided into 2, namely probability sampling and non-probability sampling. In this study using probability sampling. According to Sumargo [8] probability sampling is that each unit analyzed in the population has the same opportunity to become a member of the sample. probability sampling consists of simple random sampling, cluster sampling, disproportionate stratified random, proponate stratified random sampling. In this study, simple random sampling is used, namely the taking of sample members from the population which is carried out randomly without regard to the strata that exist in a population [9].
E. Data Analysis
Analysis is a series of processes comparing existing concepts or theories with the information found [10]. Data analysis is carried out by obtaining survey results from correspondents who will be studied with theory or the UEQ scale from studies user experience, that contains 26 questionnaires.
III. RESULTS AND DISCUSSION
Evaluation of user experience using the UEQ method at E-Learning UIN Malang is done by giving 26 statements
that offer the opinion of active students majoring in library and information science at UIN Malang regarding the use of E-Learning at UIN Malang. The results of the questionnaire distribution, there were 62 respondents who filled out the questionnaire from various generations. The data obtained from the questionnaire is processed into UEQ data analysis tools. The value of -2 is the lowest negative value and +2 is the highest positive value. The data is then processed to obtain the average value for each aspect of the UEQ. Figure 3 below shows the results of the questionnaire data processing.
Fig. 3. Results of Questionnaire Data Processing
TABLE I
QUESTIONNAIRE DATA PROCESSING RESULTS
Aspect UX
Aspe ct UX Detai ls
UEQ Scale Value
Average Value
Attracti veness
Attra ctive ness
-0.172 -0.172 Pragma
tic Quality
Pers picui ty
-0.040
-0.176 Effic
ienc y
-0.371 Depe
ndab ility
- 0.117 Hedoni
c Quality
Stim ulati on
-0.230
-0.264 Nov
elty -0.298
Based on table I the results of questionnaire data processing on the aspect Pragmatic Quality show the lowest value is in the Efficiency aspect which has a value of - 0.371. Shows that users find it difficult to operate E-Learning UIN Malang when completing tasks. The accuracy aspect has a negative value of -0.117 indicating that the E-Learning UIN Malang is less controllable by the user. E-Learning UIN Malang is less able to be learned by users because the Clarity aspect is worth - 0.040. In this pragmatic aspect, it has an average value of - 0.176. Pragmatic Quality is one of the perceptions used to
27
create or build a UEQ framework. Where the aspects include aspects of efficiency, perspicuity and dependability.
The average value on the aspect is Hedonic -0.264 with details of the Stimulation aspect of -0.230. Shows that E-Learning UIN Malang is less useful and less motivating to users, while the novelty aspect is -0.298 which indicates E-Learning UIN Malang is less innovative and monotonous.
a. Benchmark
Benchmark is a process used to measure or compare product performance against activities. So that later a company or organization can achieve the desired level of performance [11]. For more details, let's look at the following explanation.
Fig. 4. Comparison Results withScale The Benchmark
Image above (figure 4) shows a scale from -1.00 to 2.00 with value descriptions such as bad, bellows average, above average, good, and also excellent as well as color descriptions at each level .Values are bad indicated by red captions, values are average bellows indicated by orange captions, values above average is indicated by a faded green color, value is good indicated by light green description, value is excellent indicated by dark green color.
TABLE II
COMPARISON RESULTS WITH BENCHMARK SCALE
Scale Mean Comparisson
to benchmark Attractive
ness
-0.17 Bad
Clarity -0.04 Bad
Efficiency -0.37 Bad
Accuracy -0.12 Bad
Stimulatio n
-0.23 Bad
Novelty -0.30 Bad
In addition, Figure 4 and Table II show that the aspects of attractiveness, perspicuity, efficiency, dependability, stimulation and novelty being in a bad position. This is indicated by the value of -0.17 for attractiveness, -0.04 for perspicuity, -0.37 for efficiency, -0.12 for dependability, -0.23 for stimulation and -0.30 for novelty.
To know the meaning of the values of the above, the comparison between the results of the calculation scale with values in the following table,
TABLE III
BENCHMARK INTERVAL FOR UEQ SCALE [12]
Attr activ enes s
Per spi cuit
y Effi
cie ncy
Dep end abil ity
Sti mul atio n
No velt y Excellent ≥
1.75
1.9 1.7 8
1.65 1.5 5
1.4 Good 1.52 1.5
6 1.4
7
1.48 1.3 1
1.0 5
<
1.75
<
1.9
<
1.7 8
<
1.65
<
1.5 5
<
1.4 Above 1.17 1.0
8 0.9
8
1.14 0.9 9
0.7 1 Average <
1,52
<
1.5 6
<
1.4 7
<
1.48
<
1.3 1
<
1.0 5 Below 0.7 0.6
4 0.5
4
0.78 0.5 0.3 Average <
1,17
<1.
08
<0.
98
<1.
14
<0.
99
<0.
71 Bad <0.7 <0.
64
<0.
54
<0.
78
<0.
5
<0.
3 Table III shows that all aspects have value limits ranging from bad, below average, above average, good, and excellent. The attractiveness aspect has a score limit bad of less than 0.7; below average 0.7 - 1.17; above average 1.17 – 1.52; good 1.52 – 1.75; and excellent is more than equal to 1.75. The clarity aspect has a score limit bad of less than 0.64; below average 0.64 – 1.08; above average 1.08 – 1.56;
good 1.56 – 1.9; and excellent is more than equal to 1.9. The efficiency aspect has a score limit bad of less than 0.54;
below average 0.54 – 0.98; above average 0.98 – 1.47; good 1.47 – 1.78; and excellent is more than equal to 1.78. The accuracy aspect has a score limit bad of less than 0.78;
below average 0.78 – 1.14; above average 1.14 – 1.48; good 1.48 – 1.65; and excellent is more than equal to 1.65.
Stimulation aspect has a score limit bad of less than 0.5;
below average 0.5 – 0.99; above average 0.99 – 1.31; good 1.31 – 1.55; and excellent is more than equal to 1.55. The novelty aspect has a score limit bad of less than 0.3; below average 0.3 – 0.71; above average 0.71 – 1.05; good 1.05 – 1.4; and excellent is more than equal to 1.4.
b. Confidence Interval
Confidence Interval is a measure for the accuracy of the average estimate of the scale. The smaller the Confidence Interval, the higher the estimation precision and the more reliable the questionnaire results. The more consistent the opinion of the respondents, the smaller the Confidence Interval [13]. In this report, Confidence Intervals 5% for aspect averages and single item averages are shown.
28
TABLE IV
THE CONFIDENCE INTERVAL
Table above (table IV) shows that the aspect that has the Confidence Interval smallest is the attractiveness aspect with the number 0.3. Then followed by the aspect of efficiency with the number 0.36; the aspect of clarity with the number 0.369; the aspect of novelty with the number 0.509;
the aspect of stimulation with the number of 0.49; and the biggest is the aspect of accuracy with the number 0.624.
This can be interpreted that the most consistent respondents' answers are answers related to the attractiveness aspect. Meanwhile, the respondents' answers that were the least consistent were those related to the aspect of accuracy.
IV. CONCLUSIONS
From the explanation above, it can be concluded that, E-Learning UIN Malang still has many shortcomings. It was evident from the comparison results using a scale benchmark, where the aspects that exist as aspects of the attractiveness, perspicuity, efficiency, dependability, stimulation and novelty are at bad position. With a score of - 0.17 for attractiveness, -0.04 for perspicuity, -0.37 for efficiency, -0.12 for dependability, -0,23 for stimulation and - 0.30 for novelty.
ACKNOWLEDGMENT
The research funded by DIPA Faculty of Science and Technology Universitas Islam Negeri Maulana Malik Ibrahim Malang.
REFERENCES
[1] Hendri. (2014). Pemanfaatan E-Learning denganAplikasi Web 2.0 sebagaiSaranaPembelajaran pada Perguruan Tinggi di Indonesia.
JurnalIlmiah Media Processor, 9(2), 7.
[2] Williamson, J. R., & Brewster, S. (2012). A performative perspective on UX.Communications in Mobile Computing, 1(1), 3.
https://doi.org/10.1186/2192-1121-1-3.
[3] Wulandari, I. R., & Farida, L. D. (2018). Pengukuran User Experience Pada E- Learning di LingkunganUniversitasMenggunakan User Experience Questionnare (UEQ). JurnalMantikPenusa, 2(2), 6.
[4] Sularsa, A., Prihatmanto, A. S., &Nugroho, E. (2015). Evaluasi User Experiences ProdukiDigital Museum denganMenggunakan UEQ.
JurnalTeknologiInformasi, 2(2), 56–62.
[5] Hardani, H., ustiawaty, jumari, Andriani, H., istiqomah, ria, Sukmana, D., Fardani, R., auliya, nur, &Utami, E. (2020).
BukuMetodePenelitianKualitatif&Kuantitatif.
[6] Norfai. (2021). “KesulitandalamMenulisKaryaTulisIlmiah”, KenapaBingung ?Penerbit Lakeisha.
[7] Firdaus. (2021). METODOLOGI PENELITIAN KUANTITATIF;
DILENGKAPI ANALISIS REGRESI IBM SPSS STATISTICS VERSION 26.0. CV. DOTPLUS Publisher.
[8] Sumargo, B. (2020). Teknik sampling. Unj press.
[9] Tegor, Alpino Susanto, Veterson Togatorop, Lod Sulivyo, & Dwi Joko Siswanto. (2020). Metodologi Penelitian Kualitatif & Kuantitatif.
Penerbit Lakeisha.
[10] Barlian, E. (2018). MetodologiPenelitianKualitatif&Kuantitatif. INA- Rxiv. https://doi.org/10.31227/osf.io/aucjd
[11] Putri, N. A. R., & Adji, T. B. (2021). Data Benchmark pada Google BigQuery dan Elasticsearch. Jurnal Nasional Teknik Elektro Dan Teknologi Informasi, 10(3), 196–203.
[12] Henim, S. R., & Sari, R. P. (2020). Evaluasi User Experience Sistem InformasiAkademik Mahasiswa pada Perguruan Tinggi Menggunakan User Experience Questionnaire. Jurnal Komputer Terapan, 6(1),69–78.
[13] HB Santoso, M. Schrepp, R. Yugo Kartono Isal, Y. Utomo, dan B.
Priyogi , “Mengukur Pengalaman Pengguna Lingkungan e-Learning Berpusat pada Siswa,” J. Educ. Online-JEO, vol. 13, tidak. 1, hlm. 142–
16 Confidence intervals (p=0.05) per scale
Scale Mean Std. Dev. N Confidence Confidence interval
attractiveness -0.322 -0.022 0.604 62 0.150 -0.172
Clarity -0.225 0.144 0.740 62 0.184 -0.040
Efficiency -0.551 -0.191 0.723 62 0.180 -0.371
Accuracy 1.253-0.429 62 0.312 -0.117 0.195
Stimulation -0.230 1.185 62 0.295 - 0.525 0.065
Novelty -0.298 1.022 62 0.254 -0.553 -0.044
