Psychometric Properties and Construct Validity of Online Learning Readiness Scale (OLRS)
Indonesian Version
1st Dede Rahmat Hidayat Guidance and Counseling Department
Universitas Negeri Jakarta Jakarta, Indonesia [email protected]
2nd Nacep Hamrat
Curriculum Development Department Universitas Pendidikan Indonesia
Bandung, Indonesia [email protected]
Abstract—This study aims to adapt the Online Learning Readiness Scale (OLRS) into Bahasa Indonesia and to test its psychometric properties for Indonesian use. OLRS was formed by five dimensions: computer/internet self-efficacy, self-directed learning, motivation for learning, learner control, and online communication self-efficacy. The adaptation process was carried out in two phases, the first phase of translation and the second phase of empirical testing. The procedure of adaptation is performed following adequate adaptation guidelines. After being translated, the scale was administered to 749 respondents.
The result of CFA indicates a good model fit.
Keywords—distance education, online learning readiness, OLRS Indonesian version, psychometric properties, construct validity
I. INTRODUCTION
When the Covid-19 pandemic hit the entire world, many changes were put in place to prevent a massive spread [1]. One form of change must be made in learning activities, from face- to-face learning to online learning [2]. Even though online- based technology is very familiar in the community, its use for learning is minimal and causes several problems [3][4].
Several online media are used in education by students and teachers in Indonesia, e.g., Whatsapp, Microsoft learning, or Google Classroom, Zoom, Skype, etc. [5]. These various applications have become familiar to use, even though previously they were not very popular.
Changing habits in learning from face to face to online learning will provide meaningful changes. During offline learning, the teacher can directly observe student learning behaviour in the classroom. There is substantial control from the teacher over his students to keep them always focused on the material being discussed [4][6]. Meanwhile, when learning online, there is no direct physical interaction between teachers and students. Online learning requires firm control from students. Students are required to have a personal commitment and good self-regulation.
Hung, Chou, Chen, and Own have developed an instrument that can assess students' readiness to participate in online learning [7]. This instrument was called the Online learning readiness scale (OLRS). The online learning readiness scale (OLRS) consists of 18 items and has five dimensions. The dimension of Computer/Internet self- efficacy (CIS) consists of 3 items, Self-directed learning (SDL) consists of 5 items, Learner control (in an online context) (LC) consists of 3 items, Motivation for learning (in
an online context) (MFL) consists of 4 items and dimensions of Online communication self-efficacy (OCS) consists of 3 items.
The psychometric instrument has met reasonable scale requirements and has been tested in many countries, such as Australia [8] and the United States [9]. Still, in Indonesia, the Online learning readiness scale (OLRS) has never been adapted. Currently, along with the development of online learning due to the Covid-19 pandemic, this instrument is feasible to use. Still, of course, it must pass testing so that the psychometric properties are attainable.
II. METHOD A. First Stage
The first stage of the study was the translation of the scales into Bahasa Indonesia. This stage involved five subjects i.e., 2 Indonesian translators with backgrounds in psychology and education–living in Australia and Germany for more than five years, one professional translator, and two readers (teachers and students).
The two translators with backgrounds in psychology and education separately translated OLRS into Bahasa Indonesia, and the professional translators translated back the scale from Bahasa Indonesia into English. Two laypeople (teachers and students) read the translation results in Bahasa Indonesia and then were interviewed about the level of understanding of the items on the scale.
The first phase was carried out by referring to the back- translation procedure [10], [11]. The back-translation procedure was as follows. (1) Two translators translate the instrument from the original language into Bahasa Indonesia separately. (2) The two results of the translation were seen as their equivalence then discussed between the two translators to produce the Indonesian version of the OLRS. (3) The Indonesian version of the OLRS draft was compiled back into English by a professional translator. (4) The results of the re- translation into English were discussed for their suitability with the original OLRS exists with the original version, a compromise was sought for the translated item in Indonesian, so that it does not deviate too much from the meaning in the original OLRS version. (5) Furthermore, the Indonesian version of the OLRS item modified from the two to better suit the Indonesian language style and context. (6) The Indonesian version of OLRS was presented to teachers and students with the aim of seeing the level of lay understanding of the
translated items. (7) Layout of complete OLRS items was ready for use. The last stage was to convert OLRS into digital form because OLRS will be tested through electronic media, namely by. (8) Converting OLRS into google form format to be distributed to respondents.
B. Second Stage
The second stage of the study aims to test the measurement model's suitability for each dimension of the Indonesian version of OLRS. The dimension of Computer/Internet self- efficacy (CIS) consists of three items, Self-directed learning (SDL) consists of five items, Learner control (in an online context) (LC) consists of three items, Motivation for learning (in an online context) (MFL) consists of four items and dimensions of Online communication self-efficacy (OCS) consists of three items. This primary model is after this referred to as M1. The next step is: (1) CFA analysis to see the conformity of M1 with the data, and (2) if M1 does not meet the fit criteria, the researcher compiles a new model.
Besides the CFA model test, a reliability analysis was also carried out using Cronbach-Alpha on the Indonesian version of OLRS that met the fit criteria. Confirmatory Factor Analysis (CFA) involved 749 people, at first there were 750 respondents, but there was 1 who did not fill in completely, so it could not be analyzed, 62% (468) women and 37% (281)
men studied at the junior high school level, High School and College. Age from 13 to 23 years. They were recruited by filling out a questionnaire in the form of a google form submitted to their teachers and lecturers and delivered by a snowball. The Indonesian version of OLRS consists of 18 items; each item provides five possible answer options (1 = strongly disagree, 5 = strongly agree).
III. RESULT AND DISCUSSION
The results of the Online Learning Readiness Scale (OLRS) scale test were presented sequentially. It started with the first phase of research, namely the adaptation process into Indonesian, followed by the second stage of research through validity testing, reliability testing, model feasibility testing, and confirmatory factor analysis (CFA).
A. Back Translation
The back-translation results of the Online Learning Readiness Scale (OLRS) of Hung, Chou, Chen, and Own are presented in Table 1 [7]. The results of translating items for each dimension were then tested for readability by the teacher and students. The teacher and students were then interviewed regarding their understanding of each item before being distributed to respondents.
TABLE I. RESULT OF THE ONLINE LEARNING READINESS SCALE (OLRS)BACK-TRANSLATION
Indicator Original Statement Translation
CIS1 I feel confident in performing the basic functions of Microsoft Office programs (MS Word, MS Excel, and MS PowerPoint)
Saya merasa yakin dan bisa menggunakan program Microsoft Office (MS Word, MS Excel, dan MS Power Point).
CIS2 I feel confident in my knowledge and skills of how to manage software for online learning.
Saya memiliki pengetahuan dan keterampilan dalam menggunakan program Microsoft seperti Excel, Word, Powerpoint.
CIS3 I feel confident in using the Internet (Google, Yahoo) to find or gather information for online learning.
Saya merasa percaya diri menggunakan Internet (Youtube, Google dan aplikasi media sosial) untuk menemukan atau mengumpulkan informasi dalam pembelajaran online.
SDL1 Carry out my own study plan. Saya membuat rencana pembelajaran saya sendiri.
SDL2 I seek assistance when facing learning problems. Saya mencari bantuan ketika menghadapi masalah belajar (seperti tidak memahami materi atau bertanya mengenai tugas yang diberikan).
SDL3 I manage time well. Saya dapat mengatur waktu dengan baik.
SDL4 I set up my learning goals. Saya membuat tujuan dan target saya dalam belajar.
SDL5 I have higher expectations for my learning performance. Saya memiliki harapan yang tinggi dalam kegiatan pembelajaran.
LC1 I can direct my own learning progress. Saya bisa mengarahkan kemajuan belajar saya sendiri.
LC2 I am not distracted by other online activities when learning online (instant messages, internet surfing).
Saya tidak terganggu oleh kegiatan lainnya ketika belajar online (seperti membalas chat atau membuka internet/media sosial).
LC3 I repeated the online instructional materials on the basis of my needs.
Saya mengulangi materi belajar yang saya dapat secara online ketika saya membutuhkannya
MFL1 I am open to new ideas. Saya terbuka untuk ide-ide baru.
MFL2 I have motivation to learn. Saya memiliki motivasi untuk belajar.
MFL3 I improve from my mistakes. Saya dapat belajar dari kesalahan untuk meningkatkan diri menjadi lebih baik.
MFL4 I like to share my ideas with others. Saya suka membagikan ide dan pemikiran saya kepada orang lain.
OCS1 I feel confident in using online tools (email, discussion) to effectively communicate with others.
Saya merasa percaya diri dengan menggunakan media online (email, platform chat) untuk berkomunikasi dan diskusi secara efektif dengan orang lain.
OCS2 I feel confident in expressing myself (emotions and humor) through text.
Saya merasa percaya diri dengan mengekspresikan diri (emosi dan humor) melalui teks atau tulisan.
OCS3 I feel confident in posting questions in online discussions. Saya merasa percaya diri dalam mengajukan pertanyaan selama diskusi online.
B. Validity and Reliability Test item. The instrument validity test on factor analysis was
Component matrix score, component matrix, and Communalities-extraction scores.
The results of the instrument validity test are listed in Table 2. Based on the validity test results, it was shown that all dimensions of the online learning readiness instrument have a KMO value of more than 0.5, a MSA value of more than 0.5, and a significance of less than 0.05.
TABLE II. RESULT OF THE ONLINE LEARNING READINESS SCALE
(OLRS)VALIDITY
Item KMO Sig.
Bartlett’s Test of Sphericity
MSA Loading Factor
Communalities Score
CIS1
0.619 0.0
0.583 0.890 0.793
CIS2 0.583 0.892 0.796
CIS3 0.852 0.681 0.464
SDL1
0.764 0.0
0.818 0.693 0.481
SDL2 0.829 0.547 0.299
SDL3 0.761 0.712 0.507
SDL4 0.705 0.841 0.707
SDL5 0.787 0.708 0.501
LC1
0.631 0.0
0.624 0.755 0.571
LC2 0.671 0.702 0.493
LC3 0.609 0.780 0.608
MFL1
0.690 0.0
0.729 0.689 0.475
MFL2 0.667 0.770 0.593
MFL3 0.664 0.778 0.605
MFL4 0.722 0.697 0.486
OCS1
0.616 0.0
0.595 0.774 0.599
OCS2 0.602 0.759 0.577
OCS3 0.675 0.668 0.446
Thus, it can be concluded that the online learning readiness instrument was valid. The next test was the instrument reliability test. Instrument reliability test using Cronbach's Alpha. The results of the reliability test are presented in Table 3.
TABLE III. RESULT OF THE ONLINE LEARNING READINESS SCALE
(OLRS)RELIABILITY TEST
Latent Variable Cronbach’s Alpha’s
Score Description Computer/internet self-
efficacy 0.765 Reliable
Self-direct learning 0.745 Reliable Learner control 0.590 Reliable Motivation for learning 0.710 Reliable Online communication
self-efficacy 0.573 Reliable
C. Goodness of Fit Test
Confirmatory factor analysis was used to evaluate the Online Learning Readiness Scale (OLRS). The CFA performed serves to test the goodness of fit of the equation
model. The modification's model was applied to get a fit structural equation model. The modification's model was carried out by correlating the variance error between the manifest variables. In this study, the correlation was formed on items that were on the same factor. The modified structural equation measurement cross model diagram can be seen in Fig. 1.
Fig. 1 shows the path diagram of the confirmatory factor analysis measurement model of the online learning readiness scale. The measurement model in Fig. 1 was a modified measurement model. The modification was applied to get a fit structural equation model. The results of the goodness of fit from the measurement equation model can be seen in Table 4.
TABLE IV. THE RESULT OF GOODNESS OF FIT TEST
Model Cut-off value Score Description
2/df 2/df 5 4.64 Good Fit [14]
GFI 0.90 0.93 Acceptable Fit [15]
AGFI 0.85 GFT < 0.90 0.89 Acceptable Fit [15]
CFI 0.97 0.97 Good Fit [15]
TLI/NNFI 0.90 0.96 Good Fit [16]
NFI 0.95 0.96 Good Fit [17]
IFI 0.90 0.97 Good Fit [16]
RMSEA 0.08 0.059 Good Fit [17]
SRMR 0.10 0.069 Acceptable Fit [17]
D. Confirmatory Factor Analysis (CFA)
The measurement model testing was carried out using confirmatory factor analysis. The CFA test results show convergent validity and discriminant validity. Convergent validity serves to ensure that the manifest variable can measure the latent variable to be studied. The measure of convergent validity that will be used was the Standardized Loading Factor (SLF), while the reliability test will use Construct Reliability (CR) and Average Variance Extracted (AVE).
The validity test of the measurement model was carried out to see the indicators' accuracy on the latent variables being measured. Test the validity of the measurement model using the convergent validity test by looking at each indicator's standardized loading factor (SLF) value.
TABLE V. THE RESULT OF THE MEASUREMENT MODEL OF VALIDITY
TEST
Indicator SLF t-value
CIS1 0.85 21.81
CIS2 0.86 23.21
CIS3 0.49 10.25
SDL1 0.58 14.32
SDL2 0.45 9.65
SDL3 0.65 16.27
SDL4 0.76 23.34
SDL5 0.68 16.80
LC1 0.72 17.33
LC2 0.45 11.03
LC3 0.55 13.48
MFL1 0.48 11.59
MFL2 0.79 23.90
MFL3 0.67 16.17
MFL4 0.47 10.40
OCS1 0.80 14.37
OCS2 0.48 9.68
OCS3 0.79 9.21
Fig. 1. Path diagram of online learning readiness scale
Hair Jr., Tatham, and William (1995) require an SLF value of 0.50 [18], but Tabachnick and Fidell state that the SLF value of 0.45 was considered fair [19]. In this study, SLF will use the lowest limit value of 0.45 with a significance test using the t value. SLF with a t value above 1.97 was considered significant. The SLF of each indicator item can be seen in the path diagram in Fig. 1. Based on the CFA test results, the validation test results' recapitulation was obtained in Table 5.
Table 5 shows all the online learning readiness scale indicators' loading factors with a value between 0.45 and 0.86, with a t-value of more than 1.97. Thus, it can be concluded that the item indicator scale of online learning readiness has good validity. In the next step, we tested the construct reliability of the indicators that formed latent Online Learning Readiness scale. The result of the measurement of reliability test are presented in Table 6.
CR values greater than or equal to 0.7 and AVE greater than or equal to 0.5 indicate good reliability [12]. Based on the reliability test on the measurement model, the CR value was greater than 0.70, and AVE was greater than 0.50 in the dimensions of computer/internet self-efficacy (0.79) and online communication self-efficacy (0.74). This shows that the reliability of the Indonesian version of the OLRS instrument for these indicators was good.
learning (CR = 0.77; AVE =0.40), learner control (CR = 0.60;
AVE = 0.34), and motivation for learning (CR = 0.70; AVE = 0.38). Fornell and Larcker, in their research, said that CR with a value of more than or equal to 0.6 has an adequate construct, even though it has an AVE value of less than 0.5 [20]. Thus, it can be concluded that the model has fairly good reliability.
TABLE VI. THE RESULT OF THE MEASUREMENT MODEL OF
RELIABILITY TEST
Indicator SLF Error CR AVE
CIS1 0.85 0.27
0.79 0.57
CIS2 0.86 0.26
CIS3 0.49 0.76
SDL1 0.58 0.66
0.77 0.40
SDL2 0.45 0.80
SDL3 0.65 0.58
SDL4 0.76 0.42
SDL5 0.68 0.53
LC1 0.72 0.49
0.60 0.34
LC2 0.45 0.80
LC3 0.55 0.69
MFL1 0.48 0.77
0.70 0.38
MFL2 0.79 0.37
MFL3 0.67 0.56
MFL4 0.47 0.78
OCS1 0.80 0.36
0.74 0.50
OCS2 0.48 0.77
OCS3 0.79 0.38
This study aimed to test the psychometric properties of the adaptation Online Learning Readiness Scale (OLRS) in Bahasa Indonesia. Statistical analysis provided evidence that the Indonesian version of OLRS have good consistency. It suggests that the Indonesian version of OLRS can be used by researchers, psychologists, and teacher to assess student’s online learning readiness in Indonesia.
IV. CONCLUSION
The measurement of Online Learning Readiness Scale (OLRS) using Confirmatory Factor Analysis showed the scale was valid and reliable. Goodness of fit test results from Online Learning Readiness Scale (OLRS) showed fit. Model fit indices are as follow: 2 /df = 4.64; GFI = 0.93; AGFI = 0.89;
CFI = 0.97; TLI/NNFI = 0.96; NFI = 0.96; IFI = 0.97;
RMSEA = 0.059; and SRMR = 0.069. The loading factors of each indicator were fair. The Indonesian version of Online Learning Readiness Scale (OLRS) can be used for assess student’s online learning readiness in Indonesia.
REFERENCES
[1] World Health Organization. (2019). “Coronavirus disease (COVID- 19) advice for the public.” In https://www.who.int/emergencies/diseases/novel-coronavirus-
2019/advice-for-public
[2] Dhawan, S. (2020). “Online Learning: A Panacea in the Time of COVID-19 Crisis.” in Journal of Educational Technology Systems, 49(1), 5–22. https://doi.org/10.1177/0047239520934018
[3] Mulenga, E. M., & Marbán, J. M. (2020). “Prospective teachers’ online learning mathematics activities in the age of COVID-19: A cluster analysis approach.” in Eurasia Journal of Mathematics, Science and Technology Education, 16(9). https://doi.org/10.29333/EJMSTE/8345 [4] Rasmitadila, R., Aliyyah, R. R., Rachmadtullah, R., Samsudin, A., Syaodih, E., Nurtanto, M., & Tambunan, A. R. S. (2020). The Perceptions of Primary School Teachers of Online Learning during the COVID-19 Pandemic Period: A Case Study in Indonesia.” In Journal of Ethnic and Cultural Studies, 7(2), 90.
https://doi.org/10.29333/ejecs/388
[5] Bahasoan, A., Ayuandiani, W., & Mukhram, M. (2020). “Effectiveness of Online Learning In Pandemic Covid-19”.in International Journal Of Science, Technology & Management. 100–106.
[6] Lemay, D. J., & Doleck, T. (2020). “Online Learning Communities in the COVID-19 Pandemic: Social Learning Network Analysis of
Twitter during the Shutdown.” In International Journal of Learning Analytics and Artificial Intelligence for Education (IJAI), 2(1), 85.
https://doi.org/10.3991/ijai.v2i1.15427
[7] Hung, M. L., Chou, C., Chen, C. H., & Own, Z. Y. (2010). “Learner readiness for online learning: Scale development and student perceptions”. In Computers and Education, 55(3), 1080–1090.
https://doi.org/10.1016/j.compedu.2010.05.004
[8] Warner, D., Christie, G., & Choy, S. (1998). “The readiness of VET clients for flexible delivery including online learning.” Brisbane:
Australian National Training Authority, 1–64.
[9] McVay, M. (2001). “How to be a successful distance learning student:
Learning on the Internet.” New York: Prentice Hall.
[10] Brislin, R. W. (1980). “Translation and content analysis of oral and written material.” Handbook of Cross-Cultural Research, 1, 389–444.
[11] Wild, D., Grove, A., Martin, M., Eremenco, S., McElroy, S., Verjee- Lorenz, A., & Erikson, P. (2005). “Principles of good practice for the translation and cultural adaptation process for patient-reported outcomes (PRO) measures: Report of the ISPOR Task Force for Translation and Cultural Adaptation.” In Value in Health, 8(2), 94–
104. https://doi.org/10.1111/j.1524-4733.2005.04054.x
[12] Hair, J. F., Black, W. C., Babin, B. J., & Anderson, R. E. (2009).
“Multivariate Data Analysis (7th Edition) (7th ed.)”. Prentice Hall.
[13] Hinton, P. R. (1997). “Statistics Explained: A Guide for Social Science Students”. New York: Routledge.
[14] Wheaton, B., Muthen, B., Alwin, D. F., & Summers, G. (1977).
“Assessing Reliability and Stability in Panel Models.” In Sociological Methodology, 8, 84–136.
[15] Schermelleh-Engel, K., Moosbrugger, H., & Müller, H. (2003).
“Evaluating the fit of structural equation models: Tests of significance and descriptive goodness-of-fit measures.” In MPR-Online, 8(May 2003), 23–74.
[16] Wijanto, S. (2008). “Structural Equation Modelling dengan Lisrel 8.8.”
Yogyakarta: Graha Ilmu.
[17] Hu, L. T., & Bentler, P. M. (1995). “Evaluating model fit. In R. H.
Hoyle (Ed.). In tructural equation modeling: Concepts, issues, and applications.” California: SAGE Publications Inc.
[18] Hair Jr., J. F., Tatham, R. L., & William, C. (1995). “Multivariate Data Analysis with Readings.” New Jersey: Prentice-Hall.
[19] Tabachnick, B. G., & Fidell, L. S. (2006). “Using Multivariate Statistics.” Boston: Pearson Education, Inc.
[20] Fornell, C., & Larcker, D. F. (1981). “Evaluating Structural Equation Models with Unobservable Variables and Measurement Error.” In Journal of Marketing Research, 39–50.
