Copyright © 2022, Mahdiyah ‘Afifah Sari. This is an open access article distributed under the Creative Commons Attribution License, which permits
Usability Evaluation in Knowledge Management System (KMS) Using System Usability Scale (SUS) Method
Mahdiyah ‘Afifah Sari*, Ken Ditha Tania
Departement of Information System, University of Sriwijaya, Palembang, Indonesia Email: 1,*[email protected], 2[email protected]
Coressponding Author: [email protected] Submitted: 11/06/2022; Accepted: 24/11/2022; Published: 30/11/2022
Abstract−An unattractive system interface and the lack of user motivation in using the Knowledge Management System (KMS) is one of the reason why usability testing is needed. Usability testing aims to determine the usability of the system, whether the system is in accordance with the purpose of its application. Usability Testing will uses System Usability Scale (SUS) method which involves users in the test. The results of this study are usability scores and recommendations for improvement that have been developed using the prototyping method based on the results of problem identification. For the problem identification, researcher will uses interview technique with 5whys method. After all, the score results is 80.25 for prototype of system improvement. This research can be used as a reference in the development of the system later.
Keywords: Usability; Usability Testing; Knowledge Management System; System Usability Scale; Prototype
1. INTRODUCTION
In optimizing performance to create the advantages for the company, one of the ways that company can be done in maintaining competitiveness is by utilizing all available resources such as knowledge. This is expressed by [1] that an organization needs knowledge to be able to survive in a rapid change. The process of utilizing resources can be achieved by implementing Knowledge Management (KM) which aims to create, use, share, and receive knowledge. To support the implementation of Knowledge Management, a Knowledge Management System is implemented which can be a place to store or share and receive knowledge. However, the implementation of Knowledge Management does not always run as it planned, as it is known in Knowledge Management there are several aspects or goals, one of them is Knowledge Sharing.
According to [2] at least in the Knowledge Sharing process, there are more than one individu, one individu as knowledge giver and the other one as knowledge recipient.
At PT.Telekomunikasi Indonesia Witel, South Sumatra, the application of knowledge sharing is not optimal because the user's desire to use the system is low. One of the reason is caused by a lack of user motivation and an unattractive system interface. Based on what was revealed by [3] that a poorly designed system can make users reluctant to use it. Therefore, usability testing is needed to determine the usability of the system in achieving the objectives of implementing a Knowledge Management System (KMS). Usability Testing will use the System Usability Scale (SUS) method which aims to determine the level of user satisfaction in using the system. This method involves user opinions making it easier to make improvements according to user needs.
As revealed by [3] that usability testing is important to determine the level of usability. Usability Testing using the System Usability Scale (SUS) method has been carried out in several fields such as the education sector, namely by [4][5][6] and [7], government fields such as [8][9][10], to several applications intended for the public interest such as e- commerce and news applications [11][12]. In addition, evaluations on the implementation of the Knowledge Management System (KMS) have been carried out in several companies with various objectives such as [13][14].
The System Usability Scale (SUS) in the education sector is carried out on several platforms for example on e- learning, this is due to the high number of Covid-19 cases that make the uses of e-learning increased in online learning programme. Research conducted by [4] that analyzing the usability of e-learning, it was found that e-learning has a low score with a low acceptance rate so improvements are needed. And then research by [6] by conducting Usability Testing on Microsoft Teams which is an alternative tool in online learning programme, this research uses three methods, namely System Usability Scale (SUS), Human Computer Interaction and Technology Acceptance Model (TAM). Another research by [7] conducted usability testing on Google Classroom which is one of the most widely used platforms in the online learning programme and the results were 82.8 or the system was acceptable. In addition to usability testing on the e-learning platform, there are research that conducted usability testing on library website, for example a research by [5]
conducted an usability testing on the University of Indonesia library website. The study was conducted on early semester students with final semester students, testing was carried out to maintain the quality of the system in order to maintain user’s interest.
The use of the System Usability Scale (SUS) is not only used in the education sector, but also on government platforms. For example with research by [8] who conducted research for Usability Testing on the PLN Mobile application.
This is based on low user interest, as seen from user ratings on Google Playstore. Based on the service users of PT. PLN and application users there are several gap, therefore this research conducts usability testing to identify and fix the problem. Furthermore, research conducted by [9] used the System Usability Scale (SUS) for testing the acceptance of the population Information System in Tegal City. This study explains that the System Usability Scale (SUS) can not only be used for Usability Testing but also can be used for testing the acceptance of a newly developed system. Then, the research
proposed by [10] who carried out usability testing on the administrative System that had been created with the aim of creating an administrative order, the results showed that the system had a high score with excellent category.
In addition to education and government, usability testing is also often carried out on public platforms that are used by many people, such as research conducted by [12] which tested on Shopee application. As one of the well-known e-commerce in Indonesia, the interface has an important role, so usability esting is carried out to measure the level of usability of Shopee and the results are that this system is included in the OK category. In addition, research conducted by [11] testing was carried out twice using the System Usability Scale (SUS) method, after the first test the results will be analyzed and recommendations for improvement are obtained, the results of this study show that the system is still acceptable, with the existing improvements, it is expected to improve usability.
The use of the System Usability Scale (SUS) method has also been used in the Knowledge Management System (KMS) as done by [13] who proposed the incorporation of a new model in Knowledge Management (KM) as a goal to improve performance. Researchers used the System Usability Scale (SUS) method to determine the level of acceptance of the new model, the results showed that the system was accepted but had not yet reached the ideal score. Meanwhile, based on research by [14] who built a Knowledge Management System (KMS) with the aim to be place where the health workers can store, share or receive new knowledge, testing was carried out on 30 respondents and the results showed that the system had good usability with effectiveness is 80.88%.
2. RESEARCH METHODOLOGY
2.1 Testing Method
2.1.1 System Usability Scale (SUS)
This method involves user opinions that making it easier to identify problems and identify user needs. This is by [15] said that generally the use of the method is carried out on people who have used the system or users of the system. The System Usability Scale (SUS) has 10 statement items that have been interpreted into several languages such as Arabic [16], several other languages such as [17], and Indonesian by [18], The statement is as follows:
Table 1. Item in Indonesia No. Item in Indonesia
1. hh Saya Saya ber Saya berpikir akan menggunakan sistem ini lagi.
2. Saya merasa sistem ini rurmit untuk digunakan.
3. Saya merasa sistem ini mudah untuk digunakan.
4. Saya membutuhkan bantuan dari orang lain atau teknisi dalam menggunakan sistem ini 5. Saya merasa fitur-fitr sistem ini berjalan dengan semestinya.
6. Saya merasa ada banyak hal yang tidak konsisten (tidak serasi) pada sistem ini.
7. Saya merasa orang lain akan memahami cara menggunakan sistem ini dengan cepat.
8. Saya merasa sistem ini membingungkan.
9. Saya merasa tidak ada hambatan dalam menggunakan sistem ini.
10. Saya perlu membiasakan diri terlebih dahulu sebelum menggunakan sistem ini.
As stated by [15] the aims of uses odd and even question is to make respondent more focus when reading the questionnaire. The score is calculated by subtracting the value of the statement. The positive statement is reduced by 1 to X-1 and the negative statement is 5 which reduces the value of the statement to 5-X. The average calculation is as follows:
𝑥 = ∑ 𝑥
𝑛 (1) Description:
x = Average Score
∑ x = Total SUS Score n = Number of Respondents
There are three assessment indicators on the System Usability Scale (SUS) which are described in the following figure :
Figure 1. Assessment Indicators on the System Usability Scale (SUS)
The use of this method will be carried out twice, at the beginning of the study and at the end of the study to make comparison between before and after improvements.
2.2 System Development Method 2.2.1 Prototyping
According to [19] prototyping is described as a tool to communicate with ideas. Prototyping helps implement ideas, communicates specific aspects of a solution and helps clarify to users. According to [20] there are 4 stages in prototyping, namely as follows:
Figure 2. Prototyping method
The communication step is carried out by identifying problems and solutions that will use the deep interview technique and 5Whys method to find the root of the problem by asking why questions repeatedly. As well as using the Brainwriting method in finding a solution to the problem. While the Quick plan & Quick Design stages will use a Low Fidelity Prototype, namely Wireframe, these method will continue step by step until an interactive prototype is obtained that suits the user's needed.
3. RESULT AND DISCUSSION
3.1 Usability Testing
Researcher conducted usability testing on 10 respondents, this is based on [21] which says that the System Usability Scale (SUS) test can be carried out on 8-12 respondents, and based on [22] that the System The Usability Scale (SUS) does not have implicit rules about the number of research respondents. All respondents filled out the questionnaire on a scale of 1 to 5 with 1 being strongly disagreed to 5 being strongly agreeing and a score of 3 being neutral. The respondent column shows the number of respondents, namely 10 people, the initial test used paper media. It can be seen in the table that the average usability at KAMPIUN or Knowledge Management System (KMS) is 61.25, which means that if interpreted into indicators, the results are Low Acceptable and into the OK category with a Grade Scale of D. So some improvements are needed.
Table 2. Usability Testing Score
No Responden Skor Total SUS
Value Q1 Q2 Q3 Q4 Q5 Q6 Q7 Q8 Q9 Q10
1 Responden 1 2 3 3 4 4 3 3 3 2 4 17 42.5
2 Responden 2 4 3 3 2 4 2 5 2 4 2 29 72.5
3 Responden 3 4 2 4 4 4 2 2 2 5 4 25 62.5
4 Responden 4 4 2 4 2 4 2 4 2 4 2 30 75
5 Responden 5 4 3 3 2 3 4 3 3 1 2 20 50
6 Responden 6 4 4 3 4 4 3 4 2 2 4 20 50
7 Responden 7 4 2 4 1 4 2 4 2 3 4 28 70
8 Responden 8 5 1 5 1 5 1 5 1 5 1 40 100
9 Responden 9 3 2 4 4 2 4 3 4 2 4 16 40
10 Responden 10 3 3 3 3 3 3 4 3 3 4 20 50
61.25 As for after the average score is entered into the indicators on the System Usability Scale (SUS) it is as follows:
Figure 3. Assessment Indicators on the first testing
Quick Plan &
Quick Design
Prototype Construct ion Delivery
&
Feedback Communi cation
3.2 Development Steps
3.2.1 Identification of Problems and Solutions
This step uses the Prototyping Method with two cycles. After successfully identifying problems and solutions at the Communication step, the final results of the planned improvements after going through two cycles are as follows:
Table 3. Problems and Solutions Code of
Problem
Code of
Solution Problem Description Solution Plan
M-1 P-1 Nothing requires the user to actively use KAMPIUN
Added Gamification to make users feel challenged to use KAMPIUN
Connecting KAMPIUN with the User Absence Application
Added a feature that displays achievements.
Added a feature that displays achievements.
M-2 P-2 The knowledge verification process that will be published is not transparent and takes quite a long time
Adding information about the verificator to make it easier to follow-up Knowledge.
Added a feature that explains the progress or lack of Knowledge that makes Knowledge not yet published so that it can be quickly repaired
M-3 P-3 Knowledge is displayed in a monotonous and boring form (such as a long article) so that it makes users reluctant to read it.
Added a more interesting knowledge format such as in the form of a video so that users are more interested in seeking knowledge.
Do not eliminate the article format (in writing) because not everyone likes or can share knowledge in other formats.
M-4 P-4 Knowledge is not sorted by year so it is difficult if you want to find old knowledge because you have to search one by one
Added sort by year feature to make it easier to find knowledge
3.2.2 Prototype Recommendation
After going through several steps such as Low Fidelity Prototype to High Fidelity Prototype and going through the evaluation steps, the results obtained in the form of recommendations for improvement, namely interactive prototypes as follows:
a. Add Gamification
Figure 4. Homepage
At homepage interface, a Leaderboard is displayed which explains the Top Point and The Most Active User. This is made with the aim of motivating users to collect points. This feature is placed at the top of the page to make it more visible and attractive when the user opens the system. Besides that, at my achievement menu there are also several challenges for users to make users more enthusiastic, users can also check on their history of point achievement.
Figure 5. My Achievement b. Add New Types of Knowledge
One of the reason why users feel reluctant to use Knowledge Management System (KMS) is because the type of knowledge that is boring. Based on brainstorming step results, it is suggested to add more interesting types of knowledge, for example video. But, because not all users understand how to make knowledge with video type, it suggested to adding new types without eliminating the old types of knowledge.
Figure 6. Add New Document c. Add “Sort By Year” Feature
Figure 7. Search
In the previous system interface, users had to search manually if they wanted to search for knowledge from different years or different category, a feature was made to filter by year, category and type of knowledge. So it is useful to find the required knowledge.
d. Add “InProgress” Feature
Figure 8. InProgress
In the previous system interface, if users want to share knowledge, they have to wait for months without knowing anything about their works, their progress, or improvement because there is no explanation about the knowledge they submitted. With this feature, it can make it easier for users to know the progress and make it easier if they want to communicate with verificator.
3.3 Testing and Comparison
There are several tasks that the respondent must carry out before carrying out the final test. Testing of this task is carried out online and offline. Some of the tasks that must be carried out are as follows:
Table 4. Task Scenario
No Task
1. Log In
2. Adding Knowledge 3. Search Knowledge 4. Giving Feedback 5. View Achievement 6. Log Out
After successfully carrying out the task mentioned above, a retest was carried out with the same number of respondents as the initial test, there are 10 respondents using the System Usability Scale (SUS) questionnaire. The test results are obtained as follows:
Figure 9. Assessment Indicators on the final testing
The SUS score in the final test was 80.25 with Acceptability, Grade Scale and Adjective Rating level is Acceptable with Grade Scale B and included in the Excellent category. The comparison of SUS scores in the initial and final tests is as follows:
Table 5. Comparison
Nilai Usability
Pengujian pada KAMPIUN 61.25
Pengujian pada Prototype KAMPIUN 80.25
Peningkatan 19
After usability testing was carried out on the prototype, it was found that there was an increase in the usability score of 19%.
4. CONCLUSION
In accordance with the purpose of this research, namely to conduct a usefulness test on KAMPIUN or the Knowledge Management System (KMS). At the same time, this research provides recommendations for improvements to be made based on ideas, problems and solutions from KAMPIUN users themselves, so it is hoped that the results of this research can be a reference and recommendation for future system development. The usability test uses the System Usability Scale (SUS) method and the Interview Technique with the 5Whys Method to identify problems. Usability Tests using the System Usability Scale (SUS) were carried out twice, namely on the System and on the Development Prototype to compare the level of usability. The results showed that there was an increase of 19% after retesting the prototype. With an increase in the scale from being between OK and Good to Excellent.
REFERENCES
[1] J. Karamat, T. Shurong, N. Ahmad, A. Waheed, and S. Khan, “Barriers to knowledge management in the health sector of Pakistan,” Sustain., vol. 10, no. 11, 2018, doi: 10.3390/su10114155.
[2] T. P. L. Nguyen, N. M. Tran, X. H. Doan, and H. Van Nguyen, “The impact of knowledge sharing on innovative work behavior of Vietnam telecommunications enterprises employees,” Manag. Sci. Lett., vol. 10, no. 1, pp. 53–62, 2020, doi:
10.5267/j.msl.2019.8.016.
[3] S. Zikos, M. Tsourma, E. E. Lithoxoidou, A. Drosou, D. Ioannidis, and D. Tzovaras, “User Acceptance Evaluation of a Gamified Knowledge Sharing Platform for Use in Industrial Environments,” Int. J. Serious Games, vol. 6, no. 2, pp. 89–108, 2019, doi:
10.17083/ijsg.v6i2.275.
[4] D. Supriyadi, S. Thya Safitri, and D. Y. Kristiyanto, “Higher Education e-Learning Usability Analysis Using System Usability Scale,” Int. J. Inf. Syst. Technol. Akreditasi, vol. 4, no. 1, pp. 436–446, 2020.
[5] M. L. Nuriman and N. Mayesti, “EVALUASI KETERGUNAAN WEBSITE PERPUSTAKAAN UNIVERSITAS INDONESIA MENGGUNAKAN SYSTEM USABILITY SCALE,” BACA J. DOKUMENTASI DAN Inf., vol. 41, no. 2, 2020, doi:
10.14203/j.baca.v41i2.622.
[6] D. Pal and V. Vanijja, “Perceived usability evaluation of Microsoft Teams as an online learning platform during COVID-19 using system usability scale and technology acceptance model in India,” Child. Youth Serv. Rev., vol. 119, 2020, doi:
10.1016/j.childyouth.2020.105535.
[7] D. Setiawan and S. L. Wicaksono, “Evaluasi Usability Google Classroom Menggunakan System Usability Scale,” Walisongo J.
Inf. Technol., vol. 2, no. 1, 2020, doi: 10.21580/wjit.2020.2.1.5792.
[8] E. Kaban, K. C. Brata, and A. H. Brata, “Evaluasi Usability Menggunakan Metode System Usability Scale (SUS) Dan Discovery Prototyping Pada Aplikasi PLN Mobile (Studi Kasus Pt. PLN),” J. Pengemb. Teknol. Inf. dan Ilmu Komputer; Vol 4 No 10, vol.
4, no. 10, pp. 3281–3290, 2020, [Online]. Available: https://j-ptiik.ub.ac.id/index.php/j-ptiik/article/view/7941.
[9] G. W. Sasmito, L. O. M. Zulfiqar, and M. Nishom, “Usability Testing based on System Usability Scale and Net Promoter Score,”
2019 2nd Int. Semin. Res. Inf. Technol. Intell. Syst. ISRITI 2019, no. October, pp. 540–545, 2019, doi:
10.1109/ISRITI48646.2019.9034666.
[10] U. Ependi, A. Putra, and F. Panjaitan, “Evaluasi tingkat kebergunaan aplikasi administrasi penduduk menggunakan teknik system usability scale,” Regist. J. Ilm. Teknol. Sist. Inf., vol. 5, no. 1, 2019, doi: 10.26594/register.v5i1.1412.
[11] D. W. Ramadhan, “PENGUJIAN USABILITY WEBSITE TIME EXCELINDO MENGGUNAKAN SYSTEM USABILITY SCALE (SUS) (sTUDI KASUS: WEBSITE TIME EXCELINDO),” JIPI (Jurnal Ilm. Penelit. dan Pembelajaran Inform., vol.
4, no. 2, p. 139, 2019, doi: 10.29100/jipi.v4i2.977.
[12] F. G. Sembodo, G. F. Fitriana, and N. A. Prasetyo, “Evaluasi Usability Website Shopee Menggunakan System Usability Scale (SUS),” J. Appl. Informatics Comput., vol. 5, no. 2, pp. 146–150, Nov. 2021, doi: 10.30871/jaic.v5i2.3293.
[13] R. Hidayad and S. Fauziati, “Analisa Integrasi Knowledge Management System, Learning Management System, Dan Eportfolio Menggunakan Knowledge Management Performance Scale Dan System Usability Scale (Sus),” Edu Komputika J., vol. 5, no. 2, pp. 117–127, 2018.
[14] H. Hayurani, H. Hamnah, U. A. Rachmawati, and E. Suherlan, “HelloCare: APLIKASI MANAJEMEN PENGETAHUAN BERBASIS ANDROID UNTUK TENAGA KESEHATAN,” J. Teknol. Inf. Yars., vol. 6, no. 1, pp. 25–34, 2019, doi:
10.33476/jtiy.v6i1.599.
[15] J. Brooke, “SUS: A ‘Quick and Dirty’ Usability Scale,” Usability Eval. Ind., no. November 1995, pp. 207–212, 2013, doi:
10.1201/9781498710411-35.
[16] B. A. Alghannam, M. Alsuwaidi, and W. Almayyan, “Perceived Usability Using Arabic System Usability Scale (A-SUS):
Student Perspective of Smart PAAET App,” Int. J. Comput. Sci. Inf. Secur., vol. 16, no. 7, p. 66, 2018, [Online]. Available:
https://sites.google.com/site/ijcsis/.
[17] M. Gao, P. Kortum, and F. L. Oswald, “Multi-Language Toolkit for the System Usability Scale,” Int. J. Hum. Comput. Interact., vol. 36, no. 20, pp. 1883–1901, 2020, doi: 10.1080/10447318.2020.1801173.
[18] Z. Sharfina and H. B. Santoso, “An Indonesian adaptation of the System Usability Scale (SUS),” 2017, doi:
10.1109/ICACSIS.2016.7872776.
[19] R. Krone, “Not Just Guess Work: Tips for Observation, Brainstorming, and Prototyping,” Tech. Vasc. Interv. Radiol., vol. 20, no. 2, pp. 94–100, 2017, doi: 10.1053/j.tvir.2017.04.002.
[20] K. Widhiyanti and A. K. P. Atmani, “Penerapan Metode Prototyping Dalam Perancangan Interface Sistem Unggah Portofolio Penerimaan Mahasiswa Baru Diploma ISI Yogyakarta,” Teknika, vol. 10, no. 2, 2021, doi: 10.34148/teknika.v10i2.308.
[21] J. Brooke, “SUS : A Retrospective,” no. June, 2013.
[22] U. Ependi, T. B. Kurniawan, and F. Panjaitan, “SYSTEM USABILITY SCALE VS HEURISTIC EVALUATION: A REVIEW,”
Simetris J. Tek. Mesin, Elektro dan Ilmu Komput., vol. 10, no. 1, 2019, doi: 10.24176/simet.v10i1.2725.
