Sentiment Analysis of Simobi Plus Mobile Application Using Naïve Bayes Classification
Stevan Hamonangan Hardi*, Kristoko Dwi Hartomo
Faculty of Information Technology, Information System, Satya Wacana Christian University, Salatiga, Indonesia Email: 1,*682019133@student.uksw.edu, 2kristoko@uksw.edu
Correspondence Author Email: 682019133@student.uksw.edu
Abstract−Sinar Mas Bank is one of many banks operating in Indonesia. Quite a few people use Sinar Mas Bank's services as their bank of choice for their day-to-day transactions. By popular demand, Sinar Mas Bank serves users of banking services by creating an M-banking application. The M-banking application created by Bank Sinar Mas is called Simobi Plus Mobile Banking. There are already 52.3 thousand reviews regardings this application on the Google Play Store platform. Among these are positive and negative reviews from customers who use the application for their daily transactions. In reviews that use 1-5 star ratings, many people are misled by giving different ratings than the given stars. Many customers who leave 5-star app reviews, but comments on these reviews contain negative words. As a result, the application developer becomes confused because the comments given do not match the rating given by the user. Comments that are not in accordance with the rating given can involve the developer of the application to make improvements or development for the application. Therefore, Research should be conducted using techniques and analytics to categorize the user comments into several groups. This study uses sentiment analysis using the Naive Bayes method to capture positive and negative sentiments for comments on the Simobi Plus mobile banking application on the Google Play store, so that these sentiments have the appropriate value. The accuracy scores for the negative class, positive class, recall, and mood analysis are used to evaluate the test. The resulting value has an accuracy of 99%, which is almost perfect. The precision value was 100%, whereas the recall class produced a value of 98%
(positive class: negative). And the AUC value is 0.980.
Keywords: Simobi Plus Mobile Banking; Sentiment Analysis; Naïve Bayes Classification; Google Play Store
1. INTRODUCTION
Currently, the growth in smartphone use in Indonesia is rapid. Indonesia has a population of 250 million, making it an important market for the rapid growth of smartphone users. In early 2022, the Research Company Data Reportal reported that the number of mobile devices used in Indonesia will reached 370.1 million [1]. According to data reported by the Central Statistics Agency (BPS), in 2022, 67.88% of the Indonesian population aged five years and over already own a smartphone [2], which is an attractive market opportunity for some groups of state and foreign companies to try to market their applications so that they can be useful for people's lives. Indonesia’s financial industry is no exception to this trend. Recently, the banking industry in Indonesia has developed various financial innovations, where customers can only make transactions at banks; they have been transformed into technology-based banking, where customers can make transactions anywhere and anytime.
One of the features they use to market their products and services is to offer Mobile Banking features.
Mobile banking can carry out all transactions that can be done directly at the bank such as fund transfers, checking balances, investing, paying bills, etc., which can be done only by accessing the mobile phone of the mobile banking application user. This feature in mobile banking removes space and time limitations for conducting banking transactions [3].
One of the banks that has released this feature is Sinar Mas Bank . The name of the application launched by Bank Sinar Mas is Simobi Plus Mobile Banking. Currently, there are more than one million downloads of the Simobi Plus Mobile Banking application on Google Play Store. In its use, there are various types of reviews by users of the application. Users of the application evaluate its use in the comments section of the Google Play Store website. The reviews that users provide for the application can, of course, be very useful for application developers so that the application can be developed according to the wishes of the application users.
Reviews contained in this mobile banking application are entered regularly every day so that these reviews are difficult to distinguish between positive or negative responses. There are also users who give the app a good rating but give the app a bad review. The thing that can influence new users to download an application is to first look at the reviews on the application, whether the application has a good rating or not [4]. The value rating shows the value of the rating given, namely one to five, while the comments show opinions in the form of text which allows users to convey more details about their experiences in using the application [5].
Google Play Store rates an application based on the rating that users give to the application. Ratings are given starting from one which means very bad, to five which means very good. The Google Play Store will shutdown applications that have a rating of one too many because they are deemed useless for use by users. Lots of users are wrong or forget to rate the application when they want to give a review of the application. As a result, users give a rating of one when they give a good review, or users give a rating of five even though the review they provide is very bad.
Given these problems, sentiment analysis is needed to determine the correct value for these comments, whether the comments have negative or positive sentiments. With the results of this sentiment analysis, the developer can find out exactly what the application user wants based on the results of the comments given by the
user regarding the use of the Simobi Plus Mobile Banking application. Sentiment analysis is a branch of research that examines people’s thoughts, emotions, and attitudes [6]. Sentiment analysis works by categorizing the text into phrases or documents, and then determining whether the viewpoint expressed in those words or documents is positive or negative.
Sentiment analysis can also reveal sad, happy, or angry feelings. Expressions or emotions based on a particular discussion and a statement in one discussion can have a different meaning for the same statement in another discussion. This negative positive sentiment analysis can be used by application developers to make it easier to determine the wishes of application users through the positive and negative reviews they provide. Finding the polarity of the content and correlating it with favorable or unfavorable comments or reviews is the goal of this sentiment analysis [7].
Analysis can be performed using the classification method, so that the analysis process becomes faster.
This categorization can be performed using a variety of techniques, including the Naive Bayes method (NB), Support Vector Machine (SVM), and the k-nearest neighbor (kNN) approach [8]. This study uses the Naive Bayes approach to produce opinions about reviews of the Simobi Plus Mobile Banking app on the Google Play Store, both positive and negative.
The Naive Bayes Classifier is a straightforward classification technique that uses simple probabilities and a Bayesian algorithm to demonstrate that there is no association between any two classes [9]. Naive Bayes is also a method that applies probabilistic techniques in which the relationship between one feature and the other features contained in the same data is not related [10]. This model can be said to have good potential in carrying out document classification if it is equated with other methods in terms of accuracy and computational efficiency, because each component can play a role in the final decision of components that are equivalent and independent.
Because the method using this model can be used quite easily, Naive Bayes is often used in sentiment analysis research with maximum accuracy results, even with a small amount of training data.
Sentiment analysis is a text data processing technique that determines whether a sentence contains an opinion, and whether that opinion is positive or negative. The Naive Bayes Classifier algorithm can be said to be suitable for application in sentiment analysis because it has a purpose as a classification method to give positive and negative category values [11].
Sentiment analysis was conducted using data from the results of reviews provided by application users on the Simobi Plus Mobile Banking application on Google PlayStore. These data were obtained using the Web Scraping method. Web scraping is a process of collecting data sourced from the internet [12]. Web Scraping implements data retrieval by browsing HTML documents from websites whose data you want to retrieve for HTML tags, so that you can retrieve the data to be compared to the web scraping application you want to make.
The web scraping process is performed by retrieving data, such as HTML or XHTML. Furthermore, the data were analyzed, after which the required data were obtained from the website. Data mining is the process of gathering data to hunt for semantic patterns or trends from massive amounts of data, but web scraping is not data mining.
Web scraping applications focus on how to obtain data through data retrieval and varying data sizes [13].
After obtaining the data obtained from web scraping, sentiment analysis is forwarded to the next process, namely, using the RapidMiner application. RapidMiner is an open-source software that can be used freely. This program can be used for predictive analysis, text mining, and data mining. The RapidMiner application has various features in the application where these features can provide guidance to application users so that the decision chosen to carry out the analysis is the best decision. Rapidminer developed by the developer using open core model [14]. Based on the findings of the sentiment analysis test using the RapidMiner application's values for the negative and positive classes, recall, and accuracy. The resulting value has an accuracy of 99%, which is almost perfect.
The precision value was 100%, whereas the recall class produced a value of 98% (positive class: negative). And the AUC value is 0.980.
Vynska Amalia Permadi in a journal containing their research entitled "Sentiment Analysis Using Naive Bayes Algorithm Against Reviews Restaurant in Singapore ". This research was conducted using the Naïve Bayes classifier algorithm. This research concludes that sentiment analysis using the Naïve Bayes algorithm can determine the classification of a positive review that is visualized as satisfied or satisfied and a negative review that is visualized as unsatisfied. Based on the test results, sentiment analysis using the naïve Bayes algorithm gives a precision value of 73.02%, a recall of 74%, and an accuracy of 73.33%. The weakness of this research is that the training data they use is still in the form of comments according to the rating given from the satisfied to unsatisfied scale. The researchers did not filter the training data they used based on the manual according to the positive and negative definitions for a comment. As a result, the resulting precision value is relatively low, which is equal to 73.02% [15].
2. RESEARCH METHODOLOGY
This research was conducted in several stages, the first stage was the data collection stage. After that, the preprocessing stage is carried out. After the preprocessing stage, the data separation stage is carried out. After that, the final stage is the analysis and results stage. For details of the stages carried out can be seen in figure 1.
Figure 1. Research Stage
For detailed steps contained in Figure 1 can be seen in the explanation below:
2.2 Data Collection
Based on Figure 1, the first step that must be done is the data separation stage. The process used to collect data related to reviews or comments given by users of the Simobi Plus Mobile Banking application on the Google Playstore platform was obtained using web-scraping techniques. Web Scrapping is carried out using the website from Google, namely Google Colab. Google released a tool for internal research, “Colaboratory, ” which serves as a tool for machine learning research and education. The programming language used was a Jupyter notebook, which requires no setup for use. In simpler terms, this is a jupyter notebook with all Google document collaboration capabilities, where not only one person can work on it at the same time, but more than one person can do so. The main advantage of this feature is that it can be used for free [16]. The results obtained from this web scraping are reviews or comments given by users of the Simobi Plus Mobile Banking application, which were differentiated using the value of the stars given by application users.
2.2 Preprocessing
The second stage which is carried out based on Figure 1 is the preprocessing stage. Preprocessing is the first stage of unstructured data processing [17]. Preprocessing is used to prepare data or documents before proceeding to the next process [18]. The goal is to simplify the query search process for the retrieved data, simplify and speed up the processing of data, and simplify the process of sorting data [19]. There are several further steps in the preprocessing stage, including case folding, which converts all letters in the data to lowercase letters. After that, there is a stage called cleaning, in which erroneous data are eliminated and irregular or messy data are corrected.
At this stage, it is filled in for missing values in the data and discards values that are unnecessary or useless in the data. The next is the remove stopword stage, which is a process for removing useless words or words that have no meaning. After that, tokenization is continued to break sentences into units of words called tokens.
2.2 Data Separation
The third stage which is carried out based on Figure 1 is the data separation stage. The training data and test data are the two types of data that have been labeled (positive and negative). The researcher chooses how to divide the training data from the test data, and this decision must meet the following criteria. The amount of training data must be smaller than that of the test data. The training data must have a larger amount than the test data [20].
2.2 Analysis and Result
The final stage which is carried out based on Figure 1 is the analysis and results stage. At this level, the Naive Bayes algorithm was used as a categorization method. Using Bayes' theorem, naive Bayes classification is a probabilistic classification. The statistical and probability techniques popularized by British scientist Thomas Bayes were used in this classification. Based on previous experience, a technique is utilized to forecast the possibilities that will arise in the future. Because the method used is prediction of the future, this method is often known as Bayes' Theorem.
3. RESULT AND DISCUSSION
3.1 Data Collection for Training and Data Testing
The data collection process was divided into two parts. Data gathering for training purposes was conducted during the first phase. After that, data gathering for use as test data was done in the subsequent stage. Data from comments on the Simobi Plus Mobile Banking application review were used as the training and test data. These data were obtained using a web application, namely Google Colab. The first step was to visit the Google Play Store web page and search for the Simobi Plus Mobile Banking application. Next, the link address of the application is copied and pasted in Python code written on Google Colab. Before that, the Google Play Scrapper library needs to be installed so that we can obtain the data from the URL link. Next, the Python library needs to be installed to use the Python code for extracting the data, and then we used the Python code to extract the data from the URL link.
Subsequently, the data are generated in the form of a CSV file. Lastly, the CSV file from Google Collab need to be downloaded. Later, the downloaded CSV file is used to attempt sentiment analysis on the RapidMiner App.
Figure 2 shows the Python code used to gather data from the web.
Figure 2. Web Scrapping Process
For training data, 400 data were selected. These data are then labeled as positive sentiment and negative sentiment, respectively, with as many as 200, 200 negative sentiments and 200 positive sentiments. The detail can be seen in Tabel 1.
Table 1. Positive and Negative Test Data Separation Sentiment Data
Positive 200 Data Negative 200 Data
400 Data Total Data
After that, the remaining data that is not labeled positive and negative sentiment will be used as test data.
The test data used is 1170 data which contains comments from application users. These data will be separated into another file that is different from the training data file. After that, the label that will be assigned to this test data will be automatically assigned using the RapidMiner application.
3.2 Categorization of Positive Sentiment and Negative Sentiment training data
Labeling of positive and negative sentiments in the comments was performed manually. To create categories for the training data so that the data testing procedure can be carried out, positive and negative labels are manually applied to the training data. This labeling is done by looking at the comments individually contained in the data.
After looking at the comments one by one, sentiments were given based on the researcher's understanding.
Sentiment is divided into negative and positive. An examples of positive sentiment training data can be seen in Table 2. For examples of negative sentiment training data can be seen in Table 3.
Table 2. Positive Sentiment for Training Data Examples Name
Field Number
Sangat mudah dan nyaman menggunakan simobi, sangat banyak membantu dalam hal pembayaran
dll. Positive
Aplikasi zaman sangat bagus, membantu dan memudahkan dalam transaksi Positive
Name
Field Number
Simobi plus. Mobil banking the best semua fitur sdh lengkap. Sangat nyaman untuk di gunakan. Ayo buka rek di bank Sinarmas. Dan download aplikasi. Gratis transfer ke sua bank hanya di bank Sinarmas
Positive
Table 3. Negative Sentiment for Training Data Examples Name
Field Number
Kacau...saldo disimobi tiba kosong...kemana perginya uang saya ? Saya mau trf gak bisa. Itu uang
pesangon saya. Negative
Mau login susah sekali. Loading lama mau bikin username sudah sesuai zaman diminta tapi tidak bisa diklik dan tiba tibs logout sendiri. terimakasih sudah membuang waktu saya. Izin uninstal.
Terimakasih.
Negative Reset password, ribet banget data sudah benar semua malah keterangnnya failed Negative 3.3 Testing Data Analysis Process
The classification process using Naïve Bayes requires word weighting for each word contained in the test data.
Table 4 contains examples of test data that are required to obtain sentiment predictions..
Table 4. Test Data Examples Name
Field Number
Cukup jelek Password nya udah betul tp akun di blokir ?
Kenapa dengan aplikasi ini kok dikit dikit minta di update terus aplikasi nya baru beberapa hari dari di update ini minta update lagi sedangkan disini sinyalnya lemot pula lagi ? Selama ini baik baik saja mudah mudahan baik terus dan gak pernah lemot ?
Drilling and calculation are performed by calculating the probabilities of positive and negative sentiments.
The results of this training data weighting are used to carry out tests on the test data. The assignment of sentiment values is performed by calculating the level of probability generated on the test data with reference to the sentiment values of the training data. The Nave Bayes algorithm was used in the scoring procedure. The Nave Bayes method was used in the RapidMiner application to automatically classify sentiments. A comparison of the weight of each word in the training data was used to calculate the sentiment score on this test set of data. First, the training data must be filtered from the training data using the filter tool. This stage can be seen in Figure 3.
Figure 3. Test Data Filtering Step
The test data entered the pre-processing stage after filtering the training data. The sentences in the data will be reduced at this stage to a single word, and any extra words will then be eliminated using a stopword filter.
Figure 4 illustrates this process.
Figure 4. Preprocessing Step
Subsequently, the Naive Bayes algorithm is used. In this process, the value of TF-IDF from the test data is used for the naïve Bayes model to learn how often a word appears in a sentence. Each word that frequently appears in phrases with a certain sentiment will be examined by the Naive Bayes algorithm to determine whether it is correct and whether it is labeled as having a positive or negative sentiment. Finally, two distinct models must be
saved with the output of the naive bayes algorithm for the data. This model is useful for test data later. The detail of the step can be seen in Figure 5.
Figure 5. Naïve Bayes Algorithm Step
This stage must be completed for naive Bayes to predict unlabeled data later on, whether the data are positive or negative. The Tf-Idf method is used to determine a word's frequency value in a document, including how frequently it appears in sentences [21]. In the next step, the test data are used to compare the tf-idf test data values with the training data tf-idf values studied by Naive Bayes.
For the test data, the first step was the same as that for the training data. The data must be filtered because the test data have not yet received sentiments. Following filtering of the input data, the Naive Bayes algorithm merges the test data with the training data model. This merging stage is needed to examine each word in the test data using the words that have been applied to the Naive Bayes algorithm to determine whether the words contained in the test data have positive or negative sentiments. The details of this merging stage are shown in Figure 6.
Figure 6. Union Step
Next, the test data are given a model to determine whether the sentences in the test data have negative or positive sentiments using the training data model that already has a Naive Bayes value. Each training document is summed for the positive and negative probabilities. Subsequently, the weights of the documents are compared. If the document weight has a greater positive probability, the sentiment result will be positive, whereas if the document weight has a greater negative probability, the sentiment result will be negative. The data for this step are shown in Figure 7.
Figure 7. Naïve Bayes Model for Training Data 3.4 Accuracy Meassurement Results
By using 1,170 user comment data and analyzing them using Rapidminer with the TF-IDF and Term Frequency processes, accuracy, Class Recall and Class Precision values are produced in sentiment analysis. The obtained performance accuracy was 99%. The precision was set at 100%. The results obtained in Class Recall were 98%
(positive class: negative). And the AUC value is 0.980. The detail of naïve bayes classification result to the prediction can be seen in Table 5.
Table 5. Experimental Results Using Naïve Bayes Classification Index
Score Number
Accuracy 99%
Precission 100%
Class : Recall (Positive class:negative) 98%
AUC Value 0.980
3.5 Data Processing Testing Negative and Positive Sentiment Classification
By analyzing the results of testing using the RapidMiner application, a negative sentiment of 54.1% was obtained, which was greater than the positive sentiment that only scored 45.9%. The outcomes are inversely related to the value attained by the Simobi Plus mobile banking application on the Google Play Store platform. The review rating chart that can be seen on the Simobi Plus Mobile Banking application has a greater presentation of 4 and 5 star ratings than the percentage of one star and 2 star ratings. Three stars were not counted, because they were considered neutral.
4. CONCLUSION
Thus, it can be proven that sentiment analysis can be performed automatically using the Naïve Bayes method.
Using the RapidMiner application and a technique for classifying both positive and negative sentiments, sentiment analysis was conducted. From the results obtained from testing as many as 1, 170 training data points using RapidMiner, the performance accuracy value was 99%, the precision result was 100%, and the Class Recall result was 98% (positive class: negative). The AUC value was 0.980. From the results of the research related to the test data above, it was found that the presentation of negative sentiment comments was 54.1%, which was larger than the presentation of positive sentiment comments, which amounted to only 45.9%. From the results of this study, words that often appear in the comments of users of the Simobi Plus Mobile Banking application can be found.
Words that frequently appear are the word 'application' which appears 210 times in 180 documents, the word 'bank' which appears 116 times in 93 documents, the word 'update' which appears 105 times in 91 documents, and the word 'simobi. ' which appears 102 times in 96 documents. From the discovery of words that often appear, it can be said that many negative comments contain the word update where researchers see many negative comments about applications that update frequently, so that users feel irritated and give bad reviews on this application. By looking at the word 'simobi' which also has a large percentage of appearances, it can be said that negative comments regarding user reviews mention the name of the application a lot.
REFERENCES
[1] Naomi Adisty, “Mengulik Perkembangan Penggunaan Smartphone di Indonesia,” https://goodstats.id/article/mengulik- perkembangan-penggunaan-smartphone-di-indonesia-sT2LA, 5 November 2022.
[2] Adi Ahdiat, “67% Penduduk Indonesia Punya Handphone pada 2022, Ini Sebarannya,”
https://databoks.katadata.co.id/datapublish/2023/03/08/67-penduduk-indonesia-punya-handphone-pada-2022-ini- sebarannya#:~:text=Menurut%20data%20Badan%20Pusat%20Statistik,rekor%20tertinggi%20dalam%20sedekade%2 0terakhir, 8 Maret 2023.
[3] M. Khoirul, U. Hayati, dan O. Nurdiawan, “ANALISIS SENTIMEN APLIKASI BRIMO PADA ULASAN PENGGUNA DI GOOGLE PLAY MENGGUNAKAN ALGORITMA NAIVE BAYES,” 2023.
[4] E. Fitri, Y. Yuliani, S. Rosyida, dan W. Gata, “Analisis Sentimen Terhadap Aplikasi Ruangguru Menggunakan Algoritma Naive Bayes, Random Forest Dan Support Vector Machine,” TRANSFORMTIKA, vol. 18, no. 1, hlm. 71–80, 2020, [Daring]. Tersedia pada: www.nusamandiri.ac.id,
[5] A. Mittal dan S. Patidar, “Sentiment analysis on twitter data: A survey,” dalam ACM International Conference Proceeding Series, Association for Computing Machinery, Jul 2019, hlm. 91–95. doi: 10.1145/3348445.3348466.
[6] R. Wahyudi dkk., “Analisis Sentimen pada review Aplikasi Grab di Google Play Store Menggunakan Support Vector Machine,” JURNAL INFORMATIKA, vol. 8, no. 2, 2021, [Daring]. Tersedia pada:
http://ejournal.bsi.ac.id/ejurnal/index.php/ji
[7] M. N. Akbar, N. Hasanahlmar’iyah Rusydi, M. Hasrul, dan S. Ramadhanti, “Sentiment Analysis Terhadap Review Aplikasi Maxim di Google Play Store Menggunakan Support Vector Machine (SVM),” vol. 2, no. 2, hlm. 1, 2022.
[8] J. W. Iskandar dan Y. Nataliani, “Perbandingan Naïve Bayes, SVM, dan k-NN untuk Analisis Sentimen Gadget Berbasis Aspek,” Jurnal RESTI (Rekayasa Sistem dan Teknologi Informasi), vol. 5, no. 6, hlm. 1120–1126, Des 2021, doi:
10.29207/resti.v5i6.3588.
[9] P. S. M. Suryani, L. Linawati, dan K. O. Saputra, “Penggunaan Metode Naïve Bayes Classifier pada Analisis Sentimen Facebook Berbahasa Indonesia,” Majalah Ilmiah Teknologi Elektro, vol. 18, no. 1, hlm. 145, Mei 2019, doi:
10.24843/mite.2019.v18i01.p22.
[10] B. Kurniadi Widodo, N. Hafifah Matondang, dan D. Sandya Prasvita, “Penerapan Algoritma Naive Bayes Untuk Analisis Sentimen Penggunaan Aplikasi Jobstreet Implementation of Naive Bayes Algorithm For Sentiment Analysis Of The Use Of Jobstreet Application,” 2022.
[11] A. V. Sudiantoro dan E. Zuliarso, ANALISIS SENTIMEN TWITTER MENGGUNAKAN TEXT MINING DENGAN ALGORITMA NAÏVE BAYES CLASSIFIER. 2018.
[12] Ryan Mitchell, Web scraping with Python: collecting data from the modern web, 4 ed. 2018.
[13] D. Deviacita dkk., “Implementasi Web Scraping untuk Pengambilan Data pada Situs Marketplace,” vol. 7, no. 4, 2019.
[14] E. Fadilah, “Implementasi Metode Profile Matching Terhadap Sistem Pendukung Keputusan Penerimaan Dana Zakat pada Badan Amil Zakat Pertamina (BAZMA),” MATICS, vol. 10, no. 2, hlm. 39, Mar 2019, doi: 10.18860/mat.v10i2.5745.
[15] V. A. Permadi, “Analisis Sentimen Menggunakan Algritma Naïve Bayes Terhadap Review Restoran di Singapura 141,”
2020. [Daring]. Tersedia pada: https://www.kaggle.com/hj5992/restaurantreviews
[16] Anirudh Aggarwal, “Introduction To Google Colab,” https://medium.com/@animaze97/introduction-to-google-colab- 9b2e28fe691a, 20 Maret 2018.
[17] T. Krisdiyanto, E. Maricha, dan O. Nurharyanto, “Analisis Sentimen Opini Masyarakat Indonesia Terhadap Kebijakan PPKM pada Media Sosial Twitter Menggunakan Naïve Bayes Clasifiers,” Jurnal CoreIT, vol. 7, no. 1, 2021.
[18] K. Ivana Ruslim dan P. Pandu Adikara, “Analisis Sentimen Pada Ulasan Aplikasi Mobile Banking Menggunakan Metode Support Vector Machine dan Lexicon Based Features,” 2019. [Daring]. Tersedia pada: http://j-ptiik.ub.ac.id
[19] Y. Cahyono, “Analisis Sentiment Tweets Berbahasa Sunda Menggunakan Naive Bayes Classifier dengan Seleksi Feature Chi Squared Statistic,” vol. 4, no. 3, 2019, [Daring]. Tersedia pada: http://openjournal.unpam.ac.id/index.php/informatika [20] R. Apriani dkk., “ANALISIS SENTIMEN DENGAN NAÏVE BAYES TERHADAP KOMENTAR APLIKASI
TOKOPEDIA,” 2019.
[21] A. Bayhaqy, S. Sfenrianto, K. Nainggolan, dan E. R. Kaburuan, “Sentiment Analysis about E-Commerce from Tweets Using Decision Tree, K-Nearest Neighbor, and Naïve Bayes,” dalam 2018 International Conference on Orange Technologies, ICOT 2018, Institute of Electrical and Electronics Engineers Inc., Jul 2018. doi:
10.1109/ICOT.2018.8705796.
