Comparison of IndoBERTweet and Support Vector Machine on Sentiment Analysis of Racing Circuit Construction in Indonesia
Hanvito Michael Lee, Yuliant Sibaroni
School of Computing, Informatics, Telkom University, Bandung, Indonesia Email: 1,*hanvitomichaellee@student.telkomuniversity.ac.id, 2yuliant@telkomuniversity.ac.id
Correspondence Author Email: hanvitomichaellee@student.telkomuniversity.ac.id
Abstract−The construction of the circuit is one of the policies made by the Indonesian government to advance the tourism sector and improve the national economy. This policy triggers various opinions given by the public, primarily through social media Twitter, both in the form of positive and negative opinions. This study compares machine learning and deep learning algorithms, Support Vector Machine and IndoBERTweet, that will be used as a model to predict the sentiment of racing circuit construction tweets. These models are built with K-Fold cross-validation to obtain the overall model’s performance for the entire dataset. Based on the experiments that have been carried out, it shows that IndoBERTweet performs better than the Support Vector Machine, with an overall accuracy score of 86%, a precision score of 88.2%, a recall score of 88.6%, and an f1-score of 88.4% for the entire dataset. Meanwhile, the Support Vector Machine model only achieves 82% for the accuracy score, 87.3% for the precision score, 84.3% for the recall score, and 85.8% for the f1-score. In addition, the best accuracy value from each iteration for IndoBERTweet is 94%, and the Support Vector Machine is 93%.
Keywords: IndoBERTweet; K-Fold Cross Validation; Racing Circuit; Sentiment Analysis; Support Vector Machine.
1. INTRODUCTION
In this modern era, many sports sectors are starting to develop and are in great demand by the public, including the automotive sector. Automotive is an industry that produces vehicles as a means of transportation to meet people's needs [1]. Sports in this sector require facilities to be held in the form of a racing arena (circuit) to create a safe environment for racers. To have an international racing event such as MotoGP, Formula 1, and Formula E requires a circuit that meets the standards set by the organizers of these prestigious races.
Indonesia, a developing country that wants to advance the tourism sector to advance the national economy, is starting to see sports in the automotive industry as a promising sector since it has many benefits to the country that held the events [2]. This is evidenced by the construction of the Mandalika International Circuit in Lombok, the Formula E Circuit in Jakarta, and the Bintan International Circuit in Riau by the Indonesian government to hold international racing events. The construction of a racing circuit in Indonesia has sparked various responses from the public, both negative and positive. These multiple responses can happen because only some are satisfied with this policy. Some people support this policy because it is considered capable of helping the country's economy, and others do not support it because it is deemed detrimental to small traders who sell around the circuits [3]. The public also expressed many responses related to this policy through social media, especially Twitter. The freedom of expression everyone can express on Twitter makes this social media often used as a source of data to analyze the sentiment of public opinion.
With various kinds of tweets regarding the satisfaction or disappointment of the community regarding the construction of these circuits, sentiment analysis of public opinion regarding this issue will be carried out by collecting data via Twitter. Based on the results of this sentiment, it is expected to provide an overview for the government in making decisions to be able to minimize losses that may occur with this policy to the community.
Much research on sentiment analysis via Twitter has been carried out using various machine learning algorithms, such as research conducted in research [4] which examined sentiment from tweet data for an internet service provider in Indonesia using a Support Vector Machine with the addition of TF-IDF feature extraction. This study resulted in an accuracy of 87%, a precision of 86%, a recall of 95%, and an error rate of 13%. A similar study was also conducted in research [5], which examined sentiment from tweet data for e-commerce in Indonesia which had an accuracy of 93%. Similar research was conducted in the study [6], which examined sentiment in a product review using the Naive Bayes algorithm, which resulted in an accuracy and precision of 77.78%, and a recall of 93.3%. Research using the K-Nearest Neighbor was conducted in the study [7], which examined the sentiments of Twitter users regarding the election of Jakarta governor in 2017, which obtained the highest accuracy score of 67.2%. Based on several studies described above, it can be interpreted that the Support Vector Machine algorithm has a better performance than other machine learning algorithms.
Apart from machine learning algorithms, sentiment analysis can also use deep learning. Deep learning provides better performance in many cases, including the issue of sentiment analysis. Much research has been conducted to compare machine learning with deep learning algorithms, especially in sentiment analysis, such as research conducted in research [8] who compares RNN, CNN, and LSTM models with a machine learning algorithm. The research shows that the deep learning model performs better than the machine learning model.
Another research was conducted in research [9] that compares many machine learning and deep learning algorithms. The result of this research shows the difference in model performance is about only 3% between the best machine learning model and the deep learning model with the deep learning has the higher value.
The deep learning model widely used in text processing is the Bidirectional Encoder Representations Transformers (BERT). BERT has examples of models that are trained using Indonesian, including IndoBERT and IndoBERTweet. Previous research was conducted on research [10] which examined sentiment from reviews of Indonesian mobile applications using multilingual IndoBERT and BERT-Base. From those methods, the result was that IndoBERT was better than multilingual BERT-Base with an accuracy value of 84%. Similar research was conducted in the study [11], which compared the performance of IndoBERT and IndoBERTweet in various cases, including sentiment analysis. From this research, IndoBERTweet has higher performance results, with an average value for sentiment analysis of 88.3% compared to IndoBERT, which has a performance value of 86%. Based on the research above, using IndoBERTweet for sentiment analysis has the potential to provide better results.
Referring to various studies in sentiment analysis using the machine learning and deep learning algorithms above, the Support Vector Machine and IndoBERTweet methods have good performance, and IndoBERTweet has the potential to provide better results than Support Vector Machine. However, research on sentiment analysis using IndoBERTweet is still rare. Therefore, this reseach will focus on comparing Support Vector Machine and IndoBERTweet using sentiment data from public opinion on the construction of racing circuits in Indonesia.
2. RESEARCH METHODOLOGY
2.1 System Design
The flowchart of this system is illustrated in Figure 1. The system begins by preparing a dataset by retrieving data from Twitter, removing duplicate data, and labelling data that has been successfully retrieved. Data that has been labelled will enter the preprocessing process, including case folding, data cleaning, stopword removal, and word stemming. After that, the data that has been preprocessed will continue to the modelling process. Modelling was carried out using k-fold cross-validation for each classification, which is IndoBERTweet and Support Vector Machine. After that, each model will be evaluated by summing all the confusion matrix for each k iteration. The matrix will be used to see the overall model’s performance for the entire dataset.
Figure 1. System Flowchart 2.2 Dataset Preparation
Dataset preparation contains three mains process, including data collection, drop duplicate data, and labelling the data. The data used in this system retrieved by crawling the data using Twitter API in Indonesian with “sirkuit mandalika”, and “sirkuit formula e” keywords. After collecting the data, duplicate data will be removed from the dataset and then will be labelled. Data is categorized based on negative and positive sentiments. The category used in this study is based on previous research conducted in research [12]. Labelled dataset illustrated in Table 1.
Table 1. Labelled Dataset
Number Text Label
1 @RadarKorupsi @msaid_didu Bangga untuk
sirkuit Mandalika boleh, tapi hutang bayarlah ! 0 2
RT @Omyung0505 : @Mdy_Asmara1701 Indonesia sudah punya Mandalika Internasional
Sirkuit yg sudah terbukti.
1
3
Kesombongan soal sukses Formula e mendapat hadiah Ranking 1 Udara Terburuk Dunia...
Payah.... https://t.co/shX5u2z1G7
0
2.3 Preprocessing
The preprocessing stage aims to convert data from existing datasets into a format to make it more effective and efficient to get more accurate results, reduce processing time, and make the data size smaller without reducing the information contained therein so that the data can be processed and used for building systems [13]. An example of preprocessed text can be seen in Table 2. The preprocessing stages used in this study are followed.
a. Case Folding
Case folding is a process of making all the letters in a sentence lowercase that often occurs in the beginning of a sentence, city, name, and others.
b. Data Cleaning
Data cleaning is a step to clean data, including unescape HTML to remove HTML tags in sentences, URL removal to remove all links in sentences, mention removal to remove words with the prefix "@", punctuation removal to remove all punctuation marks in sentences, and number removal to remove all the numbers in the sentence.
c. Stopword Removal
Stopword removal reduces data dimensions and removes words that do not contain sentiment elements, such as personal pronouns, conjunctions, and prepositions, using the Sastrawi library [13], [14].
d. Word Stemming
Word Stemming is a process for converting a word into a basic word. This stage is essential in text-based classification [14].
Table 2. Preprocessed Text
Number Text Preprocessed Text
1 @RadarKorupsi @msaid_didu Bangga untuk sirkuit Mandalika boleh, tapi hutang bayarlah !
bangga sirkuit mandalika tapi hutang bayar
2 RT @Omyung0505 : @Mdy_Asmara1701 Indonesia sudah punya Mandalika Internasional Sirkuit yg sudah terbukti.
retweet indonesia punya mandalika internasional sirkuit sudah bukti 3
Kesombongan soal sukses Formula e mendapat hadiah Ranking 1 Udara Terburuk Dunia... Payah....
https://t.co/shX5u2z1G7
sombong soal sukses formula dapat hadiah ranking udara buruk dunia
payah 2.4 Modelling
The models used in this system are IndoBERTweet and Support Vector Machine. The models will be trained and tested as many as k since we use K-Fold Cross Validation to have the model’s performance for the entire dataset.
K-Fold Cross Validation is used to validate the evaluation results of the built model. This method divides the dataset into k groups and training k models. The number of “k” used in this system is ten based on previous research [15]. An illustration of K-Fold Cross Validation is shown in Figure 2.
Figure 2. K-Fold Cross Validation Illustration a. IndoBERTweet
IndoBERTweet is a transformer encoder-based pre-trained model with twelve hidden layers, twelve attention heads, and three feed-forward hidden layers with the Bidirectional Encoder Representations from Transformers (BERT) architecture [11]. BERT is a multi-layer bidirectional encoder with two stages: pre-training and fine- tuning. The pre-training process in BERT uses a masked-language model and next-sentence prediction [16]. The BERT approach to pre-trained IndoBERTweet is a masked-language model that predicts masked tokens based on the context of the text [16], [17]. Next, a fine-tuning process will be carried out on the new dataset following the objectives of the model to be built. This process is carried out based on the pre-training parameters that have been done before.
b. Support Vector Machine
Support Vector Machine (SVM) is a machine learning system (supervised learning) that uses a hypothesis space in the form of linear functions in high-dimensional features that are trained based on optimization theory [18]. The resulting value will be in the form of a dividing line called a hyperplane which in this study will separate tweets with positive sentiment values from negative sentiments [19]. SVM illustration can be seen in Figure 3.
Figure 3. Support Vector Machine Illustration
SVM divides the dataset into two classes separated by a hyperplane line. In Figure 3, a square-shaped vector is a negative-valued vector, and a circular-shaped vector is a positive-valued vector. The equation for determining whether data is positive or negative is contained in equation (1) and equation (2). The equation for determining the value of the vector weight (w) is found in equation (3), and the equation for finding the bias value (b) is in equation (4). To create an SVM model, it's required to performed feature extraction to improve the model's performance. Feature extraction will utilize much information from the text [20].
Feature extraction used for this research is Term Frequency-Inverse Document Frequency (TF-IDF). TF- IDF is a word weighting technique multiplying the TF value by the IDF value. TF is the process of looking for the number of occurrences of a word in a document. Meanwhile, the IDF will calculate the log in the document for the word searched for [5]. The formula for calculating TF-IDF can be seen in equation (5).
𝑋𝑖. 𝑊 + 𝑏 ≥ 1, 𝑌𝑖 = 1 (1)
𝑋𝑖. 𝑊 + b ≤ 1, 𝑌𝑖= −1 (2)
𝑤 = ∑𝑛𝑖=1𝑎𝑖𝑦𝑖𝑥𝑖 (3)
𝑏 = −1
2(𝑤. 𝑥++ 𝑤. 𝑥−) (4)
TF−𝐼𝐷𝐹 = TF (𝑡, 𝑑) × log 𝑑𝑓𝑁
𝑡 (5)
2.5 Evaluation
Evaluation metrics are used to measure the results of the classification to determine the level of effectiveness of the model that has been built. Evaluation metrics are also used to assess the performance and quality of the models built [21]. The evaluation metrics used in this study followed.
a. Confusion Matrix
The confusion matrix is a method used to evaluate the classification model that has been made to determine the accuracy of the predictions [22]. Testing the accuracy of the confusion matrix has four conditions, as shown in Table 4.
Table 3. Confusion Matrix
Classification Positive Prediction Negative Prediction Actual Positive True Positive (TP) False Negative (FN) Actual Negative False Positive (FP) True Negative (TN) b. Accuracy
Accuracy is the ratio of accurate predictions for positive and negative values with all data. The equation for calculating accuracy can be seen in equation (6).
𝐴𝑐𝑐𝑢𝑟𝑎𝑐𝑦 = (𝑇𝑃+𝑇𝑁)
(𝑇𝑃+𝐹𝑃+𝐹𝑁+𝑇𝑁) (6)
c. Precision
Precision is the ratio of true positive predictions to overall positive results. The equation for calculating precision can be seen in equation (7).
𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 = 𝑇𝑃
𝑇𝑃+𝐹𝑃 (7)
d. Recall
Recall is the ratio of true positive predictions to all data that is positive. The equation for calculating recall can be seen in equation (8).
𝑅𝑒𝑐𝑎𝑙𝑙 = 𝑇𝑃
𝑇𝑃+𝐹𝑁 (8)
e. F1-Score
F1-score is calculated by taking the average value of the precision and recall to calculate model performance.
The equation for calculating f1-score can be seen in equation (9).
𝐹1 − 𝑆𝑐𝑜𝑟𝑒 = 2 ∗ 𝑅𝑒𝑐𝑎𝑙𝑙 ∗ 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛
(𝑅𝑒𝑐𝑎𝑙𝑙 + 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛) (9)
3. RESULT AND DISCUSSION
3.1 Racing Circuit Construction Sentiment
From the data that have been collected and labelled for this study, the number of negative labels is 1005, and the number of positive labels is 1619. The distribution of these labels from the dataset can be seen in Figure 4. Label
“0” means the negative, and “1” means the positive. Therefore, this data will be used for modelling which IndoBERTweet and Support Vector Machine with a percentage of 61.7%: 38.3% for positive and negative labels.
Figure 4. Label Distribution 3.2 IndoBERTweet
Table 4. IndoBERTweet Parameter Parameter Value
batch_size 12
epoch 3
df_eval: df_test 0,8888889 random_state 0
Parameters used in this study to build the IndoBERTweet model described in Table 4, batch_size is twelve, with three epochs. The separation between data validation and test data using test data from K-Fold cross- validation with a ratio of 20%: 80% and random_state set to zero, so the data does not randomize. The model’s performance can be seen in Table 5.
Table 5. IndoBERTweet Performance K Accuracy
(%)
Precision (%) Recall (%) F1-Score (%)
Negative Positive Negative Positive Negative Positive
1 88 81 92 87 88 84 90
2 82 73 90 85 81 79 85
3 84 92 77 77 92 84 84
4 88 74 91 68 93 71 92
5 86 53 91 50 92 52 92
6 94 41 99 78 95 54 97
7 92 85 96 93 92 89 94
8 85 88 82 80 90 84 86
9 85 85 85 89 90 87 82
10 74 77 68 82 60 79 64
From the result of the experiments that have been done in Table 5, this model constantly gives a good performance of each iteration. On average, the model produces above 85% of accuracy score. However, a good accuracy score does not always follow by a good prediction for each class. It can be seen from the sixth iteration that the model produces the highest accuracy yet with poor performance in predicting negative labels. Therefore, the model's overall performance can be calculated by summing all the prediction results from the classification report for each iteration. Based on the calculations, the model accuracy value for the entire dataset is 86%, the precision value is 88.2%, the recall value is 88.6%, and the f1-score value is 88.4%.
As mentioned above, this model produces the best accuracy from the sixth iteration with a value of 94%, followed by a low score in the value of precision, recall, and f1-score for the negative labels. After analyzing the training data and testing data at the sixth iteration, it was found that there is a gap in the number of labels in the test data between the negative label and the positive label with a ratio of 1:11, so the model is difficult to predict negative texts. The confusion matrix for this model when k equals six can be seen in Figure 5. The model predicts the correct answer with 191 out of 193 for the positive label and 7 out of 17 for the negative label.
Figure 5. IndoBERTweet Confusion Matrix k = 6
Besides k equals six, k equals seven also shows good model performance with values for each attribute, including accuracy, precision, recall, and an f1-score above 85% for each label. After analyzing the train data and the test data, the number of labels in each data is sufficiently balanced so that it can produce a more consistent model. The difference between the number of labels in the test data when k equals six and k equals seven can be seen in Figure 6.
Figure 6. Data Test Labels k = 6 and k = 7 3.3 Support Vector Machine
Table 6. Support Vector Machine Parameter Parameter Value feature extraction TF - IDF
kernel linear
decision_function_shape ovr
Parameters for the Support Vector Machine model used in this study are described in Table 6. This study used Term Frequency-Inverse Document Frequency (TF-IDF) for feature extraction. The kernel for the model is linear, with decision_function_shape being “ovr” or one-vs-rest since this study classified as binary class classification.
Table 7. Support Vector Machine Performance K Accuracy
(%)
Precision (%) Recall (%) F1-Score (%)
Negative Positive Negative Positive Negative Positive
1 87 88 86 77 93 82 89
2 77 83 74 61 90 70 81
3 81 78 84 81 81 80 82
4 85 62 93 72 89 67 91
5 84 43 89 30 93 35 91
K Accuracy (%)
Precision (%) Recall (%) F1-Score (%)
Negative Positive Negative Positive Negative Positive
6 93 60 94 35 98 44 96
7 86 84 87 76 92 80 90
8 75 68 83 83 86 74 75
9 80 84 75 82 78 83 77
10 72 83 57 72 71 77 63
From the result of the experiments that have been done in Table 7, the overall model’s performance for the entire dataset was obtained by summing prediction results for each iteration. Based on the calculation conducted, the model has an accuracy value of 82%, a precision value of 87.3%, a recall value of 84.3%, and an f1-score of 85.8%.
In addition, the model produces the best accuracy with the value of 93% when k equals six. However, the value of each attribute for negative labels is relatively low, and it has been analyzed in IndoBERTweet’s result and discussion. The confusion matrix for this model when k equals six can be seen in Figure7. Negative labels on the data are few compared to positive labels. Of the 17 negative label data, the model only succeeded in predicting 6 of them. And for the positive label, the model only mispredicts 4 data out of 193.
Figure 7. SVM Confusion Matrix k = 6
The most consistent model for Support Vector Machine produces when k equals one, with values of each attribute above 82%, except the recall value for the negative label with 77%. The data distribution for this iteration is balanced compared to the sixth iteration, which can be seen in Figure 8. The difference in the number of data labels at the sixth iteration is very significant compared to the first iteration, with a ratio of 17:193 and 83:128.
This indicates that the number of labels in the test data determines the precision, recall, and f1-score values.
Figure 8. Data Test Labels k = 6 and k = 1
4. CONCLUSION
Based on the results and discussion of an experiment conducted with 2624 collected and labelled data about the construction of racing circuits in Indonesia, the ratio between positive and negative sentiments is 61.7%: 38.3%.
The dataset was also used to build the IndoBERTweet model and Support Vector machine model with K-Fold Cross Validation. The best model from accuracy produces in the sixth iteration with the IndoBERTweet accuracy value of 94% and the Support Vector Machine model's accuracy value of 93%. For each given iteration of k, the IndoBERTweet model shows better performance than the Support Vector Machine model in terms of accuracy, precision, recall, and f1-score. The difference in the overall evaluation value for the entire dataset, the accuracy is 4%, the precision is 1%, the recall is 4%, and the f1-score is 3%, with IndoBERTweet having a higher value for all attributes compared to Support Vector Machine model. With these values, it can be concluded that IndoBERTweet is a better model than SVM, especially in sentiment analysis research. More data in datasets with more labels could be used for further research to give more evidence for this conclusion, especially with the model's comparison. Furthermore, use another parameter for each model to find a fit value to improve the model's performance.
REFERENCES
[1] R. Prabowo and A. Sutanto, “Analisis Pengaruh Struktur Modal, dan Likuiditas terhadap Profitabilitas pada Perusahaan Sektor Otomotif di indonesia,” Jurnal Samudra Ekonomi dan Bisnis, vol. 10, no. 1, pp. 1–11, 2019, doi:
10.33059/jseb.v10i1.1120.
[2] L. D. Dávid, B. Remenyik, C. Molnár, R. Baiburiev, and K. Csobán, “The impact of the Hungaroring Grand Prix on the Hungarian tourism industry,” Event Management, vol. 22, no. 4, pp. 671–674, 2018, doi:
10.3727/152599518X15300559276985.
[3] I. F. Ulfah and B. Maulana, “Analisis Dampak Pembangunan Sirkuit Motogp Terhadap Umkm Di Sekitar Pantai Kuta Mandalika,” Journal of Sharia Economic and Islamic Tourism, vol. 1, no. 3, pp. 1–8, 2021, doi:
https://doi.org/10.31764/jseit.v1i1.
[4] R. Tineges, A. Triayudi, and I. D. Sholihati, “Analisis Sentimen Terhadap Layanan Indihome Berdasarkan Twitter Dengan Metode Klasifikasi Support Vector Machine (SVM),” Jurnal Media Informatika Budidarma, vol. 4, no. 3, p. 650, 2020, doi: 10.30865/mib.v4i3.2181.
[5] Z. Alhaq, A. Mustopa, S. Mulyatun, and J. D. Santoso, “Penerapan Metode Support Vector Machine Untuk Analisis Sentimen Pengguna Twitter,” Journal of Information System Management (JOISM), vol. 3, no. 2, pp. 44–49, 2021, doi:
10.24076/joism.2021v3i2.558.
[6] B. Gunawan, H. S. Pratiwi, and E. E. Pratama, “Sistem Analisis Sentimen pada Ulasan Produk Menggunakan Metode Naive Bayes,” Jurnal Edukasi dan Penelitian Informatika (JEPIN), vol. 4, no. 2, p. 113, 2018, doi:
10.26418/jp.v4i2.27526.
[7] A. Deviyanto and M. D. R. Wahyudi, “Penerapan Analisis Sentimen Pada Pengguna Twitter Menggunakan Metode K- Nearest Neighbor,” JISKA (Jurnal Informatika Sunan Kalijaga), vol. 3, no. 1, p. 1, 2018, doi: 10.14421/jiska.2018.31- 01.
[8] Y. Chandra and A. Jana, “Sentiment Analysis using Machine Learning and Deep Learning,” 7th International Conference on Computing For Sustainable Global Development (INDIACom), 2020, doi:
10.23919/INDIACom49435.2020.9083703.
[9] H. Vasudevan, A. Michalas, S. Narendra, and M. Narvekar, “Advance Computing Technologies and Applications,”
Singapore, 2020. doi: 10.1007/978-981-15-3242-9_35.
[10] K. S. Nugroho, A. Y. Sukmadewa, H. Wuswilahaken Dw, F. A. Bachtiar, and N. Yudistira, “BERT Fine-Tuning for Sentiment Analysis on Indonesian Mobile Apps Reviews,” ACM International Conference Proceeding Series, pp. 258–
264, 2021, doi: 10.1145/3479645.3479679.
[11] F. Koto, J. H. Lau, and T. Baldwin, “IndoBERTweet: A Pretrained Language Model for Indonesian Twitter with Effective Domain-Specific Vocabulary Initialization,” pp. 10660–10668, 2021, doi: 10.18653/v1/2021.emnlp-main.833.
[12] M. Yasen and S. Tedmori, “Movies Reviews Sentiment Analysis and Classification,” IEEE Jordan International Joint Conference on Electrical Engineering and Information Technology (JEEIT), 2019, doi: 10.1109/JEEIT.2019.8717422.
[13] A. Asroni, H. Fitri, and E. Prasetyo, “Penerapan Metode Clustering dengan Algoritma K-Means pada Pengelompokkan Data Calon Mahasiswa Baru di Universitas Muhammadiyah Yogyakarta (Studi Kasus: Fakultas Kedokteran dan Ilmu Kesehatan, dan Fakultas Ilmu Sosial dan Ilmu Politik),” Semesta Teknika, vol. 21, no. 1, 2018, doi: 10.18196/st.211211.
[14] M. A. Rosid, A. S. Fitrani, I. R. I. Astutik, N. I. Mulloh, and H. A. Gozali, “Improving Text Preprocessing for Student Complaint Document Classification Using Sastrawi,” IOP Conf Ser Mater Sci Eng, vol. 874, no. 1, 2020, doi:
10.1088/1757-899X/874/1/012017.
[15] A. W. Pradana and M. Hayaty, “The Effect of Stemming and Removal of Stopwords on the Accuracy of Sentiment Analysis on Indonesian-language Texts,” Kinetik: Game Technology, Information System, Computer Network, Computing, Electronics, and Control, pp. 375–380, Oct. 2019, doi: 10.22219/kinetik.v4i4.912.
[16] G. L. Chao and I. Lane, “BERT-DST: Scalable end-to-end dialogue state tracking with bidirectional encoder representations from transformer,” Proceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH, vol. 2019-Septe, pp. 1468–1472, 2019, doi: 10.21437/Interspeech.2019-1355.
[17] X. Wu, T. Zhang, L. Zang, J. Han, and S. Hu, “Mask and infill: Applying masked language model to sentiment transfer,”
IJCAI International Joint Conference on Artificial Intelligence, vol. 2019-Augus, pp. 5271–5277, 2019, doi:
10.24963/ijcai.2019/732.
[18] A. M. Puspitasari, D. E. Ratnawati, and A. W. Widodo, “Klasifikasi Penyakit Gigi Dan Mulut Menggunakan Metode Support Vector Machine,” J-Ptiik, vol. 2, no. 2, pp. 1–9, 2018.
[19] A. M. Pravina, I. Cholissodin, and P. P. Adikara, “Analisis Sentimen Tentang Opini Maskapai Penerbangan pada Dokumen Twitter Menggunakan Algoritme Support Vector Machine (SVM),” Jurnal Pengembangan Teknologi Informasi dan Ilmu Komputer, vol. 3, no. 3, pp. 2789–2797, 2019.
[20] W. K. Mutlag, S. K. Ali, Z. M. Aydam, and B. H. Taher, “Feature Extraction Methods: A Review,” in Journal of Physics:
Conference Series, Aug. 2020, vol. 1591, no. 1. doi: 10.1088/1742-6596/1591/1/012028.
[21] Z. Munawar, N. I. Putri, and D. Z. Musadad, “Meningkatkan Rekomendasi Menggunakan Algoritma Perbedaan Topik,”
Jurnal Sistem Informasi, vol. 01, no. 02, pp. 17–26, 2020.
[22] I. Nawangsih, I. Melani, S. Fauziah, and A. I. Artikel, “Pelita Teknologi Prediksi Pengangkatan Karyawan dengan Metode Algoritma C5.0 (Studi Kasus PT. Mataram Cakra Buana Agung),” Jurnal Pelita Teknologi, vol. 16, no. 2, pp.
24–33, 2021, doi: https://doi.org/10.37366/pelitatekno.v16i2.672.
