Hate Speech Hashtag Classification on Twitter Using the Hybrid Classifier Method

(1)

JURIKOM (Jurnal Riset Komputer), Vol. 9 No. 4, Agustus 2022 e-ISSN 2715-7393 (Media Online), p-ISSN 2407-389X (Media Cetak) DOI 10.30865/jurikom.v9i4.4548

Hal 828−833 http://ejurnal.stmik-budidarma.ac.id/index.php/jurikom

Hate Speech Hashtag Classification on Twitter Using the Hybrid Classifier Method

Aulia Rayhan Syaifullah¹, Yuliant Sibaroni^2,*

Faculty Of Informatics, Informatics, Telkom University, Bandung, Indonesia Email: ¹[email protected], ^2,*[email protected]

Email Penulis Korespondensi: [email protected] Submitted 25-07-2022; Accepted 16-08-2022; Published 30-08-2022

Abstract

Hate speech on social media, especially Twitter, often takes the form of racism, sexism, or political interests aimed at certain individuals or groups. These actions can trigger crime, riots, violence and even resistance to individuals or groups. Therefore, we need a process of classifying a tweet whether it is hate speech or not to reduce the abuse that occurs on Twitter. The technology used in the classification of hate speech that is most commonly used is neural networks that require user data and meta data. In previous studies, the Naïve Bayes (NB) method has been used using the bigram, unigram and feature selection features with an accuracy of 80-85%.

The k-Nearest Neighbor (kNN) method has also been used which has an accuracy of 70-85% on the clarification of hate speech by political figures. Meanwhile, the most widely used method is the Support Vector Machine (SVM) method with an accuracy of 70 to the highest 95%. To get a higher accuracy in the classification of hate speech, this study will perform a Hybrid Classifier on the Hate Speech Hashtag Classification process using a combination method of MLP, kNN, NB. The data used in this study are Twitter Tweets from November 2021 to June 2022 regarding trending hashtags. The average accuracy performance results obtained using MLP, kNN, NB were 72%, 63%,73% respectively. To improve the accuracy of the classification results of the three methods, a combination of methods using the Hybrid Classifier is carried out. Experimental results show Hybrid Classifier with voting method can increase accuracy up to 74%. It was found that the use of a hybrid can provide a better system performance than the 3 classifiers in its composition, namely kNN, NB and MLP.

Keywords: Twitter; Hate Speech; Hybrid Classifier; Classification; Social Media

1. INTRODUCTION

Based on Katadata [1], Indonesia is one of the countries with the largest population of internet users in the world, according to reports, there are 204.7 million internet users in the country as of January 2022. This number slightly increased 1.03% from the previous year, In early 2022, Indonesia's internet penetration rate reached 73,7 percent of the total population. According to the data, nearly all internet users use social media. [2]. According to the National Indonesian Police Criminal Investigation Agency, there were 143 cyber crimes in the form of hate speech in Indonesia in 2015. In 2016, this figure increased to 199. This data, however, only includes hate speech that has been criminalized and reported to the authorities. Of course, there are numerous other hate speeches on social media. [3]. Twitter is a popular social media platform in Indonesia. Twitter and other social media and web microblogging services enable users to read and analyze their tweets in real time. This analysis can aid in the early detection of hate speech, allowing it to be contained before it spreads widely.

Research on the classification of hate speech on Twitter using the Naïve Bayes method has been widely carried out and has a high accuracy of 80-85%.[4][5], this method can be used using many features, namely the unigram feature and feature selection, these features managed to get better results than other Naïve Bayes features, such as the bi gram feature.

The k-Nearest Neighbor (kNN) method is also widely used in the analysis of hate speech on Twitter, for example personality classification based on text. [6] and identification of hate speech against political figures [7] get a fairly high accuracy in the range of 70-85%.

The Support Vector Machine (SVM) method is the most widely used method in classifying hate speech on Twitter because of its high effectiveness and accuracy, SVM is proven to be able to produce an independent system that can classify tweets containing hate speech. [8][9] including hate speech in English and Spanish [10], SVM accuracy ranges from 70% to as high as 95%.

Multilayer Perceptron (MLP) method is a feedforward artificial neural network with one or more hidden layers.

In MLP, there are two important parameters, namely the function activation and optimization functions [11]. Studies shows that MLP had been used for hate speech detection [12] and cancer breast classification with 97,7% accuracy using logistic activation and Adam optimization function [13].

In previous studies, the classification of hate speech was only done by one separate method. Method innovation is needed to get a higher accuracy of hate speech classification. This is the rationale in this study to implement the Hybrid Classifier method in Hate Speech Hashtag Classification, where the MLP classification method will be combined with the kNN and NB methods, with these three methods combined using Voting Classifier, it will produces higher accuracy than using just a single method to classify Hate Speech on Twitter.

2. RESEARCH METHODOLOGY

(2)

Hal 828−833 http://ejurnal.stmik-budidarma.ac.id/index.php/jurikom 2.1 Research Flowchart

In this study, a flow chart is used to describe the modeling system used for Hate Speech Hashtag Classification Using the Hybrid Classifier Method. The following is a flow chart of the hate speech detection system, which can be seen in Figure 1.

Figure 1. Hate Speech Detection System 2.2 System Development Method

The development of the Hate Speech Detection System consists of several activities which are of course in accordance with the stages that have been carried out in the system development process flow with details:

a. Data Crawling

In conducting a research, of course we need a data that needs to be processed to get a conclusion from the research.

Data crawling activity itself is an activity where we collect data from a web or database. In the Twitter world itself, data crawling has a meaning as the process of retrieving Tweet data based on hashtag keywords using the API from Twitter Developer. The keywords used are #JokowiKejarSetoran, #BubarkanKhilafatulMuslimin. Data crawling is carried out in the period November 2021 to June 2022.

b. Data Labeling

Data labeling is the process of distinguishing data into 2 groups, namely hate speech data and non-hate speech data.

prejudice either on the part of the perpetrator of the statement or the victim of the act [14]. Labeling the data to be used as a dataset manually by reading whether the data includes hate speech or not and labeling is done with 2 people simultaneously so that if there is a difference of opinion then the data is discussed again in order to get the right type of label. Data labeling aims to determine the type of tweet on each tweet that has been crawled before. There are 2 different labels. The labels used are: HS and Non_HS.

c. Data Preprocessing

The Pre-Processing step is a data cleaning step, where the key attribute of a Tweet is filtered to reduce the number of features so that it can match what is required by the algorithm. This consists of clearing text (for example, removing stop words and punctuation marks), as well as removing spam content. There are several stages carried out in data preprocessing, namely:

1. Case Folding

Case folding is the process of converting words or phrases in tweet text to lowercase letters (a to z) using regular expressions. The goal is to help overcome the problem when words are written with different capitals [15].

(3)

Hal 828−833 http://ejurnal.stmik-budidarma.ac.id/index.php/jurikom 2. Data Cleaning

In Data Cleaning, urls, hashtags, numbers, punctuation marks, mentions, square brackets and characters are deleted using regular expressions.

3. Stemming

Stemming is the process of converting a word or token into a basic word, or the process of locating flat words by removing prefixes, infixes, suffixes, and confixes (combinations of prefixes and suffixes) from derived words[16].

d. Feature Extractions

The Feature Extractions method used in this research is the count vectorizer method, the count vectorizer is a method that converts the given sentence into a vector matrix of the number of words that appear in the given sentence. The extractions feature produces features that are extracted in the form of tokens from the text of the data they have which will later be used as input values in the form of sentences and the result is a matrix of words with the size of the matrix (amount of data x number of token data). There are several steps carried out in Feature Extractions:

1. Stopwords Removal

In Stopwords Removal, non-topic words that are not used are removed, for example: “ini”, “atau”, “itu”, “adalah”,

“dan”, “yang”, and others. The goal is to minimize irrelevant features in the data.

2. Count Vectorizer

Serves to count the number of each word that appears in the tweet then made into an array e. K-fold Cross Validation

K-fold cross validation is one of the techniques used to sort data into train data and test data. This technique is widely applied by researchers because it is found to reduce the bias obtained in taking a sample. K-fold cross validation applies continuously dividing data into train data and test data, so that every data will have the opportunity to be a test [17] K is the number of data disaggregation used for the distribution of train and test. In this study, 5 cross validation was used.

f. Model Hybrid Learning

At this stage doing learning using the Hybrid Learning Method, Hybrid is a combination of 2 or more in a function [18]. Hybrid learning works by combining more than 1 model, in this study combining MLP, KNN, NB to be trained using the Vote Classifier. The Vote Classifier determines the class based on the majority vote of the three classifier models. The hybrid process can be seen in the example in Figure 2 below.

Figure 2. Voting Classifier Model g. Performance Evaluation

Performance Metric For Machine Learning in Classification Problems with two or more output classes called Confusion Matrix. The Confusion Matrix is a table with four different combinations of predicted and actual values.

This performance evaluation uses the F1 Score and accuracy, the F1 Score is a comparison of the average recall and precision, while Accuracy describes how accurate the model is in classifying correctly, an example of a confusion matrix is in table 1 below:

Table 1. Confusion Matrix

From Table 1, 4 output values can be obtained, namely:

1. Recall

Calculations used to find out how big the percentage of true cases identified as true.

Recall =

^TP

TP+FN (1)

Predicted Class Actual Class

Negative Positive

Negative True Negative (TN) False Positive (FP) Positive False Negative (FN) True Positive (TP)

(4)

Hal 828−833 http://ejurnal.stmik-budidarma.ac.id/index.php/jurikom 2. Accuracy

Comparison of all incorrectly classified prediction results with the rest of the data.

Accuracy =

^TP+TN

TP+FN+TN+FN (2)

3. Precision

The comparison between the correct positive classification results is classified as positive with all positive predictions.

Precision =

^TP

TP+FP (3)

4. F1 Score

Weighted comparison of Precision and Recall averages.

F1 Score =

^2∗TP

2∗TP+FP+FN (4)

3. RESULT AND DISCUSSION

3.1 Data Crawling

The study using the Hybrid Classifier method with the first are generated from a crawling process that retrieves tweets with keywords based on topics according to trending topics. Data crawling is carried out in the period November 2021 to June 2022. According to the results of the crawling, there are 3119 tweets that used for training and testing. The hashtags are #JokowiKejarSetoran and #BubarkanKhilafatulMuslimin.

3.2 Data Labeling

After collecting hate speech data using the API on Twitter, then the data labeled to be used as a dataset manually by reading whether the data includes hate speech or not. Data labeling aims to determine the type of tweet in each tweet that has been crawled before. The labels used are: HS and Non_HS. Result of data labels is shown on Table 2.

Table 2. Dataset Label

Label Tweet

HS #JokowiKejarSetoran @jokowi Mundur lah!!bikin susah rakyat aja bisa ny!!

Masarakat jangan mau di bodohi dengan orang orang yang mau mendirikan Negara khilafah. Organisasi khilafatul muslimin adalah iblis bersorban. Tolak khilafah. #BubarkanKhilafatulMuslimin https://t.co/1UqqHtWTQs

Non_HS Ggaasssss pooolll

#JokowiKejarSetoran

#BubarkanKhilafatulMuslimin , usut siapa saja yg terlibat, sikaat sampe tuntas!! NKRI HARGA MATI!! https://t.co/jE9r6odKBr

3.3 Preprocessing Data

Preprocessing is carried out on tweet datasets such as case folding, eliminating numbers, removing punctuation marks, removing blank spaces, removing urls, removing unnecessary text and encoding labels. The result of preprocessing data can be seen on Table 3.

Table 3. Data Preprocessing Preprocessing

Step

Before Preprocessing After

Preprocessing

Item Changed Case Folding &

Data Cleaning

Katanya mau brantas KKN BOHONG....!!!!???

#JokowiKejarSetoran #JokowiKejarSetoran https://t.co/sVGGrtP7yo

katanya mau brantas kkn bohong

“[punctuation]”,

“#[name hashtag]”,

“[link URL]”

Stemming katanya mau brantas kkn bohong kata mau brantas

kkn bohong

“Katanya”

3.4 Feature Extractions

Feature extraction is also done using the count vectorizer feature, then each feature shows how many words are in the tweet. Stopwords removal are involved in count vectorizer for remove non-topics word such as “ini”, “atau”, “adalah”,

“dan”, “yang” and others.Stopwords Removal using NLTK library, the goal is to minimize irrelevant features in the data.

Example of count vectorizer is shown on Figure 3 and obtained 4392 unique words.

(5)

Figure 3. Count Vectorizer Example 3.5 K-fold Cross Validation and Hybrid Process

This system uses 5 – Fold Cross Validation that applies constantly to split the data into train data and test data. Data sharing utilizing the K-Fold cross validation method is demonstrated in Table 4. The MLP classifier, K-Nearest Neighbor (KNN), Naive Bayes (NB), and Hybrid MLP & KNN & NB algorithms were all used to test this system. The statistics in Table 5 are all expressed as percentages. Table 5 displays the test outcomes.

Table 4. 5-fold Cross Validation Fold 1 Fold 2 Fold 3 Fold 4 Fold 5

Test Train Train Train Train Train Test Train … Train Train Train Test Train Train

… … … Test …

Train Test Train Train Test Table 5. Test Results

Method Fold 1 Fold 2 Fold 3 Fold 4 Fold 5 Average

A F1 A F1 A F1 A F1 A F1 A F1

MLP 70 70 74 74 74 74 71 71 71 71 72 72

KNN 64 61 62 60 62 60 62 61 62 60 63 60

NB 70 70 72 72 74 74 73 72 74 74 73 72

Hybrid 71 71 73 73 77 77 75 75 74 73 74 74 Description:

A: Accuracy F1: F1 Score

The results obtained for NB with 5 Cross Validation get the highest accuracy of 74%, the lowest accuracy of 70%, and an average accuracy of 73%, and for KNN with 5 Cross Validation the highest accuracy is 64%, the lowest is 60%, the average accuracy is 63 %, and for MLP the highest accuracy is 74%, the lowest is 70% with an average of 72% and for Hybrid MLP + KNN + NB with 5 Cross Validation the highest accuracy is 77%, the lowest is 71%, the average accuracy is 74%.

4. CONCLUSION

This research has given quite good results in classifying or predicting the types of tweets on hate speech hashtags. To clean the noise in the data, the application of case folding, eliminating numbers, removing punctuation marks, eliminating empty spaces, eliminating urls, eliminating unimportant words, and encoding labels were carried out. Feature extraction is performed to remove features that are not relevant for the testing process. From the results of tests carried out on the MLP classifier, KNN classifier, NB Classifier and Hybrid MLP & KNN & NB methods. The results showed that the Hybrid MLP & KNN & NB method had the best performance. The system shows that the level of accuracy and f1-score system for predicting the type of tweet on the highest hate speech hashtag in the 3rd fold is 77%, 77% with accuracy and f1 scores on average 74% and 74%(shown on table 5).

(6)

REFERENCES

[1] C. M. Annur, “Ada 204,7 Juta Pengguna Internet di Indonesia Awal 2022 [online],” 2022.

https://databoks.katadata.co.id/datapublish/2022/03/23/ada-2047-juta-pengguna-internet-di-indonesia-awal-2022 (accessed Mar. 23, 2022).

[2] T. Febriana and A. Budiarto, “Twitter Dataset for Hate Speech and Cyberbullying Detection in Indonesian Language,” in 2019 International Conference on Information Management and Technology (ICIMTech), Aug. 2019, vol. 1, pp. 379–382. doi:

10.1109/ICIMTech.2019.8843722.

[3] M. A. Fauzi and A. Yuniarti, “Ensemble method for indonesian twitter hate speech detection,” Indonesian Journal of Electrical Engineering and Computer Science, vol. 11, no. 1, pp. 294–299, 2018, doi: 10.11591/ijeecs.v11.i1.pp294-299.

[4] M. Hakiem, M. A. Fauzi, and I. Indriati, “Klasifikasi ujaran kebencian pada twitter menggunakan metode naïve bayes berbasis N-gram dengan seleksi fitur information gain,” vol, vol. 3, no. 3, pp. 2443–2451, 2019.

[5] H. Sahi, Y. Kilic, and R. B. Saglam, “Automated Detection of Hate Speech towards Woman on Twitter,” in 2018 3rd International Conference on Computer Science and Engineering (UBMK), Sep. 2018, pp. 533–536. doi:

10.1109/UBMK.2018.8566304.

[6] G. K. Pitsilis, H. Ramampiaro, and H. Langseth, “Effective hate-speech detection in Twitter data using recurrent neural networks,” Applied Intelligence, vol. 48, no. 12, pp. 4730–4742, Dec. 2018, doi: 10.1007/s10489-018-1242-y.

[7] K. K. Kiilu, G. Okeyo, R. Rimiru, and K. Ogada, “Using Naïve Bayes Algorithm in detection of Hate Tweets,” International Journal of Scientific and Research Publications (IJSRP), vol. 8, no. 3, pp. 99–107, Mar. 2018, doi:

10.29322/IJSRP.8.3.2018.p7517.

[8] S. Saha, J. Yadav, and P. Ranjan, “Proposed Approach for Sarcasm Detection in Twitter,” Indian Journal of Science and Technology, vol. 10, no. 25, pp. 1–8, Jun. 2017, doi: 10.17485/ijst/2017/v10i25/114443.

[9] Oryza Habibie Rahman, Gunawan Abdillah, and Agus Komarudin, “Klasifikasi Ujaran Kebencian pada Media Sosial Twitter Menggunakan Support Vector Machine,” Jurnal RESTI (Rekayasa Sistem dan Teknologi Informasi), vol. 5, no. 1, pp. 17–23, Feb. 2021, doi: 10.29207/resti.v5i1.2700.

[10] I. Vogel and M. Meghana, “Profiling Hate Speech Spreaders on Twitter: SVM vs. Bi-LSTM.,” in CLEF (Working Notes), 2021, pp. 2193–2200.

[11] J. C. Pereira-Kohatsu, L. Quijano-Sánchez, F. Liberatore, and M. Camacho-Collados, “Detecting and Monitoring Hate Speech in Twitter,” Sensors (Basel), vol. 19, no. 21, p. 4654, Oct. 2019, doi: 10.3390/s19214654.

[12] A. Silva and N. Roman, Hate Speech Detection in Portuguese with Naïve Bayes, SVM,MLP and Logistic Regression. 2020. doi:

10.5753/eniac.2020.12112.

[13] J. KUSUMA, B. H. HAYADI, W. WANAYUMINI, and R. ROSNELLY, “Komparasi Metode Multi Layer Perceptron (MLP) dan Support Vector Machine (SVM) untuk Klasifikasi Kanker Payudara,” MIND Journal, vol. 7, no. 1, pp. 51–60, Jun. 2022, doi: 10.26760/mindjournal.v7i1.51-60.

[14] I. Kamalludin and B. N. Arief, “KEBIJAKAN FORMULASI HUKUM PIDANA TENTANG PENANGGULANGAN TINDAK PIDANA PENYEBARAN UJARAN KEBENCIAN (HATE SPEECH) DI DUNIA MAYA,” LAW REFORM, vol. 15, no. 1, p.

113, May 2019, doi: 10.14710/lr.v15i1.23358.

[15] Robi Kurniawan and Aulia Apriliani, “Analisis Sentimen Masyarakat terhadap Virus Corona berdasarkan Opini dari Twitter berbasis Web Scraper,” Jurnal Instek, vol. 5, no. 1, 2020.

[16] S. Gharatkar, A. Ingle, T. Naik, and A. Save, “Review preprocessing using data cleaning and stemming technique,” in 2017 International Conference on Innovations in Information, Embedded and Communication Systems (ICIIECS), Mar. 2017, pp. 1–

4. doi: 10.1109/ICIIECS.2017.8276011.

[17] T. Ridwansyah, “Implementasi Text Mining Terhadap Analisis Sentimen Masyarakat Dunia Di Twitter Terhadap Kota Medan Menggunakan K-Fold Cross Validation Dan Naïve Bayes Classifier,” KLIK: Kajian Ilmiah Informatika dan Komputer, vol. 2, no. 5, pp. 178–185, Apr. 2022, doi: 10.30865/klik.v2i5.362.

[18] Fatri Nurul Inayah, Sri Suryani Prasetiyowati, and Yuliant Sibaroni, “Classification of Dengue Hemorrhagic Fever (DHF) Spread in Bandung using Hybrid Naïve Bayes, K-Nearest Neighbor, and Artificial Neural Network Methods,” International Journal on Information and Communication Technology (IJoICT), vol. 7, no. 1, pp. 10–20, Jun. 2021, doi: 10.21108/ijoict.v7i1.562.