Aspect-Based Sentiment Analysis on iPhone Users on Twitter Using the SVM Method and Optimization of Hyperparameter Tuning

Aspect-Based Sentiment Analysis on iPhone Users on Twitter Using the SVM Method and Optimization of Hyperparameter Tuning

I GSA Putu Sintha Deviya Yuliani^*, Yuliant Sibaroni, Erwin Budi Setiawan School of Computing, Informatics Study Program, Telkom University, Bandung, Indonesia

Email: ^1,*deviyayuliani@student.telkomuniversity.ac.id, ²yuliant@telkomuniversity.ac.id,

3erwinbudisetiawan@telkomuniversity.ac.id

Correspondence Author Email: deviyayuliani@student.telkomuniversity.ac.id

Abstract−One form of information and communication technology development is a smartphone. Today's popular smartphone products are the iPhone and the social media used to share opinions is Twitter. One of the topics that is often discussed on Twitter is related to iPhone reviews which can refer to different aspects. Therefore, aspect-based sentiment analysis can be applied to iPhone reviews to get more detailed results. This study applies TF-IDF feature extraction as a weighting vocabulary and the Support Vector Machine classification method. This study also uses hyperparameter tuning to optimize parameters to get the best performance. The results of this study obtained the highest accuracy performance results by using the Support Vector Machine classification on the linear kernel and TF-IDF feature extraction on the camera aspect with accuracy 98.07%, battery aspect with accuracy 97.52%, design aspect with accuracy 96.82%, price aspect with accuracy 98.62%, and specification aspect with accuracy 97.07%. As well as getting an increase in the results of the highest accuracy performance by using hyperparameter tuning on the linear kernel for the camera aspect with accuracy 98.07%, battery aspect with accuracy 97.52%, design aspect with accuracy 97.02%, price aspect with accuracy 98.82%, and specification aspect with accuracy 97.22%.

Keywords: Aspect Based Sentiment Analysis; iPhone; Support Vector Machine; Hyperparameter Tuning.

1. INTRODUCTION

Information and communication technology (ICT) has advanced significantly during the past several years. This is the context for the transition from traditional to modern communication technology, such the internet. The availability of the internet as a contemporary communication tool has made it simpler to understand the world.

Almost everyone has a phone or other communication device that they may use to use social media to communicate with people all around the world. Social media is a group of programs that people use to interact, share, communicate, work together, and have fun[1].

Twitter is one type of social media that is currently quite popular among internet users as a medium for communication. Twitter is a medium that is frequently used for writing evaluations because thoughts can be expressed there without restriction[2]. Twitter users can communicate, share information, and post news about a variety of topics, including reviews of iPhone products. One of the most well-known smartphones, the iPhone, is frequently discussed. However, there are opinions from users regarding iPhone products, namely opinions in terms of positive, neutral and negative. Aspects that are often discussed by iPhone users on Twitter are aspects of the camera, battery, design, price, and specifications. One way that can be used to analyze and process the review text is sentiment analysis.

The goal of sentiment analysis is to identify the positive, neutral, and negative content of a text dataset[3].

Aspect-based sentiment analysis is one method of sentiment analysis[4]. The analytical method of aspect-based sentiment analysis allows for the identification of the sentiments represented in each aspect. One method for analyzing sentiment is the Support Vector Machine, which is a supervised learning method in classification which has the advantage of being applicable to high-dimensional data or being able to handle linear and non-linear regression[5].

Previous research[6] had weaknesses in classification accuracy using Naïve Bayes in the iPhone study case which was not that good, where the accuracy of the sentiment analysis system for iPhone products used the Naive Bayes method which used test data to get a value of 70.88% and there were still some errors when tested by several users due to more varied sentences.

Many studies have already been conducted to find the best performance using the Support Vector Machine (SVM) method for sentiment analysis. Bourequat and Mourad's (2021) research paper examines public opinion on the iPhone product on Twitter using the Support Vector Machine method (SVM). According to the study's findings, the two most effective methods for analyzing public discourse are Natural Language Processing (NLP) and text mining, often known as text analytics. One example of every method used in this study is text mining on a Twitter page about the release of an iPhone using techniques including scraping, labeling, preprocessing (case folding, tokenizing, filtering), TF-IDF, and classification using Support Vector Machine (SVM). The data collection used in this study is made up of roughly 1002 English-language opinions. The data is divided into 801 training data and 201 test data, then evaluated by calculating accuracy, precision, recall, and F1-score value. The results of the research accuracy were 89.21%, precision was 92.43%, recall was 95.43%, and the F1-score was 93.95%[7].

Iskandar and Nataliani published a paper in the year 2021[8] with the working title Comparison of Naïve Bayes, Support Vector Machine, and k-Nearest Neighbor for Aspect-Based Gadget Sentiment Analysis. The

purpose of this study is to understand the inner workings of a particular technology product. The Samsung Galaxy Z Flip 3 is the one that stands out. Dataset being used is from social media site YouTube, with a total of 9,597 comments and more users providing positive opinions regarding design and negative opinions regarding price, specifications, and brand names. By using the CRISP-DM model and comparing the Naive Bayes (NB), Support Vector Machine (SVM), and k-Nearest Neighbor (k-NN) classification methods, it is clear that the SVM classification model yields the best results. The average SVM accuracy is 96.43% seen from four aspects, namely the design aspect of 94.40%, the price aspect of 97.44%, the specification aspect of 96.22%, and the brand image aspect of 97.63%.

In the year 2019, Jagdale, et al[9] conducted research on the analysis of consumer product feedback using machine learning. This study's goal is to classify smartphone usage into positive and negative emotions. More than 4,000 questions were classified as positive or negative sentiments in this study. There are three classification methods that are used: Naive Bayes, Support Vector Machine, and Decision Tree. From these three classifications, the best results are predicted using the Support Vector Machine (SVM) method. Model evaluation is carried out using 10 Fold Cross Validation. The highest performance rating from the study in question using 10 Fold Cross Validation was 81.75% for SVM across the four models.

In the year 2020, Siji George and Sumathi[10] conducted research on Grid Search Tuning from Hyperparameter using Random Forest Classifier to assess client feedback. The grid search approach is applied to the hyperparameters of the Random Forest classifier. The Random Forest Classifier is used to analyze customer input, and its results are superior to those obtained after applying the grid search method. Customer feedback study results using Random Forest without using the grid search method produced good results with an accuracy of 84.93%. However, by applying the grid search method to the Random Forest Classifier, the model's accuracy increased to 90.02%. The results show that parameter adjustment for Random Forest Classifier using the grid search method has been successful in helping to produce the best model for classifying new data.

Research written by Haqmi Abas in 2020[11] discusses how to grade the quality of gin using the Support Vector Machine (SVM) and Grid Search Cross Validation Hyperparameter Tuning. The C, gamma, and kernel parameters are used to calibrate the Support Vector Machine (SVM) via grid search cross validation. Using the combination of C with a value of 1, gamma with a value of 10, and kernel Radial Basis Function (RBF), it is concluded that the best possible accuracy is 100%, and the performance measure score is 1.0.

Due to this, the purpose of this study is to offer solutions to problems from previous studies' analyses as well as to find the best results from an analysis of performance using Support Vector Machine and Hyperparameter Tuning to analyze tweets from iPhone users on Twitter at each level. The use of TF-IDF feature extraction is used as a vocabulary weighting. The use of SVM in this study is due to the fact that the method of classification in question can handle extremely large amounts of data, particularly for classification of texts[12]. According to Rodrigues (2017), the results of the study also indicate that Support Vector Machine, a machine learning classifier, performs better than other classifiers[13]. In this study, hyperparameter tuning is optimized using the Grid Search method to obtain good performance while adjusting the parameters, and Cross Validation is used as a performance indicator to find the best hyperparameter combinations, allowing the classification model to accurately predict data that has not yet been understood[14].

The data that were collected and used in this study's analysis were taken from Twitter user tweets that discussed the specifications of the iPhone's camera, battery, design, price, and specification. Data being exchanged is in Indonesian. For example, the information provided is every tweet about the iPhone from June 2012 until the day of the crawling event, which is October 29, 2022.

2. RESEARCH METHODOLOGY

2.1 System Design

In this study, Support Vector Machine and Hyperparameter Tuning classifications of the sentience analysis process are shown in flowchart Figure 1. Here are a few steps in the system for sentiment analysis, including crawling data, pre-processing data, splitting data, model building, and evaluating model performance.

Figure 1. System Design Flowchart

2.2 Crawling Data

The data was gathered using the snscrape library and the Python computer language after browsing Indonesian- language Twitter comments. There are 23 keywords that are used during the crawling process, including camera iPhone, clear iPhone, good iPhone, cool iPhone, iPhone photo quality, iPhone video quality, iPhone battery, extravagant iPhone, saving iPhone, durable iPhone, broken iPhone, thin iPhone, thick iPhone, iPhone design, iPhone screen, iPhone body, iPhone price, expensive iPhone, cheap iPhone, iPhone ram, iPhone specifications, iPhone processor speed, and iPhone chipset. A total of 5000 pieces of information were gathered, covering 5 different categories: camera, battery, design, price, and specifications. Table 1 describes the specifics of aspect and word identification.

Table 1. Detail Identification of Aspects and Words

Category Aspect Word

Camera camera, clear, good, cool, photo quality, video quality Battery battery, extravagant, saving, durable, broken Design thin, thick, design, screen, body

Price price, expensive, cheap

Specification ram, specification, processor speed, chipset 2.3 Labeling Data

Data that has been crawled is then manually labeled. Sentiment is broken down into the following categories:

camera, battery, design, price, and specifications. There are three different types of sentiment: positive, denoted by the number 1, which refers to the advantages of each aspect; neutral, denoted by the number 0; and negative, denoted by the number -1, which refers to the shortcomings of each aspect. An example of the process for labeling datasets can be seen in Table 2

Table 2. Labeling Data

Tweet Camera Battery Design Price Specification Pake iphone enak Aku

pengen punya spesifikasi yg bagus Tapi duitnya gak sanggup karna mahal, kurang suka juga sama kamera dan desainnya"

Negative Neutral Negative Negative Positive

OKE! Kamera iPhone XR Digandrungi Penyuka Fotografi, desain juga lucu, dan spesifikasi yg bagus, tapi harganya mahal

Positive Neutral Positive Negative Positive

2.4 Preprocessing Data

Data preprocessing comes before classification. Data that is not significant or necessary must be cleaned, changed, or removed during preprocessing in order to make it simpler to process the data that will be used for classification and perhaps enhance data quality. The preprocessing steps are listed below.

2.4.1 Cleaning Data

Cleaning Data is the process of purging data of any elements not required for sentiment analysis, such as the url, user, enter, numbers, hashtags, punctuation, tabs, and symbols of each phrase.

2.4.2 Case Folding

The act of folding all letters into lowercase is known as case folding.

2.4.3 Tokenizing

Tokenizing is the process of dividing text into tokens, which can be single words, phrases, or other meaningful symbols.

2.4.4 Data Normalization

When data is normalized, redundant and inappropriate words are removed from crossword puzzles and replaced with words from the KBBI vocabulary. An example of normalization can be seen in Table 3.

Table 3. Normalization

Before After

[pake, iphone, enak, aku, pengen, punya, kamera, sebagus, iphone, desain, yg, lucu, dan, spesifikasi, yg, bagus, tapi, duitnya, gak, sanggup, karna, mahal]

[pakai, iphone, enak, aku, pengin, punya, kamera, sebagus, iphone, desain, yang, lucu, dan, spesifikasi, yang, bagus, tapi, duitnya, enggak, sanggup, karena, mahal]

2.4.5 Stop Word Removal

Stop Word Removal is the process of removing words that are considered unimportant. These insignificant words are those that lack a clear definition. An example of stop word can be seen in Table 4.

Table 4. Stop Word Stop Word

aku, enak, yang, dan, tapi, karena, dengan, berkata, sebabnya, sebagai, sesudah, setelah, nya, kapan, seterusnya, sedangkan, tambah, adapun, akhirnya, adalah, antara, cukup, apalagi, misalnya, mampu 2.4.6 Stemming

Stemming is the process of turning affix-containing words into fundamental words. An example of stemming can be seen in Table 5.

Table 5. Stemming

Tweet Stemming

[pakai, iphone, enak, aku, pengin, punya, kamera, sebagus, iphone, desain, yang, lucu, dan, spesifikasi, yang, bagus, tapi, duitnya, enggak, sanggup, karena, mahal]

['pakai', 'iphone', 'pengin', 'kamera', 'bagus', 'iphone', 'desain', 'lucu', 'spesifikasi', 'bagus', 'duit', 'sanggup', 'mahal']

2.5 Feature Extraction TF-IDF

TF-IDF is the feature extraction method used. One technique for word weighting is the TF-IDF. The goal of word weighting is to assign each word a weight value. Two factors, Term Frequency (TF) and Inverse Document Frequency, are needed to calculate this weight (IDF). The frequency of particular words or phrases in a document is calculated using the term frequency (TF) formula. While the Inverse Document Frequency (IDF) is employed to give words that appear in documents less frequently and more frequently higher weight. Term Frequency (TF) and Inverse Document Frequency (IDF) are multiplied to get TF-IDF. The frequency of a phrase will increase as a word's usage in the document increases. A word's importance for the keywords used to search the document will increase the less frequently it appears in the document[15]. The TF-IDF formula is as follows[16]:

W_ki= tf_ki∗ log(N/n_k) (1)

Where W_ki is the weight of the word k in the i document, tf_ki is the number of occurrences of the word k in the i document, N is the number of all documents that have been used, and n_k is the number of documents containing the word k.

2.6 Support Vector Machine (SVM) Classification

A Machine Learning Supervised classification is the SVM classification. The fundamental idea of SVM is similar to that of a linear classifier, i.e., classification cases that can be divided linearly. In order to gain generalizations for the classification process with test data, SVM divides the training data into two classes by estimating the hyperplane line and determining the maximum distance from the hyperplane to the nearest training data[17].

However, by combining the kernel notion in high-dimensional workspaces, SVM has been evolved to be able to operate on non-linear problems[18].

Support Vector Machine (SVM) is a concept used in classification to determine the optimal hyperplane line that divides two classes in the input space, which is the maximum distance or margin to the closest pattern class points or support vectors[16]. Table 6 shows the kernel that was applied in this investigation.

Table 6. Parameters Of Support Vector Machine

Kernel Function

Linear x. z

Kernel Function Polynomial tanh⁡(σ(x. y) + c) Radial Basis Function softmax(x_i) = ⁡ exp⁡(x_i)

∑ exp⁡(x^k_{j i}) 2.7 Hyperparameter Tuning

Any machine learning (ML) method can be made to perform better by tweaking its hyperparameters[14].

Hyperparameter tuning is the process of adjusting the parameters to achieve the greatest performance and gauge the model's correctness after it has been applied[19]. In this work, initializing the optimized hyperparameters is the first stage in hyperparameter tuning. The grid search approach is then used to find the optimal values.

Additionally, by testing and validating each combination separately, this study uses cross validation to choose the model combinations and hyperparameters. While Mean Cross Validation measurement metric is used to evaluate the outcomes for the optimal model for hyperparameters[14].

The C, kernel, and gamma hyperparameters in SVM can all be optimized. To maximize SVM and reduce errors for each sample in the train dataset, parameter C is used. The influence of one sample train's data on the dividing line is calculated using the gamma parameter in the meantime[20]. Table 7 lists the hyperparameters that were employed.

Table 7. Hyperparameter Used Hyperparameter Values

C [0.001, 0.1, 1, 10, 100, 1000]

Gamma [‘auto’, ‘scale’]

Kernel [‘linear’, ‘poly’, ‘rbf’]

2.8 Performance Evaluation

The Confusion Matrix is the tool utilized in this work for model validation. One tool for evaluating the effectiveness of a classification model is the confusion matrix. Table 8 illustrates the following Confusion Matrix model.

Table 8. Confusion Matrix

Actual Value

Positive (+) Negative (-) Predicted Value Positive (+) True Positive (TP) False Positive (FP)

Negative (-) False Negative (FN) True Negative (TN)

Accuracy, precision, recall, and f1-score values are produced by system performance utilizing the Confusion Matrix.

a) Accuracy is the overall effectiveness of a classification[21]. Accuracy can be formulated in the equation (2).

Accuracy = ^TP+TN

TP+FN+TN+FP (1)

b) Precision is the class agreement of the data label with the positive label given by the classifier[21]. Precision can be formulated in the equation (3).

Precision = ^TP

TP+FP (2)

c) Recall is the effectiveness of classification to calculate the number of predictions of the same class[21].

Recall can be formulated in the equation (4).

Recall = ^TP⁡

TP+FN (3)

d) F1-score is a metric score that calculates the balance value of precision and recall[21]. F1-score can be formulated in the equation (5).

F1-score = 2⁡X⁡Precision⁡X⁡Recall

Precision+Recall (4)

3. RESULT AND DISCUSSION

The data used in this study are 5000 Indonesian language tweet data with 5 aspects, namely camera, battery, price, design, and specifications as well as the keywords used there are 23, namely camera iPhone, clear iPhone, good iPhone, cool iPhone, iPhone photo quality, iPhone video quality, iPhone battery, iPhone wasteful, save iPhone, iPhone durable, broken iPhone, iPhone thin, iPhone thick, iPhone design, iPhone screen, iPhone body, iPhone

price, iPhone expensive, cheap iPhone, iPhone ram, iPhone specifications, speed iPhone processors, and iPhone chipsets that have been crawled before using the snscrape library. The distribution of data in this study can be seen in Table 9.

Table 9. Data Distribution

Aspect Positive Neutral Negative

Camera 3523 1430 47

Battery 109 1358 3533

Design 3701 1249 50

Price 23 1460 3517

Specification 3644 1313 43

This study examines several methods and scenarios using the Support Vector Machine (SVM) classification and hyperparameter tuning with a grid search. In this study using two scenarios:

1) Testing the SVM classification method with TF-IDF feature extraction and cross validation which aims to determine the accuracy performance of the SVM model and to select a combination of models.

2) Testing the SVM classification method with hyperparameter tuning and cross validation which aims to determine the best parameters of the SVM model based on the data used and choose the accuracy performance of hyperparameter tuning and evaluation for the best model results using cross validation.

3.1 Results and analysis of testing the SVM classification method with TF-IDF Feature Extraction and Cross Validation

Evaluating the TF-IDF feature extraction and cross validation of the SVM classification algorithm using 80:20 train and test data. Testing the SVM classification method makes use of a number of kernels, including Linear, RBF, and Polynomial, to determine which kernel has the highest accuracy. For the SVM model, 5 Fold Cross Validation was used to evaluate the model. Table 10 presents the accuracy findings.

Table 10. The Best Performance Results For The SVM Model

Aspect Accuracy

Linear Polynomial Radial Basis Function

Camera 98.07% 90.05% 97.47%

Battery 97.52% 87.42% 96.92%

Design 96.82% 88.90% 96.45%

Price 98.62% 96.87% 98.30%

Specification 97.07% 89.50% 96.35%

Based on Table 10, it can be seen that the Linear kernel produces the best accuracy value among other kernels for each aspect, with accuracy values for the camera aspect of 98.07%, the battery aspect of 97.52%, the design aspect of 96.82%, the price aspect of 98.62%, and the specification aspect of 97.07%.

3.2 Results and analysis of testing the SVM classification method with Hyperparameter Tuning and Cross Validation

After getting the best accuracy using the SVM method with TF-IDF feature extraction, the next test is to find the best parameters for the SVM classification model using Hyperparameter Tuning. Hyperparameter setting is done with the help of the grid search method. For the parameter 'C' we find that the linear and polynomial kernels have the same value in each aspect. The 'degree' parameter in each aspect also has the same value. Meanwhile, the 'gamma' parameter in each kernel has the same value in every aspect. However, there are also those that have different values, such as the camera aspect and price. A summary of the hyperparameter setting results is presented in Table 11.

Table 11. The Best Performance Results Hyperparameter Tuning Aspect Kernel Best C Best gamma Best degree Accuracy

Camera

Linear 1 - - 98.07%

Polynomial 1 scale 1 98.07%

RBF 1000 auto - 97.95%

Battery

Linear 1 - - 97.52%

Polynomial 1 scale 1 97.52%

RBF 10 scale - 97.32%

Design

Linear 10 - - 97.02%

Polynomial 10 scale 1 97.02%

RBF 10 scale - 96.87%

Linear 10 - - 98.82%

Aspect Kernel Best C Best gamma Best degree Accuracy

Price Polynomial 10 scale 1 98.82%

RBF 1000 auto - 98.52%

Specification

Linear 10 - - 97.22%

Polynomial 10 scale 1 97.22%

RBF 10 scale - 96.87%

Based on Table 11, it can be seen that using the Support Vector Machine + Hyperparameter Tuning + Cross Validation classification method on Linear, Polynomial, and RBF kernels for each aspect produces a better accuracy value than the first scenario. We found an increase in accuracy for the camera aspect only from the Polynomial and RBF kernels of 8.02% and 0.48% respectively, while from the Linear kernel there was no increase in accuracy. In the battery aspect, there was also an increase in accuracy only from the Polynomial and RBF kernels, which were 10.1% and 0.4% respectively, while from the Linear kernel there was no increase in accuracy.

In the design aspect, the accuracy of the Linear, Polynomial, and RBF kernels increased by 0.2%, 8.12% and 0.42%, respectively. In the price aspect, the accuracy of the Linear, Polynomial, and RBF kernels increased by 0.2%, 1.95% and 0.22%, respectively. In the specification aspect, the accuracy of the Linear, Polynomial, and RBF kernels also increased by 0.15%, 7.72% and 0.52%, respectively.

Comparison of accuracy values between untuned kernels and those with hyperparameters that have been set is presented in Figure 2.

Camera Aspect Hyperparameter Result

Battery Aspect Hyperparameter Result

Design Aspect Hyperparameter Result

Price Aspect Hyperparameter Result

Specification Aspect Hyperparameter Result

Figure 2. Comparison Of Hyperparameter Tuning For Each Aspect

Based on Figure 2, it can be seen that the results show that the difference is not too significant between the tuned and untuned kernels in every aspect. This is because the accuracy value before setting the hyperparameter has produced a good performance value. In each aspect, we found an increase in the accuracy value of each kernel.

The highest increase occurred in the polynomial kernel because the polynomial kernel uses 3 parameters compared to other kernels. Therefore, the use of SVM classification with Hyperparameter Tuning and Cross Validation on each aspect can help improve accuracy performance results for all kernels used.

3.3 Model Validation

We need a model validation that includes accuracy, precision, recall, and f1-score values for the train data and test data from system performance utilizing the Confusion Matrix in order to achieve the right results for the created model. Table 12 displays the system performance results for each facet of the data train.

Based on the results in Table 12 for the camera aspect, the 2 best models were obtained, namely Linear and Polynomial which produced the same values of accuracy, precision, recall, and F1-score, namely 99.35%, 99.57%, 87.46%, and 92.14%, respectively. For the battery aspect, 1 best model was obtained, namely RBF which produced accuracy, precision, recall, and F1-score of 100%. In the design aspect, the 1 best model was obtained, namely RBF which produced accuracy, precision, recall, and F1-score of 100%. For the price aspect, the 2 best models were obtained, namely Linear and Polynomial with the same values of accuracy, precision, recall, and F1-score, namely 100%. Meanwhile, in terms of specifications, the best model is 1, namely RBF which produces accuracy, precision, recall, and F1-score of 100%. Thus, the results of model validation on the data train obtained the 2 best models, namely Linear and Polynomial for camera and price aspects. As for the aspects of the battery, design, and specifications, the best model is the RBF.

Table 12. Validation Result of Data Train

Aspect Model Evaluation

Accuracy Precision Recall F1-Score

Camera

Linear 99.35% 99.57% 87.46% 92.14%

Polynomial 99.35% 99.57% 87.46% 92.14%

RBF 98.95% 99.28% 77.34% 82.75%

Battery

Linear 99.40% 98.02% 93.47% 95.55%

Polynomial 99.40% 98.02% 93.47% 95.55%

RBF 100% 100% 100% 100%

Design

Linear 99.95% 99.95% 99.17% 99.56%

Polynomial 99.95% 99.95% 99.17% 99.56%

RBF 100% 100% 100% 100%

Price

Linear 100% 100% 100% 100%

Polynomial 100% 100% 100% 100%

RBF 99.22% 99.15% 68.54% 70.05%

Specification

Linear 99.97% 99.96% 99.98% 99.97%

Polynomial 99.97% 99.96% 99.98% 99.97%

RBF 100% 100% 100% 100%

Whereas in the test data, for the camera aspect, the 2 best models were obtained, namely Linear and Polynomial which produced the same values of accuracy, precision, recall, and F1-score, namely 97.70%, 98.50%, 75.60%, and 80.63% respectively. For the battery aspect, the 3 best models were obtained, namely Linear, Polynomial, and RBF which produced the same accuracy value of 98.20%, while the 1 best model was obtained, namely RBF for precision, recall, and F1-score values. For the design aspect, the 2 best models were obtained, namely Linear and Polynomial which produced the same values of precision, recall, and F1-score, namely 97.46%, 79.24%, and 84.79% respectively, while 1 best model was obtained, namely RBF for accuracy values. In terms of price, the 2 best models were found, namely Linear and Polynomial which produced the same values of accuracy, precision, recall, and F1-score, namely 98.90%, 92.45%, 85.19%, and 88.19%, respectively. While in terms of specifications, the 2 best models were obtained, namely Linear and Polynomial which produced the same accuracy, recall, and F1-score values, namely 97.50%, 84.00%, and 83.88% respectively, while 1 best model was obtained, namely RBF for precision values. System performance results are presented in Table 13.

Table 13. Validation Result of Data Test

Aspect Kernel Evaluation

Accuracy Precision Recall F1-Score

Camera

Linear 97.70% 98.50% 75.60% 80.63%

Polynomial 97.70% 98.50% 75.60% 80.63%

RBF 97.50% 98.41% 71.67% 75.43%

Battery

Linear 98.20% 82.46% 74.57% 77.12%

Polynomial 98.20% 82.46% 74.57% 77.12%

Aspect Kernel Evaluation

Accuracy Precision Recall F1-Score RBF 98.20% 84.24% 76.61% 79.29%

Design

Linear 96.80% 97.46% 79.24% 84.79%

Polynomial 96.80% 97.46% 79.24% 84.79%

RBF 96.89% 97.30% 72.32% 76.55%

Price

Linear 98.90% 92.45% 85.19% 88.19%

Polynomial 98.90% 92.45% 85.19% 88.19%

RBF 98.50% 65.24% 66.22% 65.72%

Specification

Linear 97.50% 83.78% 84.00% 83.88%

Polynomial 97.50% 83.78% 84.00% 83.88%

RBF 97.30% 97.38% 69.99% 72.95%

4. CONCLUSION

The Support Vector Machine and Hyperparameter Tuning classification methods have been used in this final project's aspect-based sentiment analysis, which is broken down into aspects of camera, battery, design, price, and specifications, using tweet data from iPhone users taken between June 2012 and the time of crawling, or on October 29, 2022. Two scenarios have been tested, namely the first scenario uses the SVM classification method with TF- IDF Feature Extraction and Cross Validation for model evaluation. While the second scenario uses the SVM classification method with Hyperparameter Tuning and Cross Validation. Based on the two scenarios that have been tested, the accuracy performance results increase with the use of Hyperparameter Tuning of the three kernels used in each aspect. In the camera aspect, the two kernels produce the same accuracy value of 98.07% and the best parameters are C=1, kernel=Linear and C=1, gamma=scale, degree=1, kernel=Polynomial. In the battery aspect, there are two kernels that produce the same accuracy value and the best is 97.52% and the best parameters are C=1, kernel=Linear and C=1, gamma=scale, degree=1, kernel=Polynomial. In the design aspect, the two kernels produce the same accuracy value of 97.02% and the best parameters are C=10, kernel=Linear and C=10, gamma=scale, degree=1, kernel=Polynomial. In terms of price, there are two kernels that produce the same accuracy and the best is 98.82% and the best parameters are C=10, kernel=Linear and C=10, gamma=scale, degree=1, kernel=Polynomial. Meanwhile, in terms of specifications, the two kernels produce the same accuracy value of 97.22% and the best parameters are parameter C=10, kernel=Linear and parameter C=10, gamma=scale, degree=1, kernel=Polynomial. Therefore, it can be concluded that the use of the SVM method in sentiment analysis of iPhone users on Twitter produces very good accuracy values, but the hyperparameter setting is also very functional and has a positive impact on increasing accuracy performance on sentiment models when using SVM.

Suggestions for future research are to conduct research using more data and compare performance with other classification methods.

REFERENCES

[1] A. C. Sari, R. Hartina, R. Awalia, H. Irianti, and N. Ainun, “Komunikasi dan Media Sosial,” J. Messenger, vol. 3, no. 2, p. 69, 2018.

[2] S. A. Evhen and W. Maharani, “Analisis Ulasan Produk pada Media Sosial ( Twitter ) untuk Meningkatkan Kualitas Produk Handphone Menggunakan Metode Aspect-Based dengan Pendekatan Lexicon,” J. Tugas Akhir Fak. Inform., vol.

8, no. 2, pp. 3621–3632, 2021.

[3] S. W. Yudha and M. Wahyudi, “Komparasi Algoritma Klasifikasi Untuk Analisis Sentimen Review Film Berbahasa Asing,” Semin. Nas. Inform. Sist. Inf. Dan Keamanan Siber, pp. 180–185, 2018.

[4] W. Paulina, F. A. Bachtiar, and A. N. Rusydi, “Analisis Sentimen Berbasis Aspek Ulasan Pelanggan Terhadap Kertanegara Premium Guest House Menggunakan Support Vector Machine,” J. Pengemb. Teknol. Inf. dan Ilmu Komput., vol. 4, no. 4, pp. 1141–1149, 2020.

[5] et al., “Klasifikasi Kabupaten Tertinggal di Kawasan Timur Indonesia dengan Support Vector Machine,” JIKO (Jurnal Inform. dan Komputer), vol. 3, no. 3, pp. 188–195, 2020, doi: 10.33387/jiko.v3i3.2364.

[6] F. Resyanto, Y. Sibaroni, and A. Romadhony, “Choosing The Most Optimum Text Preprocessing Method for Sentiment Analysis: Case:iPhone Tweets,” Proc. 2019 4th Int. Conf. Informatics Comput. ICIC 2019, pp. 2–6, 2019, doi:

10.1109/ICIC47613.2019.8985943.

[7] W. Bourequat and H. Mourad, “Sentiment Analysis Approach for Analyzing iPhone Release using Support Vector Machine,” Int. J. Adv. Data Inf. Syst., vol. 2, no. 1, pp. 36–44, 2021, doi: 10.25008/ijadis.v2i1.1216.

[8] J. W. Iskandar and Y. Nataliani, “Perbandingan Naïve Bayes, SVM, dan k-NN untuk Analisis Sentimen Gadget Berbasis Aspek,” J. RESTI (Rekayasa Sist. dan Teknol. Informasi), vol. 5, no. 6, pp. 1120–1126, 2021, doi:

10.29207/resti.v5i6.3588.

[9] R. S. Jagdale, V. S. Shirsat, and S. N. Deshmukh, “Sentiment analysis on product reviews using machine learning techniques,” Adv. Intell. Syst. Comput., vol. 768, pp. 639–647, 2019, doi: 10.1007/978-981-13-0617-4_61.

[10] C. G. Siji George and B. Sumathi, “Grid search tuning of hyperparameters in random forest classifier for customer feedback sentiment prediction,” Int. J. Adv. Comput. Sci. Appl., vol. 11, no. 9, pp. 173–178, 2020, doi:

10.14569/IJACSA.2020.0110920.

[11] M. A. Haqmi Abas, “Agarwood Oil Quality Classification using Support Vector Classifier and Grid Search Cross Validation Hyperparameter Tuning,” Int. J. Emerg. Trends Eng. Res., vol. 8, no. 6, pp. 2551–2556, 2020, doi:

10.30534/ijeter/2020/55862020.

[12] Oryza Habibie Rahman, Gunawan Abdillah, and Agus Komarudin, “Klasifikasi Ujaran Kebencian pada Media Sosial Twitter Menggunakan Support Vector Machine,” J. RESTI (Rekayasa Sist. dan Teknol. Informasi), vol. 5, no. 1, pp. 17–

23, 2021, doi: 10.29207/resti.v5i1.2700.

[13] B. K. Bhavitha, A. P. Rodrigues, and N. N. Chiplunkar, “Comparative study of machine learning techniques in sentimental analysis,” Proc. Int. Conf. Inven. Commun. Comput. Technol. ICICCT 2017, no. Icicct, pp. 216–221, 2017, doi: 10.1109/ICICCT.2017.7975191.

[14] G. Search, “Hyperparameter Tuning pada Algoritma Klasifikasi dengan Grid Search Hyperparameter Tuning on Classification Algorithm with,” vol. 11, pp. 391–401, 2022.

[15] S. Qaiser and R. Ali, “Text Mining: Use of TF-IDF to Examine the Relevance of Words to Documents,” Int. J. Comput.

Appl., vol. 181, no. 1, pp. 25–29, 2018, doi: 10.5120/ijca2018917395.

[16] E. B. Setiawan, D. H. Widyantoro, and K. Surendro, “Feature expansion for sentiment analysis in twitter,” Int. Conf.

Electr. Eng. Comput. Sci. Informatics, vol. 2018-Octob, pp. 509–513, 2018, doi: 10.1109/EECSI.2018.8752851.

[17] H. Nuraliza, O. N. Pratiwi, and F. Hamami, “Analisis Sentimen IMBd Film Review Dataset Menggunakan Support Vector Machine (SVM) dan Seleksi Feature Importance,” J. Mirai Manaj., vol. 7, no. 1, pp. 1–17, 2022.

[18] W. R. U. Fadilah, D. Agfiannisa, and Y. Azhar, “Analisis Prediksi Harga Saham PT. Telekomunikasi Indonesia Menggunakan Metode Support Vector Machine,” Fountain Informatics J., vol. 5, no. 2, p. 45, 2020, doi:

10.21111/fij.v5i2.4449.

[19] A. Michael, “Komparasi Kombinasi Pre-trained Model dengan SVM pada Klasifikasi Kematangan Kopi Berbasis Citra,”

J. Dyn. Saint, vol. 7, no. 1, pp. 42–48, 2022, [Online]. Available:

http://journals.ukitoraja.ac.id/index.php/dynamicsaint/article/view/1613

[20] I. Syarif, A. Prugel-Bennett, and G. Wills, “SVM Parameter Optimization using Grid Search and Genetic Algorithm to Improve Classification Performance,” TELKOMNIKA (Telecommunication Comput. Electron. Control., vol. 14, no. 4, p. 1502, 2016, doi: 10.12928/telkomnika.v14i4.3956.

[21] N. Sakinah, T. Badriyah, and I. Syarif, “Analisis Kinerja Algoritma Mesin Pembelajaran untuk Klarifikasi Penyakit Stroke Menggunakan Citra CT Scan,” J. Teknol. Inf. dan Ilmu Komput., vol. 7, no. 4, p. 833, 2020, doi:

10.25126/jtiik.2020743482.