Sentiment Analysis On Customer Reviews Using Support Vector Machine and Usability Scoring Using System Usability Scale

(1)

Journal of Information Technology and Computer Science Volume 6, Number 3, December 2021, pp. 236-251

Journal Homepage: www.jitecs.ub.ac.id

Sentiment Analysis On Customer Reviews Using Support Vector Machine and Usability Scoring Using System

Usability Scale

Novira Azpiranda¹, Ahmad Afif Supianto², Nanang Yudi Setiawan³, Endang Suryawati⁴, Raden Sandra Yuwana⁵, Arafat Febriandirza⁶

1,2,3 Faculty of Computer Science, Brawiajaya University, Malang, Indonesia

2,4,5,6Research Center for Informatics, National Research and Innovation Agency, Bandung, Indonesia

{¹[email protected], ²[email protected], ³[email protected]

4[email protected] ⁵[email protected]

6[email protected] }

Received 26 May 2021; accepted 03 December 2021

Abstract. Al-Ghiff Steak is a restaurant located in Cirebon City that offers quality steaks at affordable prices. For maintaining a competitive Al-Ghiff Steak advantage and reputation, it is important to build a good relationship with customers and have a business strategy that considers customer opinions.

However, in its implementation, Al-Ghiff Steak has difficulty when collecting and processing customer review data manually. Therefore, it is necessary to conduct sentiment analysis by utilizing Google Reviews to determine customer perspectives regarding Al-Ghiff Steak products and services. This analysis was conducted on 968 Google Review reviews from 2016 to 2020 using the Support Vector Machine (SVM) and Term Frequency-Inverse Document Frequency (TF- IDF) methods. Classification testing is done with a confusion matrix against four parameters: accuracy, precision, recall, and f1-score. SVM with TF-IDF gets accuracy value 83%, precision 64%, recall 60% and f1-score 59%. The sentiment classification result is then visualized in the form of a dashboard. We utilize the System Usability Scale (SUS) for usability testing, which produces a value of 77.5. This result achieve the Acceptable category and an Excellent rating.

Keyword : sentiment analysis, text mining, support vector machine, tf-idf, web scraping

1 Introduction

Customer relationship management creates relationships between companies and individual customers to maximize business result [1]. By utilizing data about customers and processing them using information technology, we can get more insight into customers and develop a customer-oriented business strategy to maximize the business's maximum profits. To create a good relationship between a business and customers, we must pay attention to customer's suggestions and criticism. Customer opinions or reviews are essential in business because they will affect customer satisfaction and business strategy.

(2)

Novira Azpiranda et al. Sentiment Anlysis On Customer Reviews: ... 237 Al-Ghiff Steak is the first steakhouse in Cirebon, which is famous for its cheap and quality steaks. This company strives to retain its customers, competitive advantage, and reputation as a quality steak restaurant with the lowest prices in Cirebon City to remain competitive and increase business profits. Because of these, Al-Ghiff Steak needs to maintain relations with customers and have a customer-oriented business strategy.

Based on some research about Al-Ghiff Steak's company, we observe that this company has limitations in extracting customer review information and cannot visualize customer sentiment towards products and services at Al-Ghiff Steak. There was too much data, and they process the review data manually. These make it difficult to get a picture of customer satisfaction and the other things that had to be improved.

Meanwhile, Al-Ghiff Steak wants to maintain its competitive advantage.

To overcome these problems, we conduct an aspect level sentiment analysis on Al- Ghiff Steak customer reviews to determine customer perspectives regarding Al-Ghiff Steak products and services whether they imply positive or negative sentiments. Then a dashboard will visualize the sentiment analysis result to understand and support decision-making in compiling a business strategy easily. Sentiment analysis is used to see the proclivity of an opinion on a problem or object by someone to lead to a positive or negative view [2].

Previous researchers have conducted some studies about sentiment analysis based on user reviews. In 2019 Rahat, Kahir, and Masum developed the sentiment analysis of airline user reviews using the Naïve Bayes algorithm and Support Vector Machine (SVM) [3]. The results showed that SVM's accuracy was greater than Naïve Bayes, 82.48 for SVM, while Naïve Bayes was 76.56. This paper focused on evaluation measures on the passenger's comments, so there is no visualization of the classification result. Another study was conducted by Sulaeman, Supianto, and Bachtiar in 2019 regarding the sentiment analysis on student opinion on lecture-performance in Computer Science Faculty [4]. The test conducted on the results of classification shows an average of 82% on Accuracy. Another similar study about sentiment analysis was performed by Parasati, Bachtiar, and Setiawan in 2020 regarding sentiment analysis at the aspect level of customer reviews at Bakso President Malang Restaurant using the Naïve Bayes Classifier method [5]. Customer opinion data is collected by web scraping on the TripAdvisor website and Google Reviews. The aspect categorizations used in the study are food, service, price, and atmosphere. These studies discuss the classification model's performance and usability testing on sentiment analysis dashboard , but did not discuss what reviews customers often gave.

Based on the background described, we are interested in conducting a sentiment analysis of Al-Ghiff Steak customer reviews was collected from the Google Review site. Sentiment analysis was carried out at the aspect level, using the web scraping method for data collection and Support Vector Machine (SVM) with TF-IDF for data classification. The aspect categorizations used in the study are food, staff service, physical environment, price, and general. Research conducted by Canny in 2013 proves that food quality, service quality, and physical environment positively affect customer satisfaction [6]. Then data processing results will be analyzed and visualized to support Al-Ghiff in decision-making and help them find out the improvement to increase customer satisfaction.

(3)

238 JITeCS Volume 6, Number 3, December 2021, pp 236-251

This paper conducts a sentiment analysis on Al-Ghiff Steak customer review based on three main research questions. The research questions are (1) what are the recommendations for Al-Ghiff Steak improvement, (2) how do the performance of SVM and TF-IDF in Al-Ghiff Steak customer reviews, and (3) how is the usability level of the sentiment analysis dashboard.

2 Method

There are several work steps to conduct sentiment analysis on Al-Ghiff Steak customer reviews using the support vector machine method, namely 1) Data Collection, 2) Aspect Categorization and Labeling, 3) Text Preprocessing, 4) Term Weighting, 5) Classification, 6) Confusion Matrix Evaluation, 7) Top Reviews Analysis, 8) Visualization, 9) Usability Testing. The stages can be seen in Fig. 1.

2.1 Data Collection

The data used for sentiment analysis were obtained from the Google Reviews site in the form of text. We collect data by the web scraping method using selenium in Python to extract the user's name, the date, and review text. This stage will provide an output in a CSV file containing 968 Al-Ghiff Steak customer review data in time range from 2016 to 2020.

2.2 Aspect Categorization and Labeling

After the data is acquired, the next step is to categorize and label each review. We categorize Al-Ghiff Steak customer reviews into five aspects. They are food, staff service, physical environment, price, and general aspect, which can be seen in Table 1.

After that, we give the review label, whether the review is positive or negative. Aspect categorization and data labeling is carried out by groups of 3 annators, with the aim of producing better label quality and reduced subjectivity. Determination of aspects and the sentiment label of each review is carried out by discussion between annators, where the opinion with the most votes will be taken.

2.3 Text Preprocessing

Formalization and translation are done by converting the whole text into formal language and change the foreign term into Bahasa Indonesia. Case folding is used for converting the entire text into lowercase. Punctuation and number removal stage will remove all the punctuation and number that comes in the review. Tokenize will split the sentences into tokens or words.

The next step is stemming. Stemming is the process of reducing inflection towards their root forms to standardize text, stemming is done by removing word prefixes and suffixes. Library Pyhton Sastrawi is used to stemming Indonesian text. Where sastrawi library applies the Nazief and Adriani stemming algorithm [7].

The last step is removing stop-word. The motive that stop-words should be removed from a text is that they make the text look heavier and less important for analysts.

Removing stop words reduces the dimensionality of term space [8].

(4)

Novira Azpiranda et al. Sentiment Anlysis On Customer Reviews: ... 239

Fig.1. Stages in Sentiment Analysis Table 1. Aspect Categorization

Aspect Description

Food Reviews discuss the menu's quality, such as the taste and texture of food or drinks, and menu appearance, both in terms of placement and size of food or beverages.

Staff Service Reviews discuss employees' services to customers, such as employees' behavior, service accuracy, and service speed.

Physical Environment Reviews discuss the physical environment of Al-Ghiff Steak, such as design, layout, location, atmosphere, and facilities offered by Al- Ghiff Steak.

Price Reviews discuss the prices of Al-Ghiff Steak's menus, such as price compatibility with food quality and price compatibility with customers' economic capabilities.

General Reviews that don't address specific aspects, such as food, staff service, physical environment, and price.

(5)

2.4 Term Weighting

The term weighting will be carried out on each word in the document using the TF-IDF algorithm. Term frequency inverse document frequency (TF-IDF) is a calculation that illustrates the importance of the word (term) in a document and a corpus [8]. It is a combination of term frequency (TF) and inverse document frequency (IDF) [9], which is one of the algorithms in the text extraction weighting feature. TF is a measure of the frequent appearance of a term in a document and also in the whole document in the corpus. Meanwhile, IDF is the logarithm of the ratio of the total number of documents in the corpus by the number of documents that have the term [8]. The IDF will weigh the rare term with a high value compared to the term which often occurs. Each term's importance is assumed to have an inverse proportion to the number of documents containing the term [10]. The IDF is calculated using an equation that can be seen in equation (1).

𝒊𝒅𝒇_𝒕= 𝒍𝒐𝒈_𝟏𝟎 ^𝑵

𝒅𝒇𝒕 (1)

In equation (1), N is the number of all documents in the collection, and df_t is the number of the document containing the term. TfidVectorizer from the Scikit-Learn library in Python will be used to weigh every word in customer review dataset.

TfidfVectorizer performs both term frequency and inverse document frequency so it’s easier to use. In Tfidvectorizer, the formula to calculate IDF shown in equation (2). The effect of adding "1" to the IDF in equation (2) is that terms with zero IDF, terms that occur in all documents in a training set, will not be entirely ignored [11].

𝒊𝒅𝒇_𝒕= 𝐥𝐧^𝑵+𝟏

𝒅𝒇+𝟏+ 𝟏 (2)

After the IDF score is obtained, then TFIDF can be calculated using equation (3) by multiplying the TF value with the IDF value.

𝒕𝒇 − 𝒊𝒅𝒇_𝒕,𝒅= 𝒕𝒇_𝒕,𝒅 ∙ 𝒊𝒅𝒇_𝒕 (3) 2.5 Classification

Sentiment classification was carried out on five aspects: food, staff service, physical environment, price and general. The datasets used for the classification process are a customer review dataset and a weighted dataset. We label the dataset into two classes for the classification process, namely a positive and a negative class.

Classification is done using sklearn library that provided by python. Support Vector Machine and LinearSVC Classifier with default parameter are used to classify sentiment classes. The default parameter for Linear SVC are LinearSVC (penalty='l2',loss='squared_hinge', *, dual=True, tol=0.0001, C=1.0,multi_class='ovr', fit_intercept=True, intercept_scaling=1, class_weight=None, verbose=0, random_state=None, max_iter=1000) [12].

Meanwhile, the dataset's division into 80% train data and 20% test data were carried out using Stratified K-fold Cross-Validation with five folds.

(6)

2.6 Confusion Matrix Evaluation

We evaluate the model's performance by calculating Confusion Matrix using the Scikit- Learn library in Python. Evaluation is carried out to see how good the model performance is in sentiment classification results. F1-Score, Precision, Recall, and Accuracy testing are applied to support the classification evaluation.

2.7 Top Review Analysis

The search for top reviews is performed on positive sentiments and negative sentiments from every aspect: food, staff service, physical environment, and price. The purpose is to find out the top 5 reviews that customers often give for each aspect; therefore, Al- Ghiff Steak knows what improvement they should make to increase customer satisfaction.

2.8 Visualization

We apply a dashboard using Google Data Studio to visualize the sentiment classification results. The purpose of visualization is to facilitate the delivery of information. Therefore the sentiment classification results can be understood well by the research object and support the decision-making process.

2.9 Usability Testing

We will test the designed dashboard on Al-Ghiff Steak to assess whether it can be well received and understood. The System Usability Scale (SUS) is implemented for the usability test. The System Usability Scale (SUS) is a simple, ten-item scale giving a global view of subjective assessments of usability. It allows you to evaluate a wide variety of products and services, including hardware, software, mobile devices, websites and applications.

3 Result and Analysis

3.1 Classification Result

We use 968 Al-Ghiff Steak customer reviews on Google Reviews to conduct the classification process. But, some data reviews cannot be used, such as irrelevant reviews and reviews that only use symbols. For those reasons, the dataset containing customer reviews needs to go through the preprocessing stage before classification.

After the preprocessing process, 958 data were obtained that were ready to be processed.

The data is further categorized into four specific aspects: food, staff service, physical environment, price, and one general aspect. One review can fall into several aspect categories. We classify some reviews into two classes of sentiment: positive and negative. The overall classification results in 1,203 positive reviews and 193 negative reviews.

Table 2 shows the actual data classification for five aspects. There are 361 positive reviews and 73 negative reviews for the food aspect. While on the staff service aspect, there are 54 positive reviews and 56 negative reviews. On the physical environment,

(7)

there are 292 positive reviews and 35 negative reviews. We also see 296 reviews with positive class and 21 reviews with negative class for the price aspect. For the general aspect, the table shows 200 positive reviews and eight negative reviews.

Besides the actual data classification, we also obtain the varied classification results using SVM, as shown in Table 3. The food aspect produces 423 positive reviews and 11 negative reviews. Meanwhile, 41 positive reviews and 68 negative reviews are produced for the staff service aspect. The SVM classifier can produce 272 positive reviews and 55 negative reviews on the physical environment. Price aspect has 288 positive reviews and 29 negative reviews. The SVM classifier also can produce 178 positive reviews and 30 negative reviews for the general aspect.

Table 2. Actual Data Classification

Table 3. SVM Classification Result

3.2 Testing Result

To solve the second research question: (RQ2) How do SVM and TF-IDF performance in Al-Ghiff Steak customer reviews classification? Testing the classification result is needs to be done to assess how good the model. We calculate the confusion matrix and use four testing parameters: Accuracy, Precision, Recall, and F1-Score. Table 4, Table

Aspect Class Reviews

Food Positive 361

Negative 73

Staff Service

Positive 54

Negative 56

Physical Environment

Positive 292

Negative 35

Price Positive 296

Negative 21

General Positive 200

Negative 8

Aspect Class Reviews

Food Positive 423

Negative 11

Staff Service Positive 41

Negative 68

Positive 272

Negative 55

Price Positive 288

Negative 29

General Positive 178

Negative 30

(8)

Novira Azpiranda et al. Sentiment Anlysis On Customer Reviews: ... 243 5, Table 6 and Table 7 shows the confusion matrix of food, staff service, physical environment and price aspect. The value of the testing results is obtained using the classification report function in the Scikit-learn library. Table 8 shows the test results for each of its aspects.

Accuracy shows how accurate the classification model predicts to find out what percentage of reviews the classifier correctly predicted. Table 4 shows that the price aspect owns the highest accuracy value, which is 0.87, while the lowest is owned by the physical environment aspect, which is 0.77. The dataset used is an imbalance because of the big gap between positive and negative class, so the accuracy value is insufficient to assess the classification model's performance. We must also notice the values of precision, recall, and f1-score. In the imbalanced dataset, the accuracy value can be high because the model tends to learn from the major class, predicting the class as a major class, and doesn't work well in predicting minor classes. That's why we have to pay attention to the values of precision, recall, and f1-score to get a better picture of model performance.

Precision will help us determine how reliable the results are when the model answers that a review is included in that class. The highest value of precision on positive sentiment is in general aspects, it is 0.96, while the lowest is in the food aspect, it is 0.85. The highest precision value on negative sentiment is in the staff service aspect, namely 0.78, while the lowest value is in the physical environment aspect, namely 0.03.

Recall will compare the correctly predicted data and the actual data to show how good the model is in detecting classes. The highest recall value for the positive sentiment is in the food aspect, which is 0.99, while the lowest is in the physical environment, which is 0.84. The highest recall value for negative sentiment is on the staff aspect, namely 0.96, the lowest is in general aspects with 0.12.

After obtaining the precision and recall values, the f1-score can be calculated. The highest f1-score value for the positive sentiment is in the price aspect; it is 0.93. In contrast, the lowest is in the staff service aspect with 0.82. The highest f1-score value for the negative sentiment is in the physical environment aspect, namely 0.84, while the lowest is in the physical environment aspect; it is 0.05.

By considering these four parameters, we can see that the best model's performance to classifies sentiment is in the staff service aspect. The staff service aspect has a reasonably high accuracy value of 0.84 and an average precision value of 0.87, recall of 0.84, and f1-score of 0.84. Meanwhile, the model's performance did not perform well in general aspects, even though the accuracy was 0.83. Still, the average values of precision, recall, and f1-score in this aspect are low, namely 0.5, 0.49, and 0.47. The difference in performance values can occur when the dataset used is imbalanced.

Therefore the model tends to predict the class into major class and make the high accuracy value. Even though the model is not good at predicting minor classes and has a value of precision and the ability to detect classes (recall) is lacking.high, even though in reality, the model is not good at predicting minor classes and has a value of precision and the ability to detect classes (recall) is lacking.

(9)

Table 4. Confusion Matrix of Food Aspect

Table 5. Confusion Matrix of Staff Service Aspect

Table 6. Confusion Matrix of Physical Environment Aspect

Table 7. Confusion Matrix of Price Aspect

Actual Prediction

Positive Negative

Positive 272 24

Negative 16 5

Table 8. Testing Result

Aspect Class Precision Recall F1 - Score Accuracy Food

Positive 0.85 0.99 0.91

0.84

Negative 0.73 0.11 0.19

Average 0.79 0.55 0.55

Staff Service

Positive 0.95 0.72 0.82

0.84

Negative 0.78 0.96 0.86

Average 0.87 0.84 0.84

Positive 0.90 0.84 0.87

0.77

Negative 0.13 0.20 0.16

Average 0.52 0.52 0.52

Price

Positive 0.94 0.92 0.93

0.87

Negative 0.17 0.24 0.20

Average 0.56 0.58 0.57

General

Positive 0.96 0.85 0.90

0.83

Negative 0.03 0.12 0.05

Average 0.5 0.49 0.47

Average 0.64 0.6 0.59 0,83

Actual Prediction

Positive Negative

Positive 358 3

Negative 65 8

Actual Prediction

Positive Negative

Positive 39 15

Negative 2 53

Actual Prediction

Positive Negative

Positive 244 48

Negative 28 7

(10)

Novira Azpiranda et al. Sentiment Anlysis On Customer Reviews: ... 245 3.3 Top Reviews Analysis

It is important to include information about the most reviews given by customers because it can help decision-makers when evaluating or improving business and service operations. The top review analysis was carried out on four aspects: food, staff service, physical environment, and price, to find out the top 5 reviews that customers often give for each aspect, both positive and negative sentiments. The top review is determined based on the weight of the sentence, where the sentence with a large weight indicates that the review often appears.

Analyzing customer reviews can help us determine what improvement should be made to increase customer satisfaction in Al-Ghiff Steak. This is addressed the first research question: (RQ1) What recommendations can be given for improvement?

a. Food Aspect

The top 5 negative reviews that customers often give are shown in Table 9. Many customers comment about the quality of the food, such as the taste and materials are not good. Therefore based on that recommendations, the company can carry out regular improvements. One of them is frequent checks on food, ingredients, and raw materials used to ensure quality. Table 10 shows the top 5 positive reviews, where many customers were satisfied with the taste of the food.

Table 9. Top 5 Negative Reviews on Food Aspect

Review Rank

The food is sometimes good sometimes not

Makanannya kadang enak kadang tidak

1 The taste of the steak is average

Rasa steaknya biasa saja 2

Cheap, average taste

Murah, rasa biasa saja 3

Tasty but the meat is not soft

Enak tetapi dagingnya tidak lembut 4 The price doesn't match the quality... The

taste is also average..

Harga tidak sesuai dengan kualitas...

Rasa juga biasa saja..

5

Table 10. Top 5 Positive Reviews on Food Aspect

Review Rank

The food is delicious

Makanannya enak 1

The best steak

Steak terenak 2

Delicious and cheap

Enak dan murah 3

Delicious and cozy

Enak dan nyaman 4

Cheap and delicious food

Harga makanan yang murah dan enak 5

(11)

b. Staff Service Aspect

The top 5 negative reviews that customers often give are shown in Table 11. From the Table, we observe that many customers comment on unsatisfactory services, such as the late orders and the waiter attitude is not friendly. Therefore it is necessary to evaluate regularly and provide training for employees. Table 12 shows the top 5 positive reviews, where many customers are satisfied with the services offered.

Table 11. Top 5 Negative Reviews on Staff Service Aspect

Review Rank

Slow service

Pesanan lama 1

The service is not satisfactory

Pelayanannya tidak memuaskan 2 Very long service

Pelayanan lama banget 3

Waiters are not friendly

Pelayan kurang ramah 4

Unsatisfactory service, slow service, cold food served

Pelayanan tidak memuaskan, lama , makanan sudah dingin disajikan

5

Table 12. Top 5 Positive Reviews on Staff Service Aspect

Review Rank

Cozy place, friendly waiter

Tempat nyaman, pelayannya ramah 1 Fast service

Pelayanannya cepat 2

Good service

Pelayanan baik 3

Comfortable place, friendly service.

Tempat nyaman, pelayanan cukup

ramah. 4

Comfortable place, good food, satisfying service

Tempat nyaman, makanan enak, pelayanan memuaskan

5

c. Physical Environment Aspect

The top 5 negative reviews that customers often give are shown in Table 13. Many customers comment about the atmosphere and the too little place. Table 14 shows the top 5 positive reviews, where many customers feel comfortable and like Al-Ghiff Steak's atmosphere.

(12)

Novira Azpiranda et al. Sentiment Anlysis On Customer Reviews: ... 247 Table 13. Top 5 Negative Reviews on Physical Environment Aspect

Review Rank

The place is not fun

Tempatnya kurang asyik 1

The place is too narrow

Tempatnya terlalu sempit 2

Parking space is not wide

Parkiran kurang luas 3

The place is noisy, not comfortable

Berisik tempatnya kurang nyaman 4 Not sure where the place is

Belum tahu jelas tempatnya 5

Table 14. Top 5 Positive Reviews on Physical Environment Aspect

Review Rank

Nice place

Tempatnya enak 1

The place is comfortable

Tempatnya nyaman 2

The place is quite comfortable

Cukup nyaman tempatnya 3

Good place

Tempatnya bagus 4

Strategic and comfortable place

Strategis dan tempatnya nyaman 5 d. Price Aspect

The top 5 negative reviews that customers often give are shown in Table 15. Many customers have commented on the increase in menu prices at Al-Ghiff Steak. Hence, the recommendation is to improve food quality so that customers feel that the price is by the quality obtained. Table 16 shows the top 5 positive reviews, where many customers feel that the price offered by Al-Ghiff matches the quality of food and services they get.

Table 15. Top 5 Negative Reviews on Price Aspect

Review Rank

It's expensive now.

Sudah mahal sekarang. 1

The price is a bit expensive now Harganya Sekarang agak lumayan mahal ya

2 Now it's more expensive

Sekarang tambah mahal harganya 3 Now a bit expensive and yet not

include tax 10%

Sekarang agak mahal belum ppn 10%

4

It's quite expensive

Walaupun lumayan mahal 5

(13)

Table 16. Top 5 Positive Reviews on Price Aspect

Review Rank

Cheap price

Harga murah 1

Delicious and cheap

Enak dan murah 2

Cheap and delicious food Harga makanan yang murah dan enak

3 Delicious and the price is still

affordable

Enak dan harga masih terjangkau 4 Cheap and delicious steak

Steak murah dan enak 5

3.4 Dashboard

The results of the sentiment analysis are then displayed in a dashboard. The tool is used to create a dashboard is the Google Data Studio. Al-Ghiff Steak's sentiment analysis dashboard consists of 3 pages. There are an overview page, a detail aspect page, and a prediction page.

Figure 2 shows the overview page that displays the results of the analysis as a whole. Graphs display total reviews, a pie chart that compares the number of positive and negative sentiments, and a trend graph to show the comparison of positive and negative reviews in the past year.

The aspect detail page is shown in Figure 3, which discusses sentiment analysis in more detail on each aspect. The graph displayed includes a bar chart that shows the number of reviews and the comparison of positive and negative reviews on each aspect.

In addition, there is also a table that displays positive reviews and negative reviews most often given by customers on aspects of food, staff service, physical environment and price.

As shown in Figure 4, the prediction page displays the information on SVM's classification results. The information display includes the overall classification and classification in every aspect.

Fig.2. Overview Page

(14)

Fig.3. Aspect Detail Page

Fig.4. Prediction Page 3.5 Usability Testing

To assess the level of usability from the dashboard and answer the third research question: (RQ3) How is the usability level of the sentiment analysis dashboard? A usability evaluation was carried out using a system usability scale (SUS) questionnaire.

The SUS questionnaire calculation is carried out with the following rules:

 For odd items: subtract one from the user response.

 For even-numbered items: subtract the user responses from 5

 Add up the converted responses for each user and multiply that total by 2.5.

This converts the range of possible values from 0 to 100 instead of from 0 to 40.

The respondents in this usability evaluation is Al-Ghiff Steak restaurant owners, Herman Rahmadi. Table 17 shows the results from the SUS questionnaire on the Al- Ghiff Steak sentiment analysis dashboard. The value obtained based on the SUS questionnaire calculation is 77.5. This value is included in the Acceptable category with

(15)

an "Excellent" rating. It shows that users can appropriately use the sentiment analysis dashboard.

Table 17. SUS Questionnaire Result

4 Conclusion

After conducting a research discussion, the main aspect for improvement is the aspect of staff service. In this aspect, the customers gave more negative reviews than positive reviews. Almost 51% of customer reviews were negative. Customers commented a lot about late and inaccurate service, because the staff often wrong when delivering food.

Recommendations given are doing regular evaluations and providing training to employees to understand their duties and carry them out properly.

Support Vector Machine (SVM) and Term Frequency-Inverse Document Frequency (TF-IDF) can be used to solve the classification problem of aspect-level sentiment analysis on Al-Ghiff Steak customer review data. Testing with confusion matrix produces an accuracy value of 83%, precision 64%, recall 60%, and an f1-score of 59%, which means that the model is quite good at providing correct predictions, but the precision and ability of the model to detect sentiment classes is still low. One of the factors that influence is the imbalance dataset, so the model tends to predict the class into the major class and is less able to detect the minor class.

A sentiment analysis dashboard is developed to facilitate information delivery to the object of research easily. The hope is dashboard can be used and useful for Al-Ghiff Steak decision making. The usability evaluation results give a score of 77.5, where this value is included in the Acceptable category with a rating of "Excellent." The SUS result shows that Al-Ghiff Steak Cirebon can appropriately use the sentiment analysis dashboard.

Statement number- Answer Result (t)

1 3 3-1 = 2

2 2 5-2 = 3

3 4 4-1 = 3

4 2 5-2 = 3

5 4 4-1 = 3

6 2 5-2 = 3

7 5 5-1 = 4

8 1 5-1 = 4

9 4 4-1= 3

10 2 5-2 = 3

Total 31

Result ( total*2,5) 77,5

(16)

References

[1] G. I, "Best Practices: Customer Relationship Management," Ivey Business Journal, 2002.

[2] M. Berry and J. Kogan, Text Mining Application and Theory, United Kingdom : Wiley, 2010.

[3] A. M. Rahat, A. Kahir and A. K. M. Masum, "Comparison of Naive Bayes and SVM Algorithm based on Sentiment Analysis Using Review Dataset," 8th International Conference System Modeling and Advancement in Research Trends (SMART), 2019.

[4] A. F. Sulaeman, A. A. Supianto and F. A. Bachtiar, "Analisis Sentimen Opini Mahasiswa Terhadap Saran Evaluasi Kinerja Dosen Menggunakan TF-IDF dan Support Vector Machine.," Jurnal Pengembangan Teknologi Informasi dan Ilmu Komputer, vol. 3, no. 6, pp. 5647-5655, 2019.

[5] W. Parasati, F. A. Bachtiar and N. Y. Setiawan, "Analisis Sentimen Berbasis Aspek pada Ulasan Pelanggan Restoran Bakso President Mlang dengan Metode Naive Bayaes Classifier," Jurnal Pengembangan Teknologi Informasi dan Ilmu Komputer, vol. 4, no. 4, pp. 1090-1099, 2020.

[6] I. U. Canny, "The Role of Food Quality, Service Quality, and Physical Environment on Customer Satisfaction and Future Behavioral Intentions in Casual Dining Restaurant," in The 7th National Research Management Conference, Sriwijaya University, Palembang, 2013.

[7] M. Adriani, J. Asian, B. Nazief, S. Tahaghoghi and H. E. Williams, "Stemming Indonesian: A Conﬁx-StrippingApproach," ACM Transactions on Asian Language Information Processing, vol. 6, no. 4, pp. 1-33, 2007.

[8] D. S. Vijayarani, M. J. Ilamathi and M. Nithya, "Preprocessing Techniques for Text Mining - An Overview," International Journal of Computer Science &

Communication Networks, vol. 5, no. 1, pp. 7-16, 2015.

[9] C. D. Manning, P. Raghavan and H. Schutze, Introduction to Information Retrieva, New York: Cambridge University Press, 2008.

[10] M. A. Fauzi, A. Arifin and A. Yuniarti, "Term Weighting Berbasis Indeks Buku dan Kelas untuk Perangkingan Dokumen Berbahasa Arab," Jurnal Ilmiah Teknologi Informasi, vol. 5, no. 2, 2014.

[11] scikit learn, "sklearn.feature_extraction.text.TfidTransformer," [Online].

Available: https://scikit-

learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfTransfo rmer.html#sklearn.feature_extraction.text.TfidfTransformer. [Accessed 30 November 2020].

[12] scikit learn, "sklearn.svm.LinearSVC," [Online]. Available: https://scikit- learn.org/stable/modules/generated/sklearn.svm.LinearSVC.html. [Accessed 14 October 2021].

[13] M. N. Saadah, R. W. Atmagi, D. S. Rahayu and A. Z. Arifin, "Information Retrieval of Text Document With Weighting TF-IDF and LCS," Journal of Computer Sciences and Information, 2013.