5.1 Performance evaluation
5.1.3 Applying Fifteen Attributes
As shown from Table (5.16), execution time for implementing ANFIS classifier when using eleven attributes has increased. As shown in the above table, MLP classifier has got the highest value of execution time with 6.36s, followed by ANFIS with 5.57s. The values of execution time for all classifiers have also increased, but the increase in time was acceptable. Decision tree has gained the lowest value of execution time of 2.95s, which indicates again its best time efficiency over the other classifiers when increasing the number of attributes. SVM comes in the second place with a value of 3.16s. The third place goes to Naïve Bayes with time execution of 4.00s.
Class Employed = yes
Employed = no
Total
Employed = yes
255 45 300
Employed = no
40 360 400
Total 295 405 700
Table 5. 17 Confusion matrix of ANFIS fifteen attibutes classifier Accuracy for ANFIS classifier = 605 / 700 = 86 %
Class Employed =
yes
Employed = no
Total
Employed = yes
233 67 300
Employed = no
60 340 400
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
Total 283 407 700
Table 5. 18 Confusion matrix of Decision Tree fifteen attibutes classifier Accuracy for Decision Tree = 563/ 700 = 80 %
Class Employed =
yes
Employed = no
Total
Employed = yes
220 80 300
Employed = no
71 329 400
Total 291 409 700
Table 5. 19 Confusion matrix of SVM classifier with fifteen attributes Accuracy for SVM classifier = 539 / 700 = 77 %
Class Employed =
yes
Employed = no
Total
Employed = yes
197 103 300
Employed = no
72 318 400
Total 267 421 700
Table 5. 20 Confusion matrix of Naive Bayes classifier with fifteen attibutes Accuracy for Naïve Bayes classifier = 505/ 700 = 72 %
Class Employed =
yes
Employed = no
Total
Employed = yes
246 54 300
Employed = no
53 347 400
Total 299 408 700
Table 5. 21 Confusion matrix of MLP classifier with fifteen attibutes Accuracy for MLP classifier = 583 / 700 = 83 %
The above tables show that the accuracy ratio of applying the ANFIS classifier has not improved when
"Talent" attribute added to selected attributes list, and it remains in the first place with accuracy ratio of 86%.
The accuracy ratio of MLP classifier not affected with the same accuracy ratio of 83 %. Decision tree came in third place and has got an accuracy of 82 %. Fourth place goes to SVM classifier with an accuracy of 77
%. Naive Bayes has got the lowest accuracy with an accuracy of 72%. As shown (Figure 5. 9), ANFIS and MLP classifiers still achieved the highest accuracy over the other classifiers.
Figure 5. 9 Efficiency comparison of classifiers with fifteen attributes
Table (5.22) shows Recall, False-Positive rate, Precision, and F-score values for building the classification model with fifteen attributes. The measure values in the table below are for both classes "Employed" and
"Not-employed".
Classifier Class label Recall (%) False-Positive rate (%)
Prec. (%) F-score (%)
ANFIS Employed 87.3 1.4 87.9 88.7
Not- employed
87.4 2.2 86.4 87.8
Employed 83.4 2.6 83.7 83.3
65%
70%
75%
80%
85%
90%
ANFIS Decision tree
SVM Naïve bayes
MLP
Accuracy
Accuracy
Decision tree
Not- employed
82.8 2.9 82.5 83.6
SVM Employed 79.4 4.2 78.6 78.5
Not- employed
78.7 4.1 78.8 77.6
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
Naïve Bayes
Employed 75.8 5.1 74.5 74.8
Not- employed
74.6 3.8 73.5 73.7
MLP Employed 85.1 2.0 85.3 84.8
Not- employed
84.7 2.1 85.4 84.5
Table 5. 22 Detailed accuracy by each class for fifteen attributes classifiers.
As shown in Table (5.22), after building prediction classifiers using fifteen attributes, the ANFIS classifier gained the highest values for Recall, Precision, Recall, F-score; and less value for False-Positive rate, which is the same result in previous stages. The accuracy ratio remains the same when added "Talent" attribute to selected attributes list.
Figure 5. 10 F-score of employed and not employed classes for each classifier with applying fifteen attributes
As shown in Figure (5.10) the prediction ratio not enhanced when "Talent" attribute added to the selected attributes list, which shows less weight of this attribute.
Figure 5. 11 False-Positive rate of employed and not employed classes for each classifier with fifteen attributes
As shown in Figure (5.11) the False-Positive rate values of the employed class for all classifier are lower than False-Positive rate values of not-employed class. This indicates also, the excellent ratio of prediction for the “employed” class when applying fifteen attributes. On the other hand, the previous result proves that the accuracy of the ANFIS classifier is the best over other classifiers using our dataset.
0 10 20 30 40 50 60 70 80 90 100
ANFIS Decision
tree SVM
Naïve bayes MLP
F-measure (%) of employed
F-measure (%) of not- employed
0 1 2 3 4 5 6
ANFIS Decision
tree SVM
Naïve bayes MLP
FP-rate (%) of employed
FP-rate (%) of not- employed
Table (5.23) shows the values of RMSE and Kappa measures to measure the efficiency of the prediction classifier when using fifteen attributes including "Talent".
Classifier RMSE Kappa statistic
ANFIS 0.1953 0.8859
Decision tree 0.2432 0.8358
SVM 0.2834 0.7969
Naïve Bayes 0.3441 0.7686
MLP 0.2011 0.8577
Table 5. 23 RMSE and Kappa statistic values for each classifier applying fifteen attributes
As shown in Table (5.23), Kappa and RMSE measure values have not enhanced when adding "Talent"
attribute. Figure (5.12) shows an efficiency comparison of classifiers according to RMSE and Kappa statistic values when applying fifteen attributes.
Figure 5. 12 An efficiency comparison of classifiers according to RMSE and Kappa statistic values when applying fifteen attributes
As shown in Figure (5.12), the efficiency performance of all classifiers has not improved according to RMSE and Kappa measures.
Table (5.24) shows the execution time for each classifier, which indicates the computational cost in building classification models implementing fifteen attributes.
Classifier Execution Time (Secs)
ANFIS 8.46
Decision tree 4.74
SVM 4.63
Naïve Bayes 6.18
MLP 8.84
Table 5. 24 Execution time for fifteen attributes classifiers
As shown from Table (5.24), execution time for implementing ANFIS classifier when using fifteen attributes has increased. As shown in the above table, MLP classifier has got the highest value of execution time with 8.84s, then ANFIS with 8.46s. The values of execution time for all classifiers have also increased, but the rate of increase of time ratio is the same for all. Decision tree has got the less value of execution time of
RMSE
00.20.40.6 0.81 RMSE
Kappa statistic
4.74s, which shows again its best time efficiency over the other classifiers when increasing the number of attributes. SVM comes in the second place with a value of 4.63s. The third place goes to Naïve Bayes with time execution of 6.18s.