5.1 Performance evaluation
5.1.4 Applying Nineteen Attributes
4.74s, which shows again its best time efficiency over the other classifiers when increasing the number of attributes. SVM comes in the second place with a value of 4.63s. The third place goes to Naïve Bayes with time execution of 6.18s.
Class Employed = yes
Employed = no
Total
Employed = yes
275 25 300
Employed = no
20 380 400
Total 295 405 700
Table 5. 25 Confusion matrix of ANFIS nineteen attibutes classifier Accuracy for ANFIS classifier = 625 / 700 = 89 %
Class Employed =
yes
Employed = no
Total
Employed = yes
253 47 300
Employed = no
40 360 400
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
Total 283 407 700
Table 5. 26 Confusion matrix of Decision Tree nineteen attibutes classifier Accuracy for Decision Tree = 583/ 700 = 83 %
Class Employed =
yes
Employed = no
Total
Employed = yes
240 60 300
Employed = no
51 349 400
Total 291 409 700
Table 5. 27 Confusion matrix of SVM classifier with nineteen attributes Accuracy for SVM classifier = 559 / 700 = 79 %
Class Employed =
yes
Employed = no
Total
Employed = yes
217 83 300
Employed = no
52 338 400
Total 267 421 700
Table 5. 28 Confusion matrix of Naive Bayes classifier with nineteen attibutes Accuracy for Naïve Bayes classifier = 525/ 700 = 75 %
Class Employed =
yes
Employed = no
Total
Employed = yes
266 34 300
Employed = no
33 367 400
Total 299 408 700
Table 5. 29 Confusion matrix of MLP classifier with nineteen attibutes Accuracy for MLP classifier = 603 / 700 = 86 %
The tables above show that the accuracy ratio of applying the ANFIS classifier has not improved when
"Team work skills" attribute added to selected attributes list, and it remains in the first place with accuracy ratio of 89%, which is the same ratio over the previous stage. The accuracy ratio of MLP classifier not affected with the same accuracy ratio of 86 %. Decision tree came in third place and has got an accuracy of 83 %. Fourth place goes to SVM classifier with an accuracy of 79 %. Naive Bayes has got the lowest accuracy with an accuracy of 75%. As shown (Figure 5.65), ANFIS and MLP classifiers keep the highest accuracy ratio over the other classifiers.
Figure 5. 13 Efficiency comparison of classifiers with nineteen attributes
Table (5.30) shows Recall, False-Positive rate, Precision, and F-score values for applying the classification model with nineteen attributes. The measure values in the following table are for both categories;
"Employed" and "Not-employed".
Classifier Class label Recall (%) False-Positive rate (%)
Prec. (%) F-score (%)
ANFIS Employed 89.3 1.2 88.7 89.9
Not- employed
89.8 2.0 87.5 89.4
65%
70%
75%
80%
85%
90%
95%
ANFIS Decision tree
SVM Naïve bayes
MLP
Accuracy
Accuracy
Decision tree
Employed 85.6 2.3 85.4 84.8
Not- employed
84.4 2.5 84.3 85.1
SVM Employed 81.3 4.0 80.6 80.4
Not- employed
80.7 3.9 81.4 80.3
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
Naïve Bayes
Employed 77.5 4.9 76.8 76.6
Not- employed
76.3 4.6 75.2 75.6
MLP Employed 87.6 1.8 87.2 86.4
Not- employed
86.3 2.0 87.5 86.4
Table 5. 30 Detailed accuracy by each class for nineteen attributes classifiers.
As shown in Table (5.30), after building prediction classifiers using nineteen attributes, the ANFIS classifier has obtained the largest values for Recall, Precision, Recall, F-score; and less value for False-Positive rate.
The accuracy ratio has not improved when added "Team_work_skills" attribute to selected attributes list.
Figure 5. 14 F-score of employed and not employed classes for each classifier with applying nineteen attributes
As shown in Figure (5.14) the prediction ratio has enhanced when "Team work skills" attribute added to the selected attributes list, which shows less weight of this attribute.
Figure 5. 15 False-Positive rate of employed and not employed classes for each classifier with nineteen attributes
As shown in Figure (5.15) the False-Positive rate values of the employed class for all classifier are lower than False-Positive rate values of not-employed class. This shows also, the excellent ratio of prediction for the “employed” class when applying nineteen attributes.
0 10 20 30 40 50 60 70 80 90 100
ANFIS Decision
tree SVM
Naïve bayes MLP
F-measure (%) of employed
F-measure (%) of not- employed
0 1 2 3 4 5 6
ANFIS Decision
tree SVM
Naïve bayes MLP
FP-rate (%) of employed
FP-rate (%) of not- employed
Table (5.31) shows the values of RMSE and Kappa measures to measure the efficiency of the prediction classifier when using nineteen attributes including "Team work skills" attribute.
Classifier RMSE Kappa statistic
ANFIS 0.1838 0.8963
Decision tree 0.2341 0.8465
SVM 0.2748 0.8054
Naïve Bayes 0.3303 0.7779
MLP 0.1965 0.8684
Table 5. 31RMSE and Kappa statistic values for each classifier applying nineteen attributes
As shown in Table (5.31), Kappa and RMSE measure values have not really enhanced when adding "Team work skills" attribute. Figure (5.16) shows an efficiency comparison of classifiers according to RMSE and Kappa statistic values when applying nineteen attributes.
Figure 5. 16An efficiency comparison of classifiers according to RMSE and Kappa statistic values when applying nineteen attributes
As shown in Figure (5.16), the efficiency performance of all classifiers has not improved based on RMSE and Kappa measures.
RMSE
00.20.40.6 0.81 RMSE
Kappa statistic
Table (5.32) shows the execution time for each classifier, which shows the computational cost in building classification models using nineteen attributes.
Classifier Execution Time (Secs)
ANFIS 12.67
Decision tree 5.45
SVM 5.64
Naïve Bayes 8.14
MLP 13.86
Table 5. 32 Execution time for nineteen attributes classifiers
As shown from Table (5.32), execution time for applying ANFIS classifier when using nineteen attributes has increased. As shown in the above table, MLP classifier as usual has got the highest value of execution time with 13.86s, which is considered long time in building this model, then ANFIS with 12.67s. The values of execution time for all classifiers have also increased. Decision tree has got the less value of execution time of 5.45s, which proves its best time efficiency over the other classifiers when increasing the number of attributes. SVM comes in the second place with a value of 5.64s. The third place goes to Naïve Bayes with time execution of 8.14s.