5.1 Performance evaluation
5.1.2 Applying Eleven Attributes
selected attributes. SVM comes in the second place with value of 3.12s. The third place goes for Naïve Bayes with time execution of 3.96s.
Now, let us start conducting the second stage of our experiments, which is applying all 22 attributes by increasing the number of selected attributes gradually. We will start applying eleven attributes as follows:
Employed = yes
245 55 300
Employed = no
50 350 400
Total 295 405 700
Table 5. 9 Confusion matrix of ANFIS eleven attibutes classifier Accuracy for ANFIS classifier = 595 / 700 = 85 %
Class Employed =
yes
Employed = no
Total
Employed = yes
223 77 300
Employed = no
70 330 400
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
Total 283 407 700
Table 5. 10 Confusion matrix of Decision Tree eleven attibutes classifier
Accuracy for Decision Tree = 553/ 700 = 79 %
Class Employed =
yes
Employed = no
Total
Employed = yes
210 90 300
Employed = no
81 319 400
Total 291 409 700
Table 5. 11 Confusion matrix of SVM classifier with eleven attributes
Accuracy for SVM classifier = 529 / 700 = 75 %
Class Employed =
yes
Employed = no
Total
Employed = yes
187 113 300
Employed = no
82 308 400
Total 267 421 700
Table 5. 12 Confusion matrix of Naive Bayes classifier with eleven attibutes Accuracy for Naïve Bayes classifier = 495/ 700 = 69.8 %
Class Employed =
yes
Employed = no
Total
Employed = yes
236 64 300
Employed = no
63 337 400
Total 299 408 700
Table 5. 13 Confusion matrix of MLP classifier with eleven attibutes Accuracy for MLP classifier = 573 / 700 = 81 %
The above tables show that the accuracy of the ANFIS classifier has increased when "English_skills" added to selected attributes list, and it has achieved the highest accuracy with value of 85%. The MLP classifier comes in second place with an accuracy of 81 %. Decision tree came in third place and has got an accuracy
of 79 %. Fourth place goes to SVM classifier with an accuracy of 74 %. Naive Bayes has got the lowest accuracy with an accuracy of 69.8%. As shown (Figure 5.5), ANFIS and MLP classifiers keep their superiority in achieving the best accuracy over the other classifiers. Based on the previous accuracy enhancement, we found that "English_skills" attribute has an effective role in improving the accuracy of the prediction.
Figure 5. 5 Efficiency comparison of classifiers with eleven attributes
Table (5.14) shows Recall, False-Positive rate, Precision, and F-score values for applying the prediction classifiers with eleven attributes. The measure values in the table below are for both classes "Employed" and
"Not-employed".
Classifier Class label Recall (%) False-Positive rate (%)
Prec. (%) F-score (%)
ANFIS Employed 86.4 1.6 87.2 87.9
Not- employed
86.7 2.2 85.7 87.5
Employed 82.4 2.8 82.8 82.5
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
ANFIS Decision tree
SVM Naïve bayes
MLP
Accuracy
Accuracy
Decision tree
Not- employed
81.6 3.1 81.7 82.6
SVM Employed 78.4 4.2 77.7 77.8
Not- employed
78.1 4.4 78.2 76.7
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
Naïve Bayes
Employed 74.2 5.2 73.7 74.2
Not- employed
73.7
1
4.1 73.4 72.9
MLP Employed 84.2 2.1 84.7 83.9
Not- employed
83.9 2.3 84.2 83.8
Table 5. 14Detailed accuracy by each class for eleven attributes classifiers.
As shown in Table (5.14), after applying eleven attributes to all prediction classifiers, the ANFIS classifier gained the highest values for Recall, Precision, Recall, F-score; and lowest value for False-Positive rate.
These predicting values for both classes (Employed, Not-employed) has enhanced when added
"English_skills" attribute to selected attributes list.
Figure 5. 6 F-score of employed and not employed classes for each classifier with applying eleven attributes
As shown in Figure (5.6) the prediction ratio enhanced when adding "English_skills" to the selected attributes list, which indicates good weight of this new added attribute.
Figure 5. 7 False-Positive rate of employed and not employed classes for each classifier with eleven attributes
65 70 75 80 85 90
ANFIS Decision
tree SVM
Naïve bayes MLP
F-measure (%) of employed
F-measure (%) of not- employed
0 1 2 3 4 5 6 7 8
ANFIS Decision
tree SVM
Naïve bayes MLP
FP-rate (%) of employed
FP-rate (%) of not- employed
As shown in Figure (5.7) the False-Positive rate values of the employed class for all classifier are lower than False-Positive rate values of not-employed class. This indicates again, the excellent ratio of prediction for the “employed” class when applying eleven attributes. Furthermore, the previous result proves that the accuracy of the ANFIS classifier is the best over other classifiers. Still, the continuous in error reduction indicates better efficiency when increasing the number of attributes used in building the classification.
Table (5.15) shows the values of RMSE and Kappa measures to estimate the efficiency of the prediction classifier when applying eleven attributes including "English_skills".
Classifier RMSE Kappa statistic
ANFIS 0.2003 0.8746
Decision tree 0.2665 0.8237
SVM 0.2978 0.7832
Naïve Bayes 0.3675 0.7564
MLP 0.2163 0.8489
1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 11 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 v 1 1 1 1 1 1 1 1 v v1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 v v1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1
Table 5. 15 RMSE and Kappa statistic values for each classifier applying eleven attributes
As shown in Table (5.15), Kappa and RMSE measure values have improved when adding "English_skills"
attribute. Figure (5.8) shows an efficiency comparison of classifiers according to RMSE and Kappa statistic values when applying eleven attributes.
Figure 5. 8 An efficiency comparison of classifiers according to RMSE and Kappa statistic values when applying eleven attributes
As shown in Figure (5.8), the efficiency performance of all classifiers has enhanced the same according to RMSE and Kappa measures. This confirms the effective of "English_skills" attribute in increasing the accuracy ratio.
Table (5.16) shows the execution time for each classifier, which indicates the computational cost in building classification models implementing eleven attributes.
Classifier Execution Time (Secs)
ANFIS 5.57
Decision tree 2.95
SVM 3.16
Naïve Bayes 4.00
MLP 6.36
Table 5. 16 Execution time for eleven attributes classifiers RMSE
00.20.40.6 0.81 RMSE
Kappa statistic
As shown from Table (5.16), execution time for implementing ANFIS classifier when using eleven attributes has increased. As shown in the above table, MLP classifier has got the highest value of execution time with 6.36s, followed by ANFIS with 5.57s. The values of execution time for all classifiers have also increased, but the increase in time was acceptable. Decision tree has gained the lowest value of execution time of 2.95s, which indicates again its best time efficiency over the other classifiers when increasing the number of attributes. SVM comes in the second place with a value of 3.16s. The third place goes to Naïve Bayes with time execution of 4.00s.