174 9 Credit Risk Analysis with a SVM-based Metamodeling Ensemble bound and kernel parameters are presented. For SVM base models, a stan- dard SVM model with Gaussian kernel function is used. In such way, we have obtained 40 SVM base models. Subsequently, model pruning step is performed. Through PCA, we can select 20 base models as a simplified set of SVM base models. Finally, using the selected 38 SVM base models and validation, we can formulate a metadata set and thus obtain the final classi- fication results with the use of the SVM metalearning strategy.
To evaluate the performance of the proposed SVM-based metamodel, several typical credit scoring models, linear discriminant analysis (LDA), logit regression analysis (LogR), single artificial neural network (ANN) and support vector machine (SVM), are selected as benchmarks. For fur- ther comparison of SVM metamodel, majority voting based metamodel and neural-network-based metamodel are also adopted for credit scoring.
In the ANN model, a three-layer back-propagation neural network with 13 input nodes, 15 hidden nodes and 1 output nodes is used. The hidden nodes use sigmoid transfer function and the output node uses the linear transfer function. In the single standard SVM, the kernel function is Gaussian func- tion with regularization parameter C = 50 and σ2=5. Similarly, the above parameters are obtained by trial and error. For two metamodels, similar four stages are performed and the unique difference is the final metalearn- ing stage because they adopt different metalearning strategies. The classi- fication accuracy in testing set is used as performance evaluation criterion.
In additionn, the classification performance is measured by its Type I, Type II, and Total accuracy.
9.3.2 Experimental Results
According to the above experiment design, different credit risk evaluation model with different parameters can be built. For comparison, the single LDA, LogR, ANN, SVM, majority-voting-based metalearning and ANN- based metalearning approach, are also used. To overcome the bias of indi- vidual models, such a test is repeated ten times and the final Type I, Type II and total accuracy is the average of the results of the ten individual tests.
Simultaneously, we make a comparison for SVM-based metamodel with PCA and without PCA. In Table 9.1, the (without PCA) refers to the SVM-based metamodeling without PCA, while the (with PCA) refers to the SVM-based metamodeling with PCA. In addition, majority-voting- based metamodel and ANN-based metamodel are not used PCA pruning step in this chapter. The computational results are shown in Table 9.1.
Note that the final column is the standard deviation of total accuracy.
9.3 Experimental Analyses 175
Table 9.1. Performance comparisons with different evaluation approaches Models Details Type I(%) Type II(%) Total(%) Std Single model LDA 79.79 81.05 80.22 6.86
LogR 84.17 83.11 83.39 4.82
ANN 81.34 83.78 82.44 7.14
SVM 82.58 84.36 83.58 4.33
Metamodel Majority-voting 83.41 85.16 84.21 5.45 ANN-based 84.24 85.38 84.73 6.34 (Without PCA) SVM-based 84.98 86.22 85.50 3.23 (With PCA) SVM-based 87.35 92.43 89.76 2.89
As can be seen from Table 9.1, we can find the following conclusions.
(1) For type I accuracy, the SVM-based metamodel is the best of all the approaches, followed by the ANN-based metamodel, single Logit analysis, majority-voting-based metamodel, single SVM, single ANN model, and liner discriminant analysis performs the worst. For Type II and total accu- racy, the SVM-based metamodel performs the best in the approaches listed here, followed by the ANN-based metamodel, majority-voting-based metamodel, single SVM, single ANN model, single Logit analysis, and liner discriminant analysis is the worst of all the methods. There are slight difference in superiority order of models between Type I and Type II, but the reason leading to this phenonoma is unknown and it is worth further exploring in later research activities.
(2) In terms of total accuracy, the four metamodels outperform the sin- gle credit scoring model, implying the strong capability and superiority of metamodeling technique in credit scoring.
(3) In the four metamodels, the performance of the SVM-based meta- model with PCA is much better than that of the majority-voting-based metamodel. The main reason is that SVM has a strong learning capability and good generalization capability that can capture subtle relationships be- tween diverse base models. Inversely, the majority voting often ignores the existence of diversity of different base models, as earlier mentioned.
(4) In the four individual models, the single SVM perform the best of the four models, indicating that the SVM possess strong learning and ca- pability. However, we also find that the logit analysis surprisedly outper- forms the linear discriminant analysis and the best artificial neural network from the view of total accuracy. Particularly, for Type I accuracy, the logit analysis is the best of the four individual models. For the example of credit cards, Type I classification is more important than Type II classification. If a bad customer is classified as a good customer, it may lead to direct eco- nomic loss. In this sense, logit analysis model is very promising approach to credit scoring although it is somewhat old.
176 9 Credit Risk Analysis with a SVM-based Metamodeling Ensemble (5) By comparison with the SVM-based metamodels with PCA and without PCA, we are not hard to find that the effects of the pruning step on the classification performance are very obvious in terms of all three meas- urements: Type I, Type II and Total accuracy. This implies that the prun- ing step in the process of metamodeling is effective and necessary step for final performance improvement.
(6) Generally, the proposed SVM-based metamodel perform the best in terms of both Type I accuracy and Type II accuracy. By two-tail paired t- test, the SVM-based metamodel with PCA pruning step achieved better performance than other several models listed here at 10 percent signifi- cance level, implying that the proposed SVM-based metalearning tech- nique is a feasible solution to improve the accuracy of credit scoring with limited data.
Meantime, in order to understand the impact of number of base models on final metamodel performance, several experiments are performed and corresponding results are reported in Fig. 9.5. Note that we use 10 different numbers to carry out this experiment and reported performance is total ac- curacy.
Fig. 9.5. Performance comparisons with different numbers of base models
Fig. 9.5 shows that the SVM-based metamodel is superior to other two metamodeling approaches under all numbers of base models. From this figure, an interesting finding is that there seems to have an optimal selec- tion for the number of base models. In this figure, with the increase of the number of base models, the performance of metamodel is increasingly im- proving until a turning point appeared. In this experiment, 20 base models seem to be a best choice. However, But this finding needs exploring and verifying further in the future research.