190 10 An EP-Based Knowledge Ensemble Model for Credit Risk Analysis
Table 10.1. Identification results of MDA and logit regression models Identification Performance (%) Model
Training data Testing data MDA 86.45 [5.12] 69.50 [8.55]
LogR 89.81 [4.13] 75.82 [6.14]
As shown in Table 10.1, the logit regression analysis slightly outper- forms MDA for both the training data and the testing data. The main rea- son leading to this effect may be that the logit regression can capture some nonlinear patterns hidden in the data.
For ANN models, Table 10.2 summarizes the results of three-layer BPNN according to experiment design.
Table 10.2. Identification results of BPNN models with different designs Identification Performance (%) Learning epochs Hidden nodes
Training data Testing data 100 6 83.54 [7.43] 68.98 [8.12]
12 84.06 [7.12] 67.35 [7.98]
18 89.01 [6.45] 71.02 [8.21]
24 86.32 [7.65] 70.14 [8.72]
32 80.45 [8.33] 64.56 [8.65]
500 6 82.29 [7.21] 69.43 [8.27]
12 85.64 [6.98] 70.16 [7.32]
18 86.09 [7.32] 71.23 [8.01]
24 88.27 [6.89] 74.80 [7.22]
32 84.56 [7.54] 72.12 [8.11]
1000 6 87.54 [7.76] 70.12 [8.09]
12 89.01 [7.13] 72.45 [7.32]
18 90.53 [6.65] 75.45 [6.86]
24 93.42 [5.89] 77.67 [6.03]
32 91.32 [6.56] 74.91 [7.12]
2000 6 85.43 [7.76] 72.23 [8.09]
12 87.56 [7.22] 70.34 [7.76]
18 90.38 [7.67] 72.12 [8.45]
24 89.59 [8.09] 71.50 [8.87]
32 86.78 [7.81] 69.21 [8.02]
From Table 10.2, we have the following finding.
(1) The identification performance of training data tends to be higher as the learning epoch increases. The main reason is illustrated by Hornik et al. (1989), i.e., To make a three-layer BPNN approximate any continuous function arbitrarily well, a sufficient amount of middle-layer units must be given.
10.4 Experiment Results 191 (2) The performance of training data is consistently better than that of the testing data. The main reason is that the training performance is based on in-sample estimation and the testing performance is based on out-of- sample generalization.
(3) The best prediction accuracy for the testing data was found when the epoch was 1000 and the number of hidden nodes was 24. The identifica- tion accuracy of the testing data turned out to be 77.67%, and that of the training data was 93.42%.
For SVM, there are two parameters, the kernel parameter σ and margin parameters C that need to be tuned. First, this chapter uses two kernel functions including the Gaussian radial basis function and the polynomial function. The polynomial function, however, takes a longer time in the training of SVM and provides worse results than the Gaussian radial basis function in preliminary test. Thus, this chapter uses the Gaussian radial function as the kernel function of SVM.
The chapter compares the identification performance with respect to various kernel and margin parameters. Table 10.3 presents the identifica- tion performance of SVM with various parameters according to the ex- periment design.
From Table 10.3, the best identification performance of the testing data is recorded when kernel parameter σ2 is 10 and margin C is 75. Detailed speaking, for SVM model, too small a value for C caused under-fit the training data while too large a value of C caused over-fit the training data.
It can be observed that the identification performance on the training data increases with C in this chapter. The identification performance on the testing data increases when C increase from 10 to 75 but decreases when C is 100. Similarly, a small value of σ2 would over-fit the training data while a large value of σ2 would under-fit the training data. We can find the iden- tification performance on the training data and the testing data increases when σ2 increase from 1 to 10 but decreases when σ2 increase from 10 to 100. These results partly support the conclusions of Tay and Cao (2001) and Kim (2003).
10.4.2 Identification Performance of the Knowledge Ensemble To formulate the knowledge ensemble models, two main strategies, major- ity voting and evolutionary programming ensemble strategy, are used.
With four individual models, we can construct eleven ensemble models.
Table 10.4 summarizes the results of different ensemble models. Note that the results reported in Table 10.4 are based on the testing data.
192 10 An EP-Based Knowledge Ensemble Model for Credit Risk Analysis
Table 10.3. Identification performance of SVM with different parameters Identification Performance (%)
C σ2
Training data Testing data 10 1 83.52 [5.67] 67.09 [6.13]
5 87.45 [6.12] 72.07 [6.66]
10 89.87 [6.54] 74.21 [6.89]
25 88.56 [6.33] 73.32 [6.76]
50 84.58 [5.98] 69.11 [6.34]
75 87.43 [6.57] 71.78 [7.87]
100 82.43 [7.01] 67.56 [7.65]
25 1 84.58 [6.56] 72.09 [6.34]
5 81.58 [7.01] 66.21 [7.33]
10 89.59 [6.52] 70.06 [7.12]
25 90.08 [6.11] 75.43 [6.65]
50 92.87 [7.04] 77.08 [7.56]
75 89.65 [6.25] 76.23 [6.71]
100 87.43 [5.92] 74.29 [6.23]
50 1 85.59 [6.65] 72.21 [6.89]
5 88.02 [6.67] 73.10 [6.92]
10 89.98 [6.83] 74.32 [7.03]
25 84.78 [6.95] 73.67 [7.32]
50 89.65 [6.32] 76.35 [6.65]
75 87.08 [6.47] 72.24 [6.67]
100 85.12 [6.81] 71.98 [7.06]
75 1 87.59 [6.12] 74.21 [6.45]
5 89.95 [6.56] 75.43 [6.78]
10 95.41 [6.09] 80.27 [6.17]
25 92.32 [6.48] 78.09 [6.75]
50 93.43 [7.65] 77.61 [7.89]
75 89.56 [6.13] 77.23 [6.69]
100 88.41 [6.71] 74.92 [7.00]
100 1 88.12 [6.32] 75.47 [6.87]
5 90.04 [6.87] 78.01 [7.21]
10 93.35 [6.37] 79.17 [6.76]
25 94.21 [6.98] 80.01 [6.29]
50 91.57 [7.33] 78.23 [7.56]
75 89.48 [6.56] 75.98 [7.12]
100 88.54 [6.77] 74.56 [6.98]
As can be seen from Table 10.4, it is not hard to find the following (1) Generally speaking, the EP-based ensemble strategy is better than the
majority voting based ensemble strategy, revealing that the proposed EP-based ensemble methodology can give a promising result for busi- ness risk identification.
10.4 Experiment Results 193 (2) The identification accuracy increases as the number of ensemble
member increase. The main reason is that different models may con- tain different useful information. However, the number of ensemble member does not satisfy the principle of “the more, the better”. For example, ensemble model E6 performs better than the ensemble mod- els E7 and E8 for majority voting strategy while the ensemble model E11 is also worse than the ensemble model E10 for EP-based ensem- ble strategy.
Table 10.4. Identification performance of different knowledge ensemble models Ensemble Strategy Ensemble
model Ensemble member
Majority voting EP E1 MDA+LogR 74.23 [6.12] 76.01 [5.67]
E2 MDA+ANN 75.54 [6.74] 77.78 [6.19]
E3 MDA+SVM 77.91 [5.98] 80.45 [6.03]
E4 LogR+ANN 78.76 [5.65] 79.87 [5.48]
E5 LogR+SVM 79.89 [6.33] 81.25 [5.92]
E6 ANN+SVM 81.87 [5.76] 84.87 [5.16]
E7 MDA+LogR+ANN 78.98 [5.87] 80.34 [6.42]
E8 MDA+LogR+SVM 80.67 [5.54] 83.56 [5.77]
E9 MDA+ANN+SVM 82.89 [6.12] 85.05 [6.22]
E10 LogR+ANN+SVM 85.01 [5.79] 88.09 [5.56]
E11 MDA+LogR+ANN+SVM 85.35 [5.51] 86.08 [6.78]
10.4.3 Identification Performance Comparisons
Table 10.5 compares the best identification performance of four individual models (MDA, LogR, BPNN, and SVM) and two best ensemble models (E11 for majority voting based ensemble and E10 for EP-based ensemble) in terms of the training data and testing data. Similar to the previous re- sults, the values in bracket are the standard deviation of 20 tests.
From Table 10.5, several important conclusions can be drawn. First of all, the ensemble models perform better than the individual models. Sec- ond, of the four individual models, the SVM model is the best, followed by BPNN, LogR and MDA, implying that the intelligent models outperform the traditional statistical models. Third, of the two knowledge ensemble models, the performance of the EP-based ensemble models is better than that of the majority-voting-based ensemble models, implying that the intel- ligent knowledge ensemble model can generate good identification per- formance.
194 10 An EP-Based Knowledge Ensemble Model for Credit Risk Analysis
Table 10.5. Identification performance comparisons with different models Individual Models Ensemble Models Data
MDA LogR BPNN SVM Majority Voting EP Training
data 86.45
[5.12] 89.81
[4.13] 93.42
[5.89] 95.41
[6.09] 95.71
[5.23] 98.89 [5.34]
Testing data
69.50 [8.55]
75.82 [6.14]
77.67 [6.03]
80.27 [6.17]
85.35 [5.51]
88.09 [5.56]
In addition, we conducted McNemar test to examine whether the intelli- gent knowledge ensemble model significantly outperformed the other sev- eral models listed in this chapter. As a nonparametric test for two related samples, it is particularly useful for before-after measurement of the same subjects (Kim, 2003; Cooper and Emory, 1995). Table 10.6 shows the re- sults of the McNemar test to statistically compare the identification per- formance for the testing data among six models.
Table 10.6. McNemar values (p values) for performance pairwise comparisons
Models Majority SVM BPNN LogR MDA
EP Ensemble 1.696 (0.128) 3.084 (0.092)4.213 (0.059)5.788 (0.035)7.035 (0.009) Majority 1.562 (0.143)2.182 (0.948)5.127 (0.049)6.241 (0.034) SVM 1.342 (0.189)1.098 (0.235)3.316 (0.065)
BPNN 0.972 (0.154)1.892 (0.102)
LogR 0.616 (0.412)
As revealed in Table 10.6, the EP-based knowledge ensemble model outperforms four individual models at 10% significance level. Particularly, the EP-based knowledge ensemble model is better than SVM and BPNN at 10% level, logit regression analysis model at 5% level, and MDA at 1%
significance level, respectively. However, the EP-based knowledge en- semble model does not significantly outperform majority voting based knowledge ensemble model. Similarly, the majority voting based knowl- edge ensemble model outperforms the BPNN and two statistical models (LogR and MDA) at 10% and 5% significance level. Furthermore, the SVM is better than the MDA at 5% significance level. However, the SVM does not outperform the BPNN and LogR, which is consistent with Kim (2003). In addition, Table 9.6 also shows that the identification perform- ance among BPNN, Logit regression analysis and MDA do not signifi- cantly differ each other.