3.3 Experimental Study
3.3.4 Experimental Results
In our test, the main setting for each method is as Table 3.2. The program are run under the environment of matlab 7.1 with LS-SVM toolbox pro- vided by Suykens et al. (2002) on an IBM compatible PC with Pentium IV CPU running at 2.5 GHz with 1GB RAM.
Table 3.2. Parameter settings for DS, GA, GS and DOE Method Parameters Setting
DS Initial point: randomly select from (1:10, 1:10); Mesh_tol: 0.001;
Fun_tol: 0.001; Max_Iter: 200; Max_fun: 4000; X_tol: 1e-6; Fun_tol:
1e-6
GA Population Size: 50; Crossover Fraction: 0.8; v_Generation = 50;
v_Time: Inf; v_FitLimit: Inf; v_StallGen: 50; StallTimeLimit: 20 GS C_min: -2, C_max: 10, Sigma_min: -10, Sigma_max: 2, Delta_C: 0.5
Delta_Sigma: 0.5
DOE C_min = -10, C_max = 2, Sigma_mi = Sigma_max = Max_iter = 6 Let the number of creditworthy cases classified as good be GG and clas- sified as bad with GB, denote the number of default cases classified as good with BG and as bad with BB. Three common used evaluation criteria measure the efficiency of the classification has the following three criteria:
Sensitivity (Se) = ×100% +GB
GG
GG (3.11)
Specificity (Sp) = ×100% + BB
BG
BB (3.12)
Overall accuracy (OA) = ×100% +
+ +
+
BB BG GB GG
BB
GG (3.13)
In the experiments, four parameter selection methods (DS, GA, GS and DOE) used identical training and testing sets with 5-fold cross validation.
The average classification accuracy for one run of 5-fold cross validation
3.3 Experimental Study 51 of the four methods and computational time for German and Australian dataset are shown in Table 3.3 and Table 3.4 respectively.
Table 3.3. Evaluation results of four methods for LSSVM on German dataset Method OA (%) Se (%) Sp(%) CPU time (s)
GSlssvm 77.3 91.0 45.3 2978.7
DOElssvm 77.0 92.1 41.6 613.6
GAlssvm 76.9 90.0 46.3 683.9
DSlssvm 77.6 90.3 47.9 1070.4
Table 3.4. Evaluation results of four methods for LSSVM on Australian dataset Method OA (%) Se (%) Sp(%) CPU time (s)
GSlssvm 87.2 88.5 86.3 1460.1
DOElssvm 87.0 87.2 86.8 259.7
GAlssvm 87.1 89.1 85.5 289.8
DSlssvm 87.2 89.7 85.3 302.3
From the above results appeared in Tables 3.3-3.4, we can find that for the average classification accuracy, DSlssvm obtained the best performance of all these four methods with the setting shown in Table 3.2. The compu- tational performance of these four methods is relative to their parameters setting and we choose the parameters for stop criteria by trial and error for GAlssvm and DSlssvm. DSlssvm and GSlssvm are the easiest methods to be im- plemented, while DSlssvm has better efficiency. Actually, the DOElssvm
methods can be viewed as a special case of DSlssvm if we set a proper pat- tern and stop criteria. Fig. 3.2 shows the average classification accuracy with 5-fold cross validation from grid search for LSSVM credit scoring model in the initial search area. The final result of DSlssvm or GSlssvm is re- stricted by the predetermined value range of the parameters. If we happen to set the initial range for C and σ to be [2-2, 20], [2-10, 2-6], the best overall accuracy from these two methods will not exceed 71.0%. Figs. 3.3-3.5 show the results of these four methods under different initial space setting on Australia data set. The size of each space is the same for all the 10 groups of space setting, only the position is different. We can see that DSlssvm has better overall accuracy and better robustness when compared with other three methods. Figs. 3.4 and 3.5 indicate that the reason why Dslssvm has good OA may be that it has good sensitivity.
52 3 Credit Risk Evaluation Using SVM with Direct Search
Fig. 3.2. Performance of each parameter pair in initial range
Fig. 3.3. Sensitivity analysis of OA on initial space setting
3.3 Experimental Study 53
Fig. 3.4. Sensitivity analysis of Se on initial space setting
Fig. 3.5. Sensitivity analysis of Sp on initial space setting
For further illustration, the performance of classification of direct search for LS-SVM credit evaluation model is also compared with other six com- monly used classifiers with 5-fold cross validation: Linear Discriminant Analysis (LDA), Quadratic Discriminant Analysis (QDA), Logistic regres- sion (LogR), Linear Programming (LP), Decision tree (DT) and k-Nearest neighbour Classifier (k-NN). The compared results of different classifiers are reported in Table 3.5 for the two datasets.
54 3 Credit Risk Evaluation Using SVM with Direct Search
Table 3.5. Performance comparisons of different classifiers
German Dataset Australian Dataset Methods
OA (%) Se(%) Sp(%) OA(%) Se(%) Sp(%)
LDA 72.00 72.43 71.00 85.80 92.50 80.42
QDA 68.00 67.71 68.67 80.59 66.48 91.91
LogR 76.40 88.14 49.00 86.53 88.28 85.12
DT 71.80 79.57 53.67 82.18 80.41 83.56
k-NN 69.90 89.85 23.33 69.13 54.39 80.94 DSlssvm 77.10 88.86 49.67 86.96 89.25 85.12
As can be seen from Table 3.5, we can find the following several con- clusions:
(1) For overall accuracy, the DSlssvm performs the best of all listed ap- proaches, followed by LogR and LDA, QDA is the worst. The perform- ance of other three methods is not stable on these two data sets. Classifica- tion performance of all these methods is also relative to the samples, so it may be difficult to choose a model which always performs the best. How- ever, the test of the models based on the same training samples and testing samples can demonstrate their performance to some extent.
(2) Although Dslssvm has the best overall accuracy when compared with other five methods, the sensitivity or specificity has not obviously superi- ority over other methods. However, we can control the specificity through modifying the objective function in (3.5) into the following form:
( )
= + 1∑
=11 2+ 2∑
=21 2,
, 2
, 1 ,
min T Nk k Nk k
b
w J wbξ w w C ξ C ξ
ξ (3.14)
where N1 and N2 are the number of cases in class 1 and class 2. It shows that this model has good flexibility and it is worth exploring further.
(3) From the general view, DSlssvm dominates the other six classifiers, revealing that it is an effective tool for credit risk evaluation.