A.3 Popup message which appears at the end of the run
10.1 Pseudocode for adding a tree to the ensemble
a best candidate. The algorithm is iterated 20 times, and if a best candidate is found, then that candidate is permanently added to the ensemble. If there is no best candidate tree then the original ensemble is returned. The weights are then updated regardless of whether or not a tree has been added into the ensemble. This is further discussed in section 10.2.3.
In this study a weight is allocated to each instance of data in the training set. The weights represent how easy or difficult an instance of data is for the ensemble to classify. A negative weight implies that the ensemble is not able to correctly classify a particular instance of data. A weight of zero implies that there is a clash amongst the individuals within the ensemble, for example if the output of the ensemble for a particular instance is {class1, class2} then there is a clash. A positive weight implies that the ensemble is able to correctly classify a particular instance of data.
Initially the weights are all set to a value of 0.
The magnitude of the weight denotes how well or how badly the ensemble can classify an instance of data. For instance, a weight of “5” implies that the ensemble is better at classifying an instance of weight “1”. A weight of “-5” denotes an instance which is much harder to classify than one which has a weight of “-1”. Section 10.2.4 describes how the weights are computed.
GP trees are evaluated according to the whether or not it correctly classifies an instance of data. For a GP tree, if an instance of data is correctly classified, then the weight of that particular instance is considered for the tree’s fitness calculation, otherwise it does not contribute to the fitness of the tree. The evaluation of GP trees are computed as follows:
n
X
i=1
f(xi)
where i represents instance i, and n represents the number of instances in the training set. The functionf(xi) is defined as follows:
f(xi) =
0 if GP tree incorrectly evaluated instance i g(xi) if GP tree correctly evaluated instance i
The function g(xi) has to be constructed in such a way that it caters for three cases. The cases are defined as follows.
• First case: the weight is negative.
• Second case: the weight is equal to 0.
• Third case: the weight is positive.
Negative weights have a greater impact on the fitness. A greater fitness implies that a GP tree is a better classifier. Since a negative weight represents an instance of data which is difficult to classify, a GP tree gains a greater fitness when it correctly classifies one of the difficult instances. The more challenging an instance is, the greater its impact on the fitness. This consequently results in the GP algorithm focusing on classifying the difficult instances whilst still taking into account the instances which are considered easy.
In order to justify the choice of the function g(xi) several functions were exam- ined. These functions take into account the magnitude of the weight in such a way as to reward a GP tree if it is able to correctly classify an instance of data. GP trees should receive a constant increment in fitness on those instances of data which are easily classified by the ensemble. This is done so that the GP algorithm can focus on optimising the GP trees on the instances which the ensemble could not correctly classify. Thus, letg(xi) = 1 when an instance of data is correctly classified by a GP tree for which the weight is positive (third case).
Possible functions
for g(xi) Resulting g(xi) value
First case: W = -1 Second case: W = 0
|W| 1 0
|W|+ 1 2 1
|W|+ 2 3 2
|W|+ 3 4 3
Table 10.1: Possible functions forg(xi).
Table 10.1 illustrates four candidate functions for g(xi). For this discussion let the weight for the first case be “-1” and the weight for the second case be “0”. The first function g(xi) = |W| is not suitable because g(xi) is equal to 1 for both the first and third case. Such a function would equally reward an instance of data which was previously incorrectly classified, and a instance which was previously correctly classified. The same argument applies for the functiong(xi) =|W|+ 1, this function is not suitable because correctly classified instances having a weight of 0 are equally rewarded as those instances in the third case. The function g(xi) = |W|+ 2 is suitable because each of the cases are rewarded differently. The first case receives a reward of 3, the second case receives a reward of 2, and the third case always receives a reward of 1. Increasing the value which is added to W is not necessary (for exampleg(xi) =|W|+3) as this will simply add a greater bias towards instances which were previously incorrectly classified, and consequently instances which fall under third case have a smaller impact. The final formulation for the functiong(xi)
CHAPTER 10. GP ENSEMBLE CONSTRUCTION 157 is as follows:
g(xi) =
|W|+ 2 if W <0 (f irst case) 2 if W = 0 (second case) 1 if W >0 (third case)
Table 10.2 illustrates different weight values and the corresponding value for g(xi). From the table, it is clear that correctly classifying instances with negative weights will greater impact the fitness of the trees, than correctly classifying in- stances with positive weights. The more negative the weight, the greater the impact on the fitness. For weights greater than zero, the impact on the fitness is constant regardless of magnitude of the weights. The pseudocode for the evaluation of GP trees is illustrated in algorithm 10.2.
Weight, W Case Value forg(xi)
-3 1 | −3|+ 2 = 5
-2 1 | −2|+ 2 = 4
-1 1 | −1|+ 2 = 3
0 2 |0|+ 2 = 2
1 3 1
2 3 1
3 3 1
Table 10.2: Different weight values and their corresponding value for g(xi).