Non-Linear Neural Ensemble

G- ANFIS S-ANFIS

3.5 Data Fusion

3.5.3 Non-Linear Neural Ensemble

The third data fusion approach used in this study is the NNE. The NNE does not have any statistical basis unlike the bootstrap aggregating and the BMA.

The individual machine learning models are hybridised via black-box operation through a neural network, in contrast to the conventional statistical way of performing data fusion. In a preliminary investigation of this research work, it was found that the stochastic enabled ELM using the whale optimisation algorithm (WOA) had the best performance when it comes to ET0 estimation,

86

Huang and Koo, 2021b). Therefore, the ELM integrated with WOA (WOA- ELM) was utilised to hybridise the MLP, SVM and ANFIS to develop an ensemble. The mathematical expression of the ELM, as proposed by Huang, Zhu and Siew (2006), is shown in Equation (3.11).

𝒀 = ℎ(𝑿)𝜷 (3.11)

where:

Y = output vector

h(X) = sum of output from each hidden neurons fed with input vector β = bias vector

An ELM consists of only one layer of hidden neurons, with all the hyper- parameters initialised at random. The absence of stochastic training in the algorithm of the ELM increases the risk of the model converging to various local optima instead of the desired global optimum. Hence, the WOA algorithm was used to complement this disadvantage and provide a continuous improvement mechanism for the base ELM. This can help to converge the model to the global optimum by increasing the iteration steps in the optimisation algorithm. The reason for selecting the WOA from all the swarm intelligence was because only three parameters (logarithmic spiral constant, number of iterations, distance between whales) need to be adjusted throughout the whole optimisation process. This number was considered low as compared to other swarm-based optimisation algorithms. Besides, the WOA had been tested on 29 different test functions to prove its stability, making it one of the most tested

87

optimisation algorithms for engineering applications. The detailed comparison of WOA with other optimisation algorithms can be found in a review by Johnvictor, et al. (2020). Equation (3.12) to Equation (3.14) proposed by Mirjalili and Lewis (2016) show the steps to search and update the position of the global optimum (target or prey).

𝑿_𝒕+𝟏 = 𝑿_{𝒓𝒂𝒏𝒅}+ 𝑨|𝑪𝑿_{𝒓𝒂𝒏𝒅}− 𝑋_𝑡| (3.12)

𝑨 = 2𝒂𝒓 − 𝒂 (3.13)

𝑪 = 2𝒓 (3.14)

where:

X = position vector of the search agents (whales) A, C = vectors of coefficient

r = randomised vector between 0 and 1

a = shrinking vector from 2 to 0 linearly throughout the iteration process

In order to capture the prey, the search agents would shrink their bubble net, and this will only happen when |A| is sufficiently small. The best position of the global optimum will be updated using Equation (3.15).

𝑿_𝒕+𝟏 = { 𝑿_𝒕^∗− 𝑨|𝑪𝑿_𝒕^∗− 𝑿_𝒕|, 𝑤ℎ𝑒𝑛 𝑝 < 0.5

|𝑿_𝒕^∗− 𝑿_𝒕|𝑒^𝑏𝐿cos(2πL) + 𝑿_𝒕^∗, 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒 (3.15) where:

Xt* = best position vector at iteration t b = shape parameter of the bubble net L = spiral coefficient

p = randomised value between 0 and 1

88

whales in their marine habitat. The piecewise function shown in Equation (3.15) assigns a 50:50 chance for the whales to attack their prey. If the whales are not exploiting for the prey (approaching the global optimum), then they will continue to encircle the prey and wait for the next chance (exploration phase).

By incorporating the WOA to the ELM, it helps to optimise the hyper- parameters within the ELM model so that they are better suited for the hybridisation of multiple models.

To improve the position of the prey after each iteration, a fitness function has to be evaluated to assess the goodness of the current position. Numerous fitness functions have been developed to assess the fitness from different perspectives, including the mean square error (MSE) and Taylor’s skills score (TSS). In this investigation, a fitness function that evaluates the positions from multiple aspects was selected as shown in Equation (3.16). Equation (3.16) was designed such that it had the advantage of a single-objective optimisation (efficient computation) and also multi-objective optimisation (involves competing aspects of a model). It incorporated the competitive nature of different metrics to achieve the optimum balance between accuracy and generalisability.

Fitness = (MAE + RMSE) × (1 – R²) (3.16) where:

MAE = mean absolute error RMSE = root mean square error

89 R² = coefficient of determination

The role of the optimisation algorithm is to minimise the fitness function.

By reaching that objective, as shown in Equation (3.16), the mean absolute error (MAE) and root mean square error (RMSE) would converge to the minimum point, whereas the coefficient of determination (R²) would approach a maximum value of 1. The sum of MAE and RMSE are taken instead of their product due to the fact that both of these metrics measure the deviation of estimated ET0 from the actual ET0 and exist in the same dimension as each other (Chia, Huang and Koo, 2021b). The working mechanism of the WOA-ELM is shown in Figure 3.4.

Figure 3.4: Mechanism of the WOA-ELM

Initialise ELM Hyperparameters Initialise Whales

Population and Position Calculate Fitness Function for Each

Whale

Encircle the Prey (Exploration)

Shrink Bubble Net (Exploitation) Initialise WOA

Parameters

Update Whale Position Update ELM

Hyperparameters

Termination

Maximum Iteration?

Update Fitness Function No

Yes

Update WOA Parameters 50 % Chance

50 % Chance

90

Various training scenarios were designed to assess the robustness of the developed models under different circumstances. Specifically, the developed model should be less data-hungry and adapt well when exposed to different data or deployed in different areas. The three proposed simulating scenarios adopted are explained in this section. Note that the k-fold cross-validation was performed in each of the scenarios. Hence, the training and testing set ratio was set to be 9:1 for each fold, where all data points had the opportunity to be the testing data (in rotation).