Chapter 8: Comparative investigations for reliable conclusions
8.3 Robustness test two: Prediction efficacy of the models in incremental forecasting
8.3 Robustness test two: Prediction efficacy of the models in
the Multivariate LSTM, resulting in the percentage changes being negative in most ten-by-ten increments. Table E.1.2 shows the descriptive summary statistics for the percentage difference of Univariate-LSTM from the Multivariate-LSTM. The average of -0.2181 and the median of -0.1591 are negative, which confirms that, on average, the RMSE of Univariate LSTM is smaller than the RMSE of the Multivariate LSTM. In addition, Figure 8.1 shows the RMSE results of each assessed model presented in a comparative box plot which depicts the five- number summary (maximum, quartile three, median, quartile one and minimum). The box plot of Univariate LSTM is positioned closest to the horizontal axis (cross at zero) and followed by the Multivariate LSTM, confirming the predictive superiority of Univariate LSTM, and then the Multivariate LSTM as the second best. The two LSTM models' box plots are positioned significantly lower than HEWS Models 1-2 and ARIMA Models 1-2. These findings based on the RMSE criterion confirm the predictive superiority of Univariate LSTM over the other assessed models when applied to the NZX 50 Index forecasting. It is also evident that both LSTM models significantly supersede the conventional statistical models.
Similarly, the evidence in Table E2.1 (Table E3.1) shows the MAE (MAPE) results of each model evaluated in ten-by-ten incremental prediction slots (from 10 to 650). For example, Table E2.1 (Table E3.1) show at the first ten incremental prediction slot, the lowest MAE (MAPE) are 1.1830 (0.0004) from the Univariate LSTM, followed by the next lowest MAE (MAPE) are 2.3663 (0.0008) from the Multivariate LSTM. Thus, it is evident that in the first ten incremental prediction slot, the Univariate LSTM is the best, followed by the Multivariate LSTM, the next best. Since the MAE (MAPE) results in ten-by-ten incremental prediction slots are the lowest for the LSTM models, it is possible to conclude that the two best prediction models are the Univariate and Multivariate LSTM models.
Closer scrutiny reveals that except for the incremental prediction slots between 310 to 470 in MAE analysis (presented in Table E.2.1), the MAE of the Univariate LSTM is
significantly smaller than the MAE of the Multivariate LSTM, resulting in the percentage changes to be negative in ten-by-ten increments. Table E.2.2 shows the descriptive summary statistics for the percentage difference of Univariate-LSTM from the Multivariate-LSTM. The average (-0.1648) and median (-0.0979) are negative, suggesting that, on average, the MAE of Univariate LSTM is smaller than the MAE of the Multivariate LSTM. Additionally, Figure 8.2, which portrays MAE results for each assessed model presented in a comparative box plot, which depicts the five-number summary, confirms the forecasting superiority of Univariate LSTM amongst the other tested models as the Univariate LSTM’s box plot is positioned closest to the horizontal zero axis, ensuring it has the lowest MAE. Therefore, the MAE criterion too confirms the forecasting superiority of Univariate LSTM over all other assessed models when applied to the NZX 50 Index forecasting.
Likewise, the MAPE analysis in Table E.3.1 shows that the LSTM models significantly outperform the other evaluated models. A closer examination uncovers that except for the incremental prediction slots between 320 to 450, the MAPE of the Univariate LSTM is smaller than the MAPE of the Multivariate LSTM, resulting in the percentage changes to be negative in most ten-by-ten increments. Table E.3.2 shows the descriptive summary statistics for the percentage change of Univariate LSTM from the Multivariate LSTM. The average (-0.1803) and median (-0.1085) are negative, suggesting that, on average, the MAPE of Univariate LSTM is smaller than the MAE of the Multivariate LSTM. Furthermore, Figure 8.3, which portrays MAPE results for each assessed model presented in a comparative box plot, confirms the forecasting superiority of Univariate LSTM amongst the other tested models as the Univariate LSTM’s box plot is positioned closest to the horizontal zero axis in compassion to the other evaluated models and ensures it has the lowest MAPE. Finally, the above findings based on the MAPE criterion emphasise the predictive superiority of Univariate LSTM over the other assessed models when applied to the NZX 50 Index forecasting.
In summary, all the above findings confirm the predictive superiority of the Univariate LSTM, followed by the Multivariate LSTM as the next-best forecasting model for NZX 50 Index prediction. Figures 8.1–8.3 clearly show that the box plots of Univariate and Multivariate LSTM models portraying the error statistics (RMSE, MAE and MAPE) are positioned significantly lower than the box plots of error statistics for ARIMA and HWES models. This evidence confirms that LSTM models consistently outperformed the ARIMA and HWES forecasting models when applied to the NZX 50 Index forecasting. This evidence validates the predictive dominance of LSTM models in comparison to the conventional statistical models.
Finally, these findings corroborate my conclusions in the robustness test one presented in Section 8.2.
5 10 15 20
HWES Mo del 1
HWES Mo del 2
ARIMA Mo del 1
ARIMA Mo del 2
Univariate LSTM Multivariate LS
TM RMSE
Figure 8.1: RMSE results of each model tested in incremental forecasting
0 2 4 6 8 10 12 14 16
HWES Mo del 1
HWES Mo del 2
ARIMA Model 1
ARIMA Model 2
Univariate LSTM Mu
ltivariate LSTM MAE
Figure 8.2: MAE results of each model tested in incremental forecasting
.000 .001 .002 .003 .004 .005
HWES Mo del 1
HWES Mo del 2
ARIMA Model 1
ARIMA Model 2
Univariate LSTM
Multivariate LSTM MAPE
Figure 8.3: MAPE results of each model tested in incremental forecasting
8.4 Robustness test three: Prediction efficacy of the models when tested on the ASX 200 Index
The identical models I reformulated and implemented to forecast the NZX 50 Index are applied to the Australian stock market index. This investigation enables me to ascertain whether the reformulated models predominantly designed to predict the NZX 50 index can be effectively and efficiently used to analyse different time series. This study also allows me to evaluate whether the conclusions are consistent when the revised models are applied to the NZX 50 Index and ASX 200 Index. Since S&P/ASX 200 (ASX 200) Index is Australia's preeminent benchmark index [20], I use ASX 200 Index in this investigation.
The ideal configurations of ARIMA, HWES, and LSTM models are constructed when the standard models have been applied to analyse the NZX 50 Index prediction. The identical models designed for the NZX 50 Index prediction are subsequently employed to analyse the ASX 200 Index movements to determine how effective these models are in different time series. For example, the univariate LSTM model with identical hyperparameters (i.e., optimum batch size of 500, epoch size of 740, identical architecture with the same learning rate) is used for the ASX 200 Index to determine the model's forecasting effectiveness in a different time series to the NZX 50 Index. For the model development and application, approximately 70 percent of the data are used for algorithm training. The rest are used to test the reformulated models' predictive capability. The data segregation used for NZX 50 Index is applied to ASX 200 Index for effective comparisons.
Table 8.5 shows the prediction results of the reformulated models when the identical models are applied to the NZX 50 Index and the ASX 200 Index.
Table 8.5: Comparison of the tested models applied to ASX 200 Index and NZX 50 Index Comparison of the tested models applied to the ASX 200 Index and NZX 50 Index
ASX 200 Index NZX 50 Index
Univariate-LSTM
Optimum
epochs 740 740
Optimum
Batch 500 500
RMSE 2.9468 1.6787
MAE 2.4037 1.5929
MAPE 0.0004310 0.0004804
Multivariate-LSTM
Optimum
epochs 702 702
Optimum
Batch 610 610
RMSE 16.7130 2.0405
MAE 13.5666 1.8254
MAPE 0.0024027 0.0005547
Model one: ARIMA (1, 1, 0) plus intercept
RMSE 48.3677 18.5389
MAE 36.1034 13.7985
MAPE 0.0066955 0.0042029
Model two: ARIMA (0, 1, 1) plus intercept
RMSE 48.4003 18.6817
MAE 36.0483 13.9093
MAPE 0.0066858 0.0042358
Model one: HWES (alpha, beta) on the chosen time series
RMSE 48.7719 18.5813
MAE 36.4678 13.7962
MAPE 0.0067628 0.0042044
Model two: HWES (alpha) on the differenced of the chosen time series
RMSE 48.7507 18.5826
MAE 36.4221 13.7967
MAPE 0.0067543 0.0042045
Data Split:70 percent for model training and 30 percent for model testing
A close inspection of Table 8.5 revealed that Univariate LSTM error statistics are the lowest, followed by Multivariate LSTM, the next lowest error statistics in both investigations.
For example, the Univariate LSTM’s error statistics are RMSENZX-50 of 1.6789, MAENZX-50 of 1.5929, MAPENZX-50 of 0.0004804, RMSEASX-200 of 2.9468, MAEASX-200 of 2.4037 and MAPEASX-200 of 0.0004310. Similarly, Multivariate LSTM’s error statistics are RMSENZX-50
of 2.0405, MAENZX-50 of 1.8254, MAPENZX-50 of 0.0005547, RMSEASX-200 of 16.7130,
MAEASX-200 of 13.5666 and MAPEASX-200 of 0.0024027. These were the two lowest error statistics when all the models were assessed on the NZX 50 Index and ASX 200 Index. These findings strongly endorse that Univariate LSTM is the best and most solid forecasting model, followed by Multivariate LSTM as the second-best. These conclusions are robust and valid for both investigations. The superiority of Univariate LSTM is proven in many testing procedures in my thesis; thus, these findings contribute to the direction of statistical generalisation.
Table 8.6 shows the order of model preference based on RMSE, MAE, and MAPE when the redesigned models were applied to the NZX 50 Index. Similarly, Table 8.7 shows the order of the model preference results when the identical models were applied to the ASX 200 Index.
Table 8.6: Order of model preference when the analysis is conducted on the NZX 50 Index
Order of model preference based on RMSE
Order of model preference based on MAE
Order of model preference based on MAPE
Univariate-LSTM 1 1 1
Multivariate-LSTM 2 2 2
Model one: ARIMA (1, 1, 0) plus
intercept 3 5 3
Model two: ARIMA (0, 1, 1) plus
intercept 6 6 6
Model one: HWES (alpha, beta) on
the NZX 50 Index 4 3 4
Model two: HWES (alpha, beta) on
the difference of the NZX 50 Index 5 4 5
The preference orders in Tables 8.6 and 8.7 are determined based on the prediction performance evaluation measures (RMSE, MAE, and MAPE) presented in Table 8.5.
Table 8.7: Order of model preference when the analysis is conducted on the ASX 200 Index
Order of model preference based on RMSE
Order of model preference based on MAE
Order of model preference based on MAPE
Univariate-LSTM 1 1 1
Multivariate-LSTM 2 2 2
Model one: ARIMA (1, 1, 0) plus
intercept 3 4 4
Model two: ARIMA (0, 1, 1) plus
intercept 4 3 3
Model one: HWES (alpha, beta) on
the NZX 50 Index 6 6 6
Model two: HWES (alpha, beta) on
the difference of the NZX 50 Index 5 5 5
The evidence in Table 8.5 reveals that Univariate LSTM is the best performing prediction model, followed by the Multivariate LSTM is the next best. The conclusions are based on the lowest error statistics inherent in the LSTM models. More precisely, the RMSE of Univariate LSTM, when applied to the NZX 50 Index, is 1.6787 and RMSE, when applied to ASX 200 Index, is 2.9468. MAE of Univariate LSTM, when applied to the NZX 50 Index, is 1.5929 and MAE, when applied to ASX 200 Index, is 2.4037. MAPE of Univariate LSTM, when applied to the NZX 50 Index, is 0.0004804 and MAPE when applied to ASX 200 Index is 0.0004310.
Likewise, the RMSE of Multivariate LSTM, when applied to the NZX 50 Index, is 2.0405 and RMSE, when applied to ASX 200 Index, is 16.7130. The MAE of Multivariate LSTM, when applied to the NZX 50 Index, is 1.8254 and MAE, when applied to ASX 200 Index, is 13.5666.
Similarly, The MAPE of Multivariate LSTM, when applied to the NZX 50 Index, is 0.0005547 and MAPE, when applied to ASX 200 Index, is 0.0024027.
These findings confirm that Univariate LSTM has the lowest error statistics and is valid for each performance evaluation criterion; thus, it should be recognised as the number “one”
prediction model when the model was applied to the NZX 50 Index and the ASX 200 Index.
The second-lowest error statistics are with the Multivariate LSTM model. This conclusion is valid for each performance evaluation criterion; thus, Multivariate LSTM must be the number
“two” prediction model. Because of the consistency of the findings under different testing processes, these conclusions provide justified evidence for statistical generalisation.
However, a consistent and unanimous order of preferences does not exist with ARIMA and HWES models. For example, based on the RMSE criterion, ARIMA model 1 is considered the third-best model to predict NZX 50 Index. However, the MAE criterion concludes that ARIMA model 1 is the fifth-best to predict NZX 50 Index. The MAPE criterion concludes that ARIMA model 1 is the third-best model to predict NZX 50 Index. Similarly, the RMSE criterion concludes that ARIMA model 1 is the third best when applied to ASX 200 Index prediction model. ARIMA model 1 is the fourth-best when applied to ASX 200 Index with MAE and MAPE criteria.
In summary, the results in Tables 8.6 and 8.7 show the predictive strength of the Univariate and Multivariate LSTM models as they consistently reach the first and the second positions in the order of preference when the models were applied to the NZX 50 Index [16]
and the ASX 200 Index [20]. However, our findings reveal that the model preference order on the predictive precision is inconsistent for ARIMA and HWES.
The conclusions in Section 8.4 reinforce the findings I have arrived at in Section 8.2 Robustness Tests one and Section 8.3 Robustness Tests two.