Chapter 8: Comparative investigations for reliable conclusions
8.2 Robustness test one: Prediction efficacy of the models in varied samples
The NZX 50 Index [16] forecasting results using ARIMA [Model 1 (1,1,0) plus intercept], ARIMA [Model 2 (0,1,1) plus intercept], HWES [Model 1 (alpha, beta) on the undifferenced], HWES [Model 2 (alpha) on the differenced], Univariate LSTM and Multivariate LSTM are produced for each tested sample period in this thesis. The main results are presented in Tables 6.4, 6.5, 7.1, and 7.2, and the detailed prediction results are given in Appendices B-D.
Table 8.1 replicates the key error summary statistics (RMSE, MAE and MAPE) presented in Tables 6.4, 6.5, 7.1, and 7.2. Their detailed statistics are produced in Appendices (Appendix B, Appendix C and Appendix D). The evidence in Table 8.1 reveals that Univariate LSTM is the best performing prediction model for each sample period tested, followed by the Multivariate LSTM is the next best. This conclusion is based on the principle that all the error measures (RMSE, MAE, and MAPE) of Univariate LSTM are the lowest among all the models for each sample period assessed, followed by the Multivariate LSTM’s error measures are the second-lowest. This conclusion is valid and consistent for every sample period evaluated in this study. Table 8.2 illustrates the two most efficient models' percentage changes of each error
criterion (RMSE, MAE and MAPE). Univariate LSTM was adjudged as the best, followed by the Multivariate LSTM as the second best. Thus, the percentage changes in Table 8.2 are from Multivariate LSTM to Univariate LSTM.
For example, the RMSE of Univariate LSTM for period 1 is 1.6787, whilst the Multivariate LSTM’s RMSE for the same period is 2.0405, which confirms that the RMSE of Univariate LSTM is 17.7% smaller than the RMSE of Multivariate LSTM during sample period 1. Similarly, the Univariate LSTM’s RMSE for period 2 is 1.4494, whilst Multivariate LSTM’s RMSE for the same period is 2.0208, which confirms that the RMSE of Univariate LSTM is 28.3% smaller than the RMSE of Multivariate LSTM during sample period 2.
Likewise, the RMSE of Univariate LSTM for period 3 is 2.83351, whilst Multivariate LSTM’s RMSE for the same period is 4.8205, which confirms that the RMSE of the Univariate LSTM is 41.2% smaller than the RMSE of Multivariate LSTM during sample period 3.
The MAE (MAPE) of Univariate LSTM for period 1 is 1.5929 (0.0004804), whilst Multivariate LSTM’s MAE (MAPE) for the same period is 1.8254 (0.0005547). These results confirm that the MAE (MAPE) of Univariate LSTM is 12.7% (13.4%) smaller than the MAE (MAPE) of Multivariate LSTM during sample period 1. Similarly, the MAE (MAPE) of Univariate LSTM for period 2 is 1.3311 (0.0004007), while Multivariate LSTM’s MAE (MAPE) for period 2 is 1.8028 (0.0005510). These findings validate that the MAE (MAPE) of Univariate LSTM is 26.2% (27.3%) smaller than the MAE (MAPE) of the Multivariate LSTM during sample period 2. Sample period 3 is unique as this sample includes data during the COVID-19 pandemic. The MAE (MAPE) of Univariate LSTM for period 3 is 2.6299 (0.0006381), while Multivariate LSTM’s MAE (MAPE) for period 3 is 4.0431 (0.0009179).
These findings confirm that the MAE (MAPE) of Univariate LSTM is 35% (30.5%) smaller than the MAE (MAPE) of the Multivariate LSTM during sample period 3.
Table 8.1: Comparisons of RMSE, MAE, and MAPE of the assessed models on three sample periods
Tested Model
Sample period one: Approximate upward trend (2009 to 2017)
Sample period two: Nonlinear trend excluding COVID-19
impact (2007 to 2017)
Sample period three: Nonlinear trend including COVID-19
impact (2007 to 2020)
RMSE MAE MAPE RMSE MAE MAPE RMSE MAE MAPE
ARIMA (1, 1, 0) plus intercept 18.5389 13.7985 0.0042029 17.7920 13.3062 0.0041311 36.1768 23.5530 0.0055218 ARIMA (0, 1, 1) plus intercept 18.6817 13.9093 0.0042358 17.9429 13.4525 0.0041770 36.5775 23.8968 0.0055894 HWES (alpha, beta) on the
undifferenced NZX 50 Index 18.5813 13.7962 0.0042044 17.9428 13.2982 0.0041304 36.5848 23.7588 0.0055672 HWES (alpha) on the differenced
NZX 50 Index 18.5826 13.7967 0.0042045 17.9416 13.2973 0.0041302 36.5850 23.7598 0.0055674 Univariate-LSTM 1.6787 1.5929 0.0004804 1.4494 1.3311 0.0004007 2.8351 2.6299 0.0006381 Multivariate-LSTM 2.0405 1.8254 0.0005547 2.0208 1.8028 0.0005510 4.8205 4.0431 0.0009179
Standard statistics for all the models tested
Sample period one Sample period two Sample period three
Total number of observations 2173 2672 3401
Training data set 1521 1870 2381
Testing data set 652 802 1020
When all the sample periods are collectively evaluated, the RMSE for the Univariate LSTM is, on average, 29% lower than the RMSE for Multivariate LSTM (the next best model).
Similarly, on average, MAE (MAPE) for the Univariate LSTM is 25% (24%) below the MAE (MAPE) for the Multivariate LSTM. The above findings reinforce my conclusion about the predictive superiority of the Univariate LSTM followed by the Multivariate LSTM is the second best.
Table 8.3 shows the comparative error statistics (RMSE, MAE and MAPE) for the mean (average) of DL models (viz, Univariate LSTM and Multivariate LSTM) and the mean (average) of the statistical models (namely HWES Models 1-2 and ARIMA Models 1-2) for each sample period tested. Table 8.4 shows the percentage changes of each error criterion (RMSE, MAE and MAPE) from the mean (average) statistical model to the average DL model.
Table 8.3 discloses that the mean (average) of the DL models significantly outperformed the mean (average) of statistical models for each sample period tested. For example, the RMSE of the mean (average) DL model for period 1 is 1.8596, whilst the RMSE of the mean (average) statistical model for the same period is 18.5954, which confirms that the RMSE of the mean (average) DL model is 90.0% smaller than the RMSE of the mean (average) statistical model.
Similarly, the RMSE of the average DL model for period 2 is 1.7351, whilst the RMSE of the average statistical model for the same period is 17.9048, which confirms that the RMSE of the average DL model is 90.3% smaller than the RMSE of the average statistical model. Likewise, the RMSE of the average DL model for period 3 is 3.8278, whilst the RMSE of the average statistical model for the same period is 36.4811, which confirms that the RMSE of the average DL model is 89.5% smaller than the RMSE of the average statistical model.
Table 8.3 also shows that the MAE (MAPE) of the mean (average) DL model for period 1 is 1.7092 (0.0005176), whilst the mean (average) statistical model’s MAE (MAPE) for the
same period is 13.8252 (0.0042119). These results confirm that the MAE (MAPE) of the average DL model is 87.6% (87.7%) smaller than the MAE (MAPE) of the average statistical model. Similarly, the MAE (MAPE) of the average DL model for period 2 is 1.5669 (0.0004758), whilst the average statistical model’s MAE (MAPE) for the same period is 13.3385 (0.0041422). These results confirm that the MAE (MAPE) of the average DL model is 88.3% (88.5%) smaller than the MAE (MAPE) of the average statistical model. Likewise, the MAE (MAPE) of the average DL model for period 3 is 3.3365 (0.0007780), whilst the average statistical model’s MAE (MAPE) for the same period is 23.7421 (0.0055615). These results confirm that the MAE (MAPE) of the average DL model is 85.9% (86.0%) smaller than the MAE (MAPE) of the average statistical model. These findings of the predictive superiority of the average DL model over the average statistical model confirm the forecasting superiority of LSTM over the statistical models assessed.
In summary, the forecasting capabilities of the redesigned models were evaluated when the models were applied to the NZX 50 Index in three sample periods. RMSE, MAE and MAPE have been used as the performance evaluation measures. The above investigation confirms the predictive superiority of the Univariate LSTM, followed by the Multivariate LSTM as the next- best model. When the two LSTM models were comparatively evaluated, Univariate LSTM consistently outperformed Multivariate LSTM. It was also discovered that both LSTM models consistently outperformed the statistical models (ARIMA and HWES) by a significant margin.
These findings were valid for all the samples under investigation, and each performance evaluation criterion was consistent with these conclusions.
Table 8.2: Comparison of the two most efficient models: Percentage difference of Univariate-LSTM from the Multivariate-LSTM
Sample period one: (2009 to 2017) Sample period two: (2007 to 2017) Sample period three: includes COVID-19 impact (2007 to 2020)
RMSE MAE MAPE RMSE MAE MAPE RMSE MAE MAPE
-17.7% -12.7% -13.4% -28.3% -26.2% -27.3% -41.2% -35.0% -30.5%
Table 8.3: Comparison of Deep Learning (DL) models and statistical models
Tested Model Sample period one Sample period two Sample period three
RMSE MAE MAPE RMSE MAE MAPE RMSE MAE MAPE
Mean (average) of DL
models 1.8596 1.7092 0.0005176 1.7351 1.5669 0.0004758 3.8278 3.3365 0.0007780
Mean (average) of Statistical
models 18.5954 13.8252 0.0042119 17.9048 13.3385 0.0041422 36.4811 23.7421 0.0055615
Table 8.4: Percentage difference of the average of DL models from the average of Statistical models
Sample period one: (2009 to 2017) Sample period two: (2007 to 2017) Sample period three: includes COVID-19 impact (2007 to 2020)
RMSE MAE MAPE RMSE MAE MAPE RMSE MAE MAPE
-90.0% -87.6% -87.7% -90.3% -88.3% -88.5% -89.5% -85.9% -86.0%