Is the Time-Varying Parameter Model the Preferred Approach to Tourism Demand
7.2 Significance Tests of Forecast Accuracy
Consider two sets of forecasts,{ˆyit}Tt=1and{ˆyjt}Tt=1, of the time series{yt}Tt=1. Define the forecast errors as
eit = ˆyit−yt, for i=1 and 2. (7.4) The loss associated with forecast i is assumed to be a function of the actual and forecast values only through the forecast error eit, and is denoted by
g(yt,yˆit)=g(yˆit−yt)=g(eit). (7.5) Typically, g(eit)is the square (squared-error loss) or absolute value (absolute error loss) of eit.1The loss differential between the two forecasts is then denoted by
dt=g(e1t)−g(e2t). (7.6)
1Both the squared error and absolute error loss functions are used in the study, i.e., the mean square error (MSE) and mean absolute percentage error (MAPE), respectively.
The null hypothesis of equal forecast accuracy between the two forecasts is E[g(e1t)]=E[g(e2t)], i.e., E(dt)=0 for all t.
7.2.1 The Morgan-Granger-Newbold (MGN) Test
Consider the following assumptions.
(i) Loss is quadratic.
(ii) The forecast errors are (a) zero mean, (b) Gaussian, (c) serially uncorrelated or (d) contemporaneously uncorrelated.
Maintaining assumptions (i) and (ii) (a)–(c), Granger and Newbold (1977) devel- oped a test for equal forecast accuracy based on the following orthogonalisation (see Morgan 1939):
xt=e1t+e2t (7.7)
zt=e1t−e2t. (7.8)
Then, the null hypothesis of a zero mean loss differential is equivalent to the equality of the two forecast error variances or a zero covariance between xtand zt (i.e.,ρxz=0), and the test statistic is
MGN= r
[(1−r2)/(T−1)]1/2, (7.9) where
r=xz/[(xx)(zz)]1/2 (7.10) and x and z are the T×1 vectors with tth elements xt and zt, respectively. Under the null hypothesis of a zero covariance between xtand zt, MGN has a t-distribution with T−1 degrees of freedom. As the test is based on the maintained assumption that forecast errors are white noise, it is only applicable to one-step prediction. In addition, it is valid as a test of the equality of forecast accuracy only under squared error loss (Granger and Newbold 1977).
7.2.2 The Diebold-Mariano (DM) Test
Diebold and Mariano (1995) considered model-free tests for forecast accuracy that can easily be applied to non-quadratic loss functions, multi-step forecasts and forecast errors that are non-Gaussian, non-zero mean, serially correlated and contemporaneously correlated. The basis of the DM test is the sample mean of
the observed loss differential series {dt: t=1,2,3,. . .,T}, when assumptions (i) and (ii) (a)–(d) need not hold. Assuming covariance stationarity and other regularity conditions on the process {dt}, the test is based on the following statistic:
DM= ¯d/[2πˆfd(0)/T]1/2, (7.11) whered is the sample mean of d¯ t, andfˆd(0)is a consistent estimate of fd(0). The null hypothesis that E(dt)=0 for all t is rejected in favour of the alternative hypothesis that E(dt)=0 when DM, an absolute value, exceeds the critical value of a standard unit Gaussian distribution.
Consistent estimators of fd(0) can be of the form fˆd(0)=(1/2π)
m(t) k=−m(T)
w(k/m(T))γˆd(k), (7.12) where
ˆ
γd(k)=(1/T) T t=|k|+1
(dt− ¯d)(dt−|k|− ¯d) (7.13)
and d¯= T
t=1
[g(e1t)−g(e2t)]/T. (7.14) m(T), the bandwidth or lag truncation, increases with T but at a slower rate, and w(•) is the weighting scheme or kernel.2One weighting scheme, called the trun- cated rectangular kernel and used in Diebold and Mariano (1995), is the indicator function that takes the value of unity when the argument has an absolute value less than one:
w(x)=I(|x|<1). (7.15)
The DM test has been widely used to test for forecast accuracy equality in the macroeconomic context (see, for example, Swanson and White 1997, Wu 1999, Ocal 2000). An important advantage of the test is its direct applicability to non- quadratic loss functions. In addition, the test is robust to contemporaneous and serial correlations, and when the forecast errors are non-Gaussian, it maintains approxi- mately correct size. However, the condition for this is that the DM test should be applied in large samples, as in small samples it tends to be oversized. Because the forecast samples are small in the present study, the HLN test is used, which is a small-sample modification of the DM test.
2See, for example, Andrews (1991) for econometric applications.
7.2.3 The Harvey-Leybourne-Newbold (HLN) Test
Harvey et al. (1997) proposed a small-sample modification of the Diebold-Mariano test. The modification concerns an approximately unbiased estimate of the variance of the mean loss differential when forecast accuracy is measured in terms of the mean squared prediction error, and h-steps-ahead forecast errors are assumed to have zero autocorrelations at order h and beyond.
Because the optimal h-steps-ahead predictions are likely to have forecast errors that are a moving average process of order h–1, i.e., MA (h–1), the HLN test assumes that for h-steps-ahead forecasts, the loss differential dthas autocovariance:
ˆ
γ(k)=(1/T) T t=k+1
(dt− ¯d)(dt−k− ¯d). (7.16) The exact variance of the mean loss differential is
V(d)¯ =(1/T)[γ0+(2/T)
h−1
k=1
(T−k)γk]. (7.17)
The original DM test would estimate this variance by V(ˆ d)¯ =(1/T)[γˆ∗(0)+(2/T)
h−1
k=1
(T−k)γˆ∗(k)] (7.18)
ˆ
γ∗(k)=Tγˆ(k)/(T−k). (7.19) With d based on the squared prediction error, the HLN test obtains the following approximation of the expected value ofV(ˆ d):¯
E(V(ˆ d))¯ ∼V(d)[T¯ +1−2 h+h(h−1)/T]/T. (7.20) Therefore, Harvey et al. (1997) suggested modifying the DM test statistic to
DM∗=DM/[[T+1−2 h+h(h−1)/T]/T]1/2. (7.21) In addition, Harvey et al. (1997) suggested comparing DM∗ with critical val- ues from the t-distribution with (T–1) degrees of freedom instead of the standard unit normal distribution. The current study applies the HLN test to multiple-steps- ahead forecasts of different models based on the MAPE. Similar to the DM test, an important advantage of the HLN test is its direct applicability to nonquadratic loss functions. The loss differential series is defined in this study as
dt= |(yˆ1,t/yt)−1| − |(yˆ2,t/yt)−1|. (7.22)
Because of the limitations of the MGN test, in this study it is only applied to the one-year-ahead forecasts of different models for each origin country based on the criterion of the MSE. The study also investigates whether the two statistical tests based on different loss functions generate consistent results.
So far, no published study that uses the MGN test has been found in the tourism forecasting literature, and only one study has adopted the HLN test (De Mello and Nell 2005). De Mello and Nell (2005) use the HLN test to examine the forecast accuracy of three vector autoregressive (VAR) models and an almost ideal demand system (AIDS) model, but do not find significant differences in the forecast accu- racy between the two kinds of models. The current study employs both statistical tests to test for differences in the forecast accuracy between the TVP and static regression models.