Full Terms & Conditions of access and use can be found at
http://www.tandfonline.com/action/journalInformation?journalCode=ubes20
Download by: [Universitas Maritim Raja Ali Haji] Date: 11 January 2016, At: 19:12
Journal of Business & Economic Statistics
ISSN: 0735-0015 (Print) 1537-2707 (Online) Journal homepage: http://www.tandfonline.com/loi/ubes20
Comment
Atsushi Inoue
To cite this article: Atsushi Inoue (2015) Comment, Journal of Business & Economic Statistics, 33:1, 9-11, DOI: 10.1080/07350015.2014.969428
To link to this article: http://dx.doi.org/10.1080/07350015.2014.969428
Published online: 26 Jan 2015.
Submit your article to this journal
Article views: 153
View related articles
Inoue: Comment 9
Granger, C. W. J., and Timmermann, A. (2004), “Efficient Market Hy-pothesis and Forecasting,” International Journal of Forecasting, 20, 15–27. [7]
Hamilton, J. D. (1989), “A New Approach to the Economic Analysis of Nonstationary Time Series and the Business Cycle,”Econometrica, 57, 357–384. [2]
Hansen, P. R. (2010), “A Winner’s Curse for Econometric Models: On the Joint Distribution of In-Sample Fit and Out-of-Sample Fit and its Implica-tions for Model Selection,” manuscript, Department of Economics, Stanford University. [8]
Hansen, P. R., Lunde, A., and Nason, J. M. (2011), “The Model Confidence Set,”Econometrica, 79, 453–497. [6,7]
Hansen, P. R., and Timmermann, A. (2011), “Choice of Sample Split in Out-of-Sample Forecast Evaluation,” manuscript, Stanford and UCSD. [7] ——— (2013), “Equivalence Between Out-of-Sample Forecast Comparisons
and Wald Statistics,” manuscript, EUI, UCSD, and CREATES. [6] Inoue, A., and Kilian, L. (2004), “In-Sample or Out-of-Sample Tests of
Pre-dictability: Which One Should We Use?,”Econometric Reviews, 23, 371– 402. [6]
——— (2006), “On the Selection of Forecasting Models,”Journal of Econo-metrics, 130, 273–306. [6]
Li, T. (2009), “Simulation-Based Selection of Competing Structural Economet-ric Models,”Journal of Econometrics, 148, 114–123. [6]
McCracken, M. W. (2007), “Asymptotics for Out-of-Sample Tests of Granger Causality,”Journal of Econometrics, 140, 719–752. [6]
Rapach, D. E., Strauss, J. K., and Zhou, G. (2010), “Out-of-Sample Equity Pre-mium Prediction: Combination Forecasts and Links to the Real Economy,” Review of Financial Studies, 23, 821–862. [7]
Rivers, D., and Vuong, Q. (2002), “Selection Tests for Nonlinear Dynamic Models,”The Econometrics Journal, 5, 1–39. [6]
Rossi, B., and Inoue, A. (2012), “Out-of-Sample Forecast Tests Robust to the Choice of Window Size,”Journal of Business and Economic Statistics, 30, 432–453. [7]
Schorfheide, F., and Wolpin, K. I. (2013), “To Hold Out or Not to Hold Out,” manuscript, Department of Economics, University of Pennsylvania. [7] Schwarz, G. (1978), “Estimating the Dimension of a Model,”The Annals of
Statistics, 6, 461–464. [5]
Vuong, Q. (1989), “Likelihood Ratio Tests for Model Selection and Non-Nested Hypotheses,”Econometrica, 57, 307–333. [6]
Wei, C. Z. (1992), “On Predictive Least Squares Principles,”The Annals of Statistics, 20, 1–42. [6]
West, K. D. (1996), “Asymptotic Inference About Predictive Ability,” Econo-metrica, 64, 1067–1084. [3]
——— (2006), “Forecast Evaluation,” in Handbook of Economic Forecast-ing (Vol. 1), eds. G. Elliott, C. Granger, and A. Timmerman, Amsterdam: Elsevier, pp. 100–134 [3]
White, H. (2000), “A Reality Check for Data Snooping,”Econometrica, 68, 1097–1126. [7]
Yang, Y. (2005), “Can the Strengths of AIC and BIC be Shared?”Biometrika, 92, 937–950. [6]
Comment
Atsushi INOUE
Department of Economics, Vanderbilt University, Nashville, TN 37235 ([email protected])
While it is known that pseudo-out-of-sample methods are not optimal for comparing models, they are nevertheless often used to test predictability in population. In this comment, I elaborate on the often complicated relationship between in-sample and pseudo-out-of-sample inference. I develop an in-sample likeli-hood ratio test that has a pseudo-out-of-sample flavor to it.
First, consider the predictive models, yt = εt and yt = µ+εt, where εt is known to have a standard normal distri-bution for simplicity. We are interested in testingH0: µ=0.
As Diebold (2014) points out, the pseudo-out-of-sample method is not optimal for testingµ=0 (see Inoue and Kilian 2004). By the Neymann–Pearson lemma, the in-sample likelihood ra-tio test is most powerful. Even in the presence of a break to which Diebold alludes as a possible reason for the pseudo-out-of-sample method, one can still conduct an in-sample likelihood ratio test. For example, consider
yt = δI(t >[τ T])+εt. (1)
When the break occurs within the observed sample, t = 1, . . . , T, one can define an in-sample likelihood ratio test for testing yt = εt against (1), which is most powerful by the Neymann–Pearson lemma (see, e.g., Rossi2005).
Below I will consider an alternative environment in which an in-sample likelihood ratio test is closely related to pseudo-out-of-sample inference. Consider the simple time-varying-parameter model:
pendence ofyonT to simplify the notation. Robinson (1989) and Cai (2007) developed nonparametric estimation methods for such time-varying-parameter models. In related work, Gia-comini and Rossi (2013) developed a test for nonnested model comparisons using the local Kullback–Leibler information cri-terion in this environment.
The local log-likelihood function for the parameterµ(t /T) is defined as is the bandwidth (Fan, Farmen, and Gijbels1998). To establish a link between the resulting nonparametric estimator and the rolling regression estimator, I focus on the following asymmetric flat kernel:
© 2015American Statistical Association
Journal of Business & Economic Statistics
January 2015, Vol. 33, No. 1 DOI:10.1080/07350015.2014.969428
10 Journal of Business & Economic Statistics, January 2015
Then the local maximum likelihood estimator ofµ(t /T) that maximizes the local log-likelihood function (3) is given by
ˆ
Note that this is precisely the one-step-ahead forecast based on the rolling scheme with rolling window size W. In other words, the rolling regression forecast is a nonparametric esti-mator of the time-varying parameter with asymmetric flat kernel and bandwidth given by the rolling window size.
Note that the kernel (4) is not centered on zero. This fact causes a problem known as the boundary problem in the litera-ture on nonparametrics. Under the null hypothesis thatµ(·)=0, however, no bias will arise from the boundary problem. Also note that the window size plays an important role under the al-ternative hypothesis. The fixed window size used in Giacomini and White (2006) will yield a variance that is not asymptotically negligible. Moreover, the window size that is proportional to the sample size, which has been considered in both old-school and new-school WCM will yield biased estimates under the alter-native because the bias term is decreasing in the window size
W. In our context, the window size needs to go to infinity at a slower rate than the sample size,W/T → 0 asT , W → ∞ to consistently estimateµ(·) under the alternative hypothesis.
To derive a valid test of no predictive ability in population, define the log-likelihood function ofµ(·) by summing (3) for t =W+1, . . . , T: following log-likelihood ratio test statistic:
LRT =2(lnL(µT(·))−lnL(0))
This statistic can also be obtained by taking theL2norm of the
score functions giving a Lagrange multipler test interpretation to (7).
The log-likelihood ratio test statistic has an intuitive form. Becauseµ(t /T)=0 for allt =1,2, . . . , T under the null hy-pothesis, the sum of squares of their estimates is expected to be small under the null hypothesis. On the other hand, under the alternative hypothesis thatµ(t /T) = 0 for manyt, the sum of squares should diverge to infinity as the sample size grows, resulting in consistency of the test.
It is interesting to compare (7) to the DM test statistic. The numerator of the DM test statistic based on the rolling scheme
can be written as
T
The second term is the negative of the log-likelihood ratio test statistic. It is interesting to note that Clark and West (2006) removed this term by recentering the test statistic under their parametric setup.
Because infinitely many parameters are involved, the asymp-totic null distribution of the log-likelihood ratio test statistic is not chi-square, however. To make the testing procedure opera-tional, it is convenient to normalize the test statistic as follows:
T
where ˆσT2 is an estimator of the long-run variance of W(µT(t /T))2. Under the null hypothesis that µ(·)=0, one can show that (9) is asymptotically normally distributed based on the central limit theorem form-dependent random variables withmdiverging to infinity as in Romano and Wolf (2000).
Below I conduct a small Monte Carlo experiment that com-pares the size and power of the DM and LR tests whenyt=εtis tested againstyt =µ(t /T)+εt. I postulate thatεt
iid
∼ N(0,1). Note that this is not what the DM test is designed to do, of course. In the first data-generating process,µ(t /T)=0. In the second DGP, µ(t /T)=1 for all t =1,2, . . . , T. In the third DGP, µ(t /T)=sin(2π t /T). The sample size is set to 100 (T =100) and I considerW =5,10,15,20,25,50,75. The numbers ( Ta-ble 1) are the rejection frequencies when the nominal size is set to 5%.
Under the null hypothesis, the two tests are both undersized; the DM test because it does not take into parameter estimation uncertainty, the LR test because it is a nonparametric test and requires larger samples. Under the no change alternative, the DM test has good power for all the window size considered whereas the LR test has power only when W is small. The power of the LR test drops significantly whenWis greater than 16. This is becauseW is the bandwidth and is not supposed to be large relative to the sample size. Finally, under the smooth change alternative, the LR test dominates the DM test whenWis small. The optimal bandwidth is expected to be a function of the smoothness ofµ(·), that is, the more rapidlyµ(·) is changing, the smaller the window size should be. The optimal window size should be smaller for the third DGP than for the second DGP. It should also be noted that while the DM test may not be optimal, it is less sensitive to the choice of window size than the LR test is.
This discussion shows that an in-sample likelihood ratio test can have a pseudo-out-of-sample interpretation and that the local likelihood ratio test has good power in a simple Monte Carlo ex-periment. The nonparametric approach often brings new insights to the forecasting literature. For example, in related work, In-oue, Rossi, and Jin (2012) showed that the pseudo-out-of-sample model selection criterion can be made consistent, which is in
Inoue: Comment 11
Table 1. Rejection frequencies of the DM and LR tests with sizeα=0.05
Size Power Power
µTt=0 µTt=1 µTt=sin2πTt
W DM LR DM LR DM LR
5 0.000 0.018 0.997 1.000 0.523 0.976
10 0.000 0.013 1.000 1.000 0.823 0.841
15 0.000 0.007 1.000 0.908 0.785 0.202
20 0.000 0.005 0.999 0.001 0.550 0.000
25 0.000 0.004 0.999 0.000 0.185 0.000
50 0.003 0.000 0.980 0.000 0.003 0.000
75 0.012 0.000 0.775 0.000 0.036 0.000
contrast to theparametric results in Inoue and Kilian (2006). It is an open question which asymptotic approximation per-forms better in practice. The choice of window size also has significant impacts on results for DM-type tests as shown by Hansen and Timmermann (2012) and Rossi and Inoue (2012). In this context, the nonparametric interpretation of the simulated out-of-sample forecasting scheme provides insight for choosing the window size (Giraitis, Kapetanios, and Price2013; Inoue, Jin, and Rossi 2014). The nonparametric approach also has implications for comparing forecasts. When parameters are changing at the time of making a forecast, there will be a bias term that needs to be taken into account in addition to a variance term.
ACKNOWLEDGMENT
The author thanks Lutz Kilian and Barbara Rossi for helpful suggestions and comments, and the National Science Founda-tion for financial support.
REFERENCES
Cai, Z. (2007), “Trending Time-Varying Coefficient Time Series Models with Serially Correlated Errors,”Journal of Econometrics, 136, 163–188. [9]
Clark, T. E., and West, K. D. (2006), “Using Out-of-Sample Mean Squared Prediction Errors to Test the Martingale Difference Hypothesis,”Journal of Econometrics, 135, 155–186. [10]
Fan, J., Farmen, M., and Gijbels, I. (1998), “Local Maximum Likelihood Es-timation and Inference,”Journal of Royal Statistical Society, Series B, 60, 591–608. [9]
Giraitis, L., Kapetanios, G., and Price, S. (2013), “Adaptive Forecasting in the Presence of Recent and Ongoing Structural Change,”Journal of Economet-rics, 177, 153–170. [11]
Hansen, P. R., and Timmermann, A. (2012), “Choice of Sample Split in Out-of-Sample Forecast Evaluation,” unpublished manuscript, European University Institute and University of California, San Diego. [11]
Inoue, A., Rossi, B., and Jin, L. (2012), “Consistent Model Selection Over Rolling Windows,” inRecent Advances and Future Directions in Causality, Prediction, and Specification Analysis: Essays in Honor of Halbert L. White Jr, eds. X. Chen, and N. R. Swanson, New York: Springer, pp. 299–330. [10]
Inoue, A., Jin, L., and Rossi, B. (2014), “Window Selection for Out-of-Sample Forecasting with Time-Varying Parameters,” unpublished manuscript, Van-derbilt University, North Carolina State University, and Universitat Pompeu Fabra. [11]
Robinson, P. M. (1989), “Nonparametric Estimation of Time-Varying Pa-rameters,” inStatistical Analysis and Forecasting of Economic Structural Change, ed. P. Hackl, Berlin: Springer, pp. 253–264. [9]
Romano, J. P., and Wolf, M. (2000), “A More General Central Limit Theo-rem form-dependent Random Variables with Unboundedm,”Statistics & Probability Letters, 47, 115–124. [10]
Rossi, B. (2005), “Optimal Tests for Nested Model Selection with Un-derlying Parameter Instability,” Econometric Theory, 21, 962–990. [9]
