Einstein is quoted as defining insanity as doing the same thing over and over again and expecting dif- ferent results, a comment that applies as much to choice of methods as to other activities. While it is generally conceded that psychiatric disorders are
‘complex’, the methods commonly used to investi- gate such disorders have often been selected for their simplicity and ease of use, often despite evidence that they cannot resolve complex problems:
• Sampling: Case–control studies are notoriously subject to sampling and measurement biases.
These as well as cross-sectional studies cannot be use to establish time precedence and thus have limited utility in identification of risk factors, or of how risk factors ‘work together’. Case–control and cross-sectional studies are easy; the prospec- tive cohort studies necessary to risk research are difficult, but essential.
• Terminology: There are terms in common use (beyond ‘moderators’ and ‘mediators’) that are questionable as they are currently applied. A
‘confounder’ is defined [38] (p. 35) as ‘A variable that can cause or prevent the outcome of interest, is not an intermediate variable (mediator), and is associated with the factor under investigation’.
If the causal paths to an outcome were known, there would be little point to study of risk factors.
The causal references in the definition aside, a
‘confounder’ may be proxy to the factor under investigation or that factor proxy to the con- founder, or the two might be overlapping. It makes a difference whether the ‘confounder’ should be set aside and the risk factor under investigation retained or vice versa.
Efforts to ‘control for’ or ‘adjust for’ certain
‘confounders’ are often motivated by the desire to estimate the specificcausaleffect of a selected risk factor of interest. However, causation cannot be inferred simply from correlation, and if one risk factor moderates or mediates another, even if causal, their effects cannot be separated.
The phrase ‘independent risk factor’ is often applied to a risk factor that adds to the predictive value for the outcome after another risk factor
is considered. Thus the term might be applied to overlapping risk factors, to a moderator, to a mediator, as well as to what in the MacArthur model is more narrowly defined as an indepen- dent risk factor. The usual use of the term seems vague and can be misleading. Thus the MacArthur approach proposes the definition in Table 6.2, which requires both that the factors be indepen- dent of each other, andthat their effects on the outcome be independent.
• Analysis: Entering multiple risk factors into mul- tiple regression models omitting interactions is easy; one need merely enter all the data into a computer program and interpret what results.
Carefully examining every pair-wise association and taking the correct action for each pair, as sug- gested in the MacArthur methods, is challenging.
Including interactions in such models requires appropriate centring [39], larger sample sizes, and careful and thoughtful interpretations, and is diffi- cult. However, omitting interactions that exist in the population both biases results and reduces power to detect associations. Ignoring interactions that might signal moderation effects is particularly troublesome.
Since the paths leading to the disorder may dif- fer in the subgroups defined by a moderator of subsequent risk factors on the disorder, problems associated with Simpson’s paradox may be quite prevalent [40–43]. In brief, if the risk factors in subpopulations defined by a moderator differ, the correlation obtained by ‘muddling’ the subpopula- tions mixes within group associations (which differ and are meaningful) with between group associa- tions (which may be irrelevant). The associations one observes may be misleading.
Moreover, even in absence of interactions in the population, the inclusion of proxies or overlapping variables in regression analyses induce problems associated with multicollinearity, again introducing bias and reducing power.
The motivation in developing the MacArthur approach: to examine whether the methods in common use may actually be slowing progress in risk research. Whether the MacArthur approach will lead to more rapid gains in understanding the aetiology of psychiatric disorders remains to be seen.
References
[1] Baron, R.M. and Kenny, D.A. (1986) The moderator- mediator variable distinction in social psychological research: conceptual, strategic, and statistical consid- erations.J. Pers. Soc. Psychol.,51, 1173–1182.
[2] Kraemer, H.C., Stice, E., Kazdin, A. and Kupfer, D.
(2001) How do risk factors work together to produce an outcome? Mediators, moderators, independent, overlapping and proxy risk factors.Am. J. Psychiatry, 158, 848–856.
[3] Kraemer, H.C., Wilson, G.T., Fairburn, C.G.et al.
(2002) Mediators and moderators of treatment effects in randomized clinical trials.Arch. Gen. Psychiatry, 59, 877–883.
[4] Kraemer, H.C., Kiernan, M., Essex, M.J. et al.
(2008) How and why criteria defining moderators and mediators differ between the Baron & Kenny and MacArthur approaches.Health Psychol.,27(2), S101–S108.
[5] King, A.C., Ahn, D.F., Atienza, A.A. et al. (2008) Exploring refinements in targeted behavioral medical intervention to advance public health. Ann. Behav.
Med.,35(3), 251–260.
[6] Kraemer, H.C., Frank, E. and Kupfer, D.J. (2006) Moderators of treatment outcomes: clinical, research, and policy importance.J. Am. Med. Assoc.,296(10), 1–4.
[7] Owens, E.B., Hinshaw, S.P., Kraemer, H.C. et al.
(2003) What treatment for whom for ADHD: moder- ators of treatment response in the MTA.J. Consult.
Clin. Psychol.,71(3), 540–552.
[8] Boyce, W.T., Essex, M.J., Alkon, A.et al. (2006) Early father involvement moderates biobehavioral suscepti- bility to mental health problems in middle childhood.
J. Am. Acad. Child Adolesc. Psychiatry, 45 (12), 1510–1520.
[9] Essex, M.J., Kraemer, H.C., Armstong, J.M. et al.
(2006) Exploring risk factors for the emergence of children’s mental health problems.Arch. Gen. Psychi- atry.,63, 1246–1256.
[10] Berkson, J. (1946) Limitations of the application of fourfold table analysis to hospital data. Biometrics Bull.,2, 47–53.
[11] Berkson, J. (1955) The statistical study of association between smoking and lung cancer.Proc. Staff Meet.
Mayo Clin.,30, 56–60.
[12] Abelson, R.P. (1997) On the surprising longevity of flogged horses: why there is a case for the significance test.Psychol. Sci.,8(1), 12–15.
[13] Borenstein, M. (1997) Hypothesis testing and effect size estimation in clinical trials.Ann. Allergy Asthma Immunol.,78, 5–16.
[14] Borenstein, M. (1998) The shift from significance test- ing to effect size estimation, inResearch and Methods, Comprehensive Clinical Psychology, vol.3(eds A.S.
Bellak and M. Hersen), Elsevier Science Publishing Company, Burlington, MD, pp. 319–349.
[15] Cohen, J. (1995) The earth is round (p< .05).Am.
Psychol.,49, 997–1003.
[16] Dar, R., Serlin, R.C. and Omer, H. (1994) Misuse of statistical tests in three decades of psychotherapy research.J. Consult. Clin. Res.,62, 75–82.
[17] Hunter, J.E. (1997) Needed: a ban on the significance test.Psychol. Sci.,8(1), 3–7.
[18] Shrout, P.E. (1997) Should significance tests be banned? Introduction to a special section exploring the pros and cons.Psychol. Sci.,8(1), 1–2.
[19] Wilkinson, L., The_Task_Force_on_Statistical_Infer- ence (1999) Statistical methods in psychology jour- nals: guidelines and explanations.Am. Psychol.,54, 594–604.
[20] Altman, D.G., Schulz, K.F., Hoher, D.et al. (2001) The revised CONSORT statement for reporting ran- domized trials: explanation and elaboration.Ann. Int.
Med.,134(8), 663–694.
[21] Begg, C., Cho, M., Eastwood, S.et al. (1999) Improv- ing the quality of reporting of randomized controlled trials: the CONSORT statement.J. Am. Med. Assoc., 276, 637–639.
[22] Rennie, D. (1996) How to report randomized con- trolled trials: the CONSORT Statement.J. Am. Med.
Assoc.,276(8), 649.
[23] Kraemer, H.C. (2007) Correlation coefficients in med- ical research: from product moment correlation to the odds ratio.Stat. Methods Med. Res.,15(6), 525–545.
[24] Kraemer, H.C. (2004) Reconsidering the odds ratio as a measure of 2X2 Association in a population.Stat.
Med.,23(2), 257–270.
[25] Kraemer, H.C., Kazdin, A.E., Offord, D.R. et al.
(1999) Measuring the potency of a risk factor for clinical or policy significance.Psychol. Methods, 4 (3), 257–271.
[26] Newcombe, R.G. (2006) A deficiency of the odds ratio as a measure of effect size.Stat. Med.,25, 4235–4240.
[27] Sackett, D.L. (1996) Down with odds ratios!.Evid.
Based Med.,1, 164–166.
[28] Altman, D.G. (1998) Confidence intervals for the number needed to treat. Br. Med. J., 317 (7168), 1309–1312.
[29] Altman, D.G. and Andersen, K. (1999) Calculating the number needed to treat for trials where the outcome is time to an event.Br. Med. J.,319, 1492–1495.
[30] Cook, R.J. and Sackett, D.L. (1995) The number needed to treat: a clinically useful measure of treat- ment effect.Br. Med. J.310, 452–454.
[31] Kraemer, H.C. and Kupfer, D.J. (2006) Size of treat- ment effects and their importance to clinical research and practice.Biol. Psychiatry,59(11), 990–996.
[32] Cornfield, J. (1956) A statistical problem arising from retrospective studies, in Proceedings of the Third Berkeley Symposium, vol. 4 (ed J. Neyman), Uni- versity of California Press, Berekely, CA, p. 135.
[33] Helzer, J.E., Kraemer, H.C. and Krueger, R.F. (eds) (2008)Dimensional Approaches in Diagnostic Clas- sification: Refining the Research Agenda for DSM-5, American Psychiatric Association, Arlington, VA.
[34] Poderycki, M.J., Simoes, J.M., Todorova, M.A.et al.
(1998) Environmental influences on epilepsy gene mapping in EL mice.J. Neurogenet.,12(2), 67–85.
[35] Todorova, M.T., Mantis, J.G., Le, M.et al. (2006) Genetic and environmental interactions determine seizure susceptibility in epileptic EL mice.Genes Brain Behav.,5(7), 518–527.
[36] Thiemann, S., Csernansky, J.G. and Berger, P.A.
(1987) Rating scales in research: the case of negative symptoms.Psychiatry Res.,20, 47–55.
[37] Kraemer, H.C. (2008) Toward non-parametric and clinically meaningful moderators and mediators.Stat.
Med.,27, 1679–1692.
[38] Last, J.M. (1995) A Dictionary of Epidemiology, Oxford University Press, New York.
[39] Kraemer, H.C. and Blasey, C. (2004) Centring in regression analysis: a strategy to prevent errors in statistical inference.Int. J. Methods Psychiatr. Res., 13(3), 141–151.
[40] Hand, D.J. (1979) Psychiatric examples of Simpson’s paradox.Br. J. Psychiatry,135, 90–96.
[41] Kraemer, H.C. (1978) Individual and ecological cor- relation in a general context: investigation of testos- terone and orgasmic frequency in the human male.
Behav. Sci.,23, 67–72.
[42] Samuels, M.L. (1951) Simpson’s Paradox and related phenomena.J. Am. Stat. Assoc.,88, 81–88.
[43] Wagner, C.H. (1982) Simpson’s paradox in real life.
Am. Stat.,36, 46–48.
7 Validity: Definitions and applications to psychiatric research
Jill M. Goldstein,
1,2Sara Cherkerzian
1,2and John C. Simpson
31Departments of Psychiatry and Medicine at Brigham and Women’s Hospital (BWH), Harvard Medical School, Boston, MA, USA
2Connors Center for Women’s Health and Gender Biology, Department of Medicine, Brigham & Women’s Hospital, Boston, MA, USA
3Department of Psychiatry at VA Boston Healthcare System, Harvard Medical School, Boston, MA, USA