As demonstrated throughout this chapter, multivariate analyses’ diverse character leads to quite powerful analytical and predictive capabilities. This power becomes especially tempting when the researcher is unsure of the most appro- priate analysis design and relies instead on the multivariate technique as a substitute for the necessary conceptual development. And even when applied correctly, the strengths of accommodating multiple variables and relationships create substantial complexity in the results and their interpretation.
Faced with this complexity, we caution the researcher to proceed only when the requisite conceptual foundation to support the selected technique has been developed. We have already discussed several issues particularly appli- cable to multivariate analyses, and although no single “answer” exists, we find that analysis and interpretation of any multivariate problem can be aided by following a set of general guidelines. By no means an exhaustive list of considerations, these guidelines represent more of a “philosophy of multivariate analysis” that has served us well.
The following sections discuss these points in no particular order and with equal emphasis on all.
eStABLiSH PRActicAL SiGniFicAnce AS WeLL AS StAtiSticAL SiGniFicAnce
The strength of multivariate analysis is its seemingly magical ability of sorting through a myriad number of possi- ble alternatives and finding those with statistical significance. However, with this power must come caution. Many researchers become myopic in focusing solely on the achieved significance of the results without understanding their interpretations, good or bad. A researcher must instead look not only at the statistical significance of the results but also at their practical significance. Practical significance asks the question, “So what?” For any managerial appli- cation, the results must offer a demonstrable effect that justifies action. In academic settings, research is becoming more focused not only on the statistically significant results but also on their substantive and theoretical implications, which are many times drawn from their practical significance.
For example, a regression analysis is undertaken to predict repurchase intentions, measured as the probability between 0 and 100 that the customer will shop again with the firm. The study is conducted and the results come back significant at the .05 significance level. Executives rush to embrace the results and modify firm strategy accordingly.
What goes unnoticed, however, is that even though the relationship was significant, the predictive ability was poor—
so poor that the estimate of repurchase probability could vary by as much as 620 percent at the .05 significance level. The “statistically significant” relationship could thus have a range of error of 40 percentage points! A customer predicted to have a 50 percent chance of return could really have probabilities from 30 percent to 70 percent, representing unacceptable levels upon which to take action. Had researchers and managers probed the practical or managerial significance of the results, they would have concluded that the relationship still needed refinement before it could be relied upon to guide strategy in any substantive sense.
RecoGniZe tHAt SAMPLe SiZe AFFectS ALL ReSULtS
The discussion of statistical power demonstrated the substantial impact sample size plays in achieving statistical significance, both in small and large sample sizes. For smaller samples, the sophistication and complexity of the multivariate technique may easily result in either (1) too little statistical power for the test to realistically identify significant results or (2) too easily “overfitting” the data such that the results are artificially good because they fit the sample yet provide no generalizability.
A similar impact also occurs for large sample sizes, which as discussed earlier, can make the statistical tests overly sensitive. Any time sample sizes exceed 400 respondents, the researcher should examine all significant results to ensure they have practical significance due to the increased statistical power from the sample size.
Sample sizes also affect the results when the analyses involve groups of respondents, such as discriminant analysis or MANOVA. Unequal sample sizes among groups influence the results and require additional interpretation or analysis. Thus, a researcher or user of multivariate techniques should always assess the results in light of the sample used in the analysis.
KnoW YoUR DAtA
Multivariate techniques, by their very nature, identify complex relationships that are difficult to represent simply.
As a result, the tendency is to accept the results without the typical examination one undertakes in univariate and bivariate analyses (e.g., scatterplots of correlations and boxplots of mean comparisons). Such shortcuts can be a
prelude to disaster, however. Multivariate analyses require an even more rigorous examination of the data because the influence of outliers, violations of assumptions, and missing data can be compounded across several variables to create substantial effects.
A wide-ranging set of diagnostic techniques enables discovery of these multivariate relationships in ways quite similar to the univariate and bivariate methods. The multivariate researcher must take the time to utilize these diagnostic measures for a greater understanding of the data and the basic relationships that exist. With this under- standing, the researcher grasps not only “the big picture,” but also knows where to look for alternative formulations of the original model that can aid in model fit, such as nonlinear and interactive relationships.
StRiVe FoR MoDeL PARSiMonY
Multivariate techniques are designed to accommodate multiple variables in the analysis. This feature, however, should not substitute for conceptual model development before the multivariate techniques are applied. Although it is always more important to avoid omitting a critical predictor variable, termed specification error, the researcher must also avoid inserting variables indiscriminately and letting the multivariate technique “sort out” the relevant variables for two fundamental reasons:
1 Irrelevant variables usually increase a technique’s ability to fit the sample data, but at the expense of overfitting the sample data and making the results less generalizable to the population. We address this issue in more detail when the concept of degrees of freedom is discussed in Chapter 5.
2 Even though irrelevant variables typically do not bias the estimates of the relevant variables, they can mask the true effects due to an increase in multicollinearity. Multicollinearity represents the degree to which any variable’s effect can be predicted or accounted for by the other variables in the analysis. As multicollinearity rises, the ability to define any variable’s effect is diminished. The addition of irrelevant or marginally significant variables can only increase the degree of multicollinearity, which makes interpretation of all variables more difficult.
Thus, including variables that are conceptually not relevant can lead to several potentially harmful effects, even if the additional variables do not directly bias the model results.
LooK At YoUR eRRoRS
Even with the statistical prowess of multivariate techniques, rarely do we achieve the best prediction in the first anal- ysis. The researcher is then faced with the question, “Where does one go from here?” The best answer is to look at the errors in prediction, whether they are the residuals from regression analysis, the misclassification of observations in discriminant analysis, or outliers in cluster analysis. In each case, the researcher should use the errors in prediction not as a measure of failure or merely something to eliminate, but as a starting point for diagnosing the validity of the obtained results and an indication of the remaining unexplained relationships.
SiMPLiFY YoUR MoDeLS BY SePARAtion
As you will see in many of the chapters, researchers have at their disposal a wide array of variable transformations (e.g., polynomial terms for non-linear effects) and variable combinations (e.g., interaction terms for moderation effects) that can portray just about any characteristic of a variable. Moreover, as research questions become more complex, more factors must be considered. Yet as we integrate more of these effects into a single model, the inter- actions among the effects may have unintended impacts on the results or the interpretations of the model. Thus, we encourage researchers to always try and simplify the model and reduce these interactions among effects. One common situation is when moderation effects are expected. A moderator is a variable, such as gender, which is expected to have different model parameters based on the moderator’s value (e.g., different regression coefficients for males versus females). This situation can be estimated in a single model through interaction terms as we will
discuss in later chapters, but doing that in a model with even a few independent variables becomes quite complicated and interpretation of the resulting parameter estimates even more difficult. A more direct approach would be to estimate separate models for males and then females so that each model can be easily interpreted. Another common situation is where the research question can be divided into a series of sub-questions, each representing its own dependence relationship. In these situations you may find that moving to a series of equations, such as structural equation modeling, helps separate the effects so that they are more easily estimated and understood. In all instances researchers should avoid building overly complex models with many variable transformations and other effects, and always attempt to separate the models in some fashion to simplify their basic model form and interpretation.
VALiDAte YoUR ReSULtS
The purpose of validation is to avoid the misleading results that can be obtained from overfitting – the estimation of model parameters that over-represent the characteristics of the sample at the expense of generalizability to the population at large. The model fit of any model is an optimistic assessment of how well the model will fit another sample of the population. We most often encounter overfitting when (a) the size of the data set is small, or (b) when the number of parameters in the model is large.
Split-Sample Validation The simplest form of validation is the split-sample approach, where the sample is divided into two sub-samples: one used for estimation (estimation sample) and the other (holdout or validation sample) for validation by “holding it out” of the estimation process and then seeing how well the estimated model fits this sample. Since the holdout sample was not used to estimate the model, it provides an independent means of validating the model. We demonstrate this approach in exploratory factor analysis (Chapter 3).
Cross-validation While it may seem simple to just divide the sample into two sub-samples, many times either limited sample size or other considerations make this impossible. For these situations cross-validation approaches have been developed. The basic principle of cross-validation is that the original sample is divided into a number of smaller sub-samples and the validation fit is the “average” fit across all of the sub-samples. Three of the more popular cross-validation approaches are k-fold, repeated random/resampling or leave-one-out/jackknife. The K-fold cross-validation randomly divides the original sample into k sub-samples (k folds) and a single sub-sample is held- out as the validation sample while the other k21 sub-samples are used for estimation. The process is repeated K times, each time a different sub-sample being the validation sample. An advantage is a smaller validation sample (e.g., 10%) is possible, thus making it useful for smaller samples. The repeated random/resampling validation approach randomly draws a number of samples to act as validation samples [39]. Its advantage is that the validation sample can be of any size and is not dependent on the number of sub-samples as in the k-fold approach. Finally, the leave-one- out (or jackknife) approach is an extreme version of the k-fold approach in that each fold has a single observation (i.e., one observation at a time is left out). This is repeated until all observations have been used once as a validation sample. This approach is demonstrated in discriminant analysis (Chapter 7).
Whenever a multivariate technique is employed, the researcher must strive not only to estimate a significant model but to ensure that it is representative of the population as a whole. Remember, the objective is not to find the best
“fit” just to the sample data but instead to develop a model that best describes the population as a whole.