new government programs, turning over a part of nonprofi t agency operations to volunteers, contracting with a fund-raising fi rm to increase donations to an agency, and so on. With adequate foreknowledge of such developments, the sever- ity of threats to the external validity of experimental designs posed by problems of sampling and context can be attenuated signifi cantly. In the future, social science researchers will probably become increasingly adept at warding off threats to the external validity of experiments.
is very diff erent in the two types of design, with the case study providing much more depth and detail than the cross-section:
X O
A panel study is a series of cross-sectional studies based on the same sample of individuals over time; that is, a group of individuals is surveyed repeatedly over time. An examination of the eff ects of college that followed incoming stu- dents until their graduation would be a panel study. Or, a study that periodically interviewed a sample (or “panel”) of executive directors of nonprofi t agencies over time is a panel study. If the researcher collected data at six time points denoted by the subscripts, her research design could be diagrammed as follows:
O1 O2 O3 X O4 O5 O6
Finally, trend studies monitor and attempt to account for shifts over time in various indicators, such as gross national product, unemployment, attitude toward the president, number of nonprofi t organizations registered with the Inter- nal Revenue Service (IRS), and so on. Examples are plentiful; regularly published reports chart the course of myriad economic measures (consumer price index, infl ation rate), as well as indicators of public opinion (Harris poll, Gallup poll).
Th e diagram for the trend study would appear similar to the panel study, but again the O or measurement or observation would diff er (the panel study collects infor- mation from the same individual respondents over time). If the trend study had eight time points, as denoted by the subscripts, it could be diagrammed as follows:
O1 O2 O3 O4 X O5 O6 O7 O8
Some studies combine elements from several research designs in a mixed research design. For example, a researcher may conduct a large sample survey and include in the study an analysis of a particularly interesting or noteworthy case(s). Similarly, a researcher may supplement a trend study with a survey or an in-depth case study at one or more program sites. It is also possible to combine in the same study highly qualitative information (such as in-depth interviews with offi cials at a nonprofi t organization) with quantitative analysis (for example, organizational performance measurement at the same agency; see Chapter 24) to evaluate the implementation and eff ectiveness of eff orts to measure performance.
Quasi-experimental research designs can be evaluated with respect to three of the four criteria for establishing a relationship as causal (internal validity):
covariation, time order, and nonspuriousness. Th e remaining criterion is theory.
As discussed before, the substantive plausibility of a causal relationship stands apart from considerations of research design or statistical techniques.
Quasi-experimental designs are relatively strong in demonstrating covaria- tion between independent and dependent variables. Many statistics have been developed for assessing the magnitude of covariation or association between two variables (see Chapters 15 through 21). As long as the independent and depen- dent variables are measured across a sample of subjects, the researcher can use statistics to assess the degree of covariation.
An exception to this general conclusion should be noted. Because most case studies are based on a single unit of analysis (the case), establishing covariation may be problematic. For example, in a given case study, both the independent and the dependent variables may assume high values, thus tempting the researcher to conclude that one is the cause of the other. However, because other cases in which the independent variable takes on diff erent values are not examined, it is not possible to observe how the dependent variable changes with changes in the independent variable—the essence of the concept of covariation. Th is situation resembles an experiment in which a control group is mistakenly omitted: With- out the control group, it is very diffi cult to evaluate the eff ect of the experimental treatment on the dependent variable.
In quasi-experimental designs that employ repeated measurements over time—called longitudinal studies—the time order of the independent and de- pendent variables is relatively clear. Th us, in panel studies, the criterion of time order for demonstrating causality is usually substantiated because changes can be tracked over time. In-depth studies that seek to reconstruct the chronology of important events may also be able to establish time order.
Th is conclusion does not hold for static or single-point-in-time studies, how- ever. In cross-sectional studies especially, because of the lack of data collected over time, except in the most obvious cases (relationships between ascribed charac- teristics such as sex or race or age and various attitudes and behaviors), the time order of a relationship may be a matter of faith or assumption (such as relation- ships between attitudes or between attitudes and behaviors). On the one hand, we can be confi dent that such variables as race and age may lead to particular attitudes and behaviors, and not the reverse. On the other hand, the direction of relationship between years in a job and attitude toward the work is open to de- bate. As a consequence, in many static (that is, cross-sectional) studies the causal inference is weakened. For example, if all variables are measured just once, it is unclear whether employee work motivation leads to self-confi dence or vice versa, or whether either variable is the cause or the result of employee productivity.
Th e major threat to the internal validity of quasi-experimental designs is spu- riousness. Th e nonspuriousness criterion requires that the relationship between the independent variable and the dependent variable hold in the presence of all third variables. In experimental designs, control over exposure to the experi- mental treatment, random assignment of subjects to experimental and control groups, and isolation of subjects from extraneous influences enhance signifi- cantly the ability of the researcher to satisfy this condition. In quasi-experimental designs, circumstances are not as fortuitous. Exposure to the independent vari- able is beyond the control of the investigator; there is no reason to assume that those exposed are otherwise identical to those not exposed; and in the real world, confounding factors abound.
For example, a researcher interested in the determinants of effi ciency in pub- lic organizations may conduct a survey of state agencies. Th e results of the survey may show that effi ciency covaries with agency size, measured according to per- sonnel and budget. Might this relationship be causal? Th e answer is complicated.
Many other variables covary with effi ciency and may be responsible for the ob- served relationship. Th e type of technology used in an agency will aff ect both its effi ciency and its size. Similarly, the training of agency employees will aff ect how effi ciently the agency operates and the amount of personnel and budget needed.
Th e structure of the agency may infl uence the degree of effi ciency attained, as well as the level of personnel and budget. In order to determine whether the rela- tionship between agency size and effi ciency is nonspurious, the researcher would have to show that even after the eff ects of third variables such as agency technol- ogy, employee training, and organization structure have been taken into account, this relationship persists in the survey data. Chapter 17 presents an extensive dis- cussion of nonspuriousness and appropriate statistical procedures to examine this criterion of causality for nominal and ordinal data.
To test for nonspuriousness in a quasi-experimental design, researchers at- tempt to compensate statistically for their lack of control over the actual situ- ation. They employ statistical control techniques that assess the magnitude of relationship between the independent and dependent variables, taking into account (controlling for) the eff ects of plausible third variables (see Chapters 17 and 21). Unfortunately, these techniques are complex. Also, it is not possible logically to eliminate all third variables as the putative cause of an observed rela- tionship. Moreover, to control statistically for their eff ects, the researcher must hypothesize in advance the likely third variables and collect data on them. Th is task is difficult, time-consuming, and expensive—but essential nonetheless.
Because statistical control techniques require data from several cases or subjects, case studies are especially vulnerable with respect to the nonspuriousness crite- rion of causality. Although case studies are not typically strong with respect to validity, they can be very useful in uncovering important factors and suggesting hypotheses for further study.
External Validity
Although quasi-experimental designs of research must overcome serious chal- lenges to internal validity, they tend to be relatively strong with respect to exter- nal validity. Two major reasons account for this fact. First, it is easier to obtain representative samples of the population for quasi-experimental designs. Con- sequently, in these designs more confi dence can be placed in inferring fi ndings from sample to population.
Second, in general, quasi-experimental designs are conducted in more natu- ral (less artifi cial) settings than are experimental designs. Experiments often place subjects in a contrived environment controlled and monitored by the researcher.
In contrast, in quasi-experimental designs, subjects may not realize that they are the focus of study (as in the use of highly aggregated statistics pertaining to the economy, traffi c fatalities, and so on), or relevant information may be ascertained from them in comfortable and familiar surroundings (as in surveys of public attitudes and behaviors administered in the home or offi ce). Whereas few would contend that these settings are totally free of possible bias, there is consensus that they are less reactive than most experimental designs—subjects are less likely to
react to the context of the study itself. Hence, the results obtained are more likely to hold outside the study in other settings, thereby increasing external validity.
Again, exceptions must be appended to these conclusions. Because in panel studies the same respondents are interviewed repeatedly over time, they may grow sensitized to the fact that they are under study and thus become less typical of the population they were originally chosen to represent. Th is problem may be allevi- ated by limiting participation in the panel to a short period of time. Case studies are less tractable with respect to external validity. Th e case is usually selected pre- cisely because there is something distinctive, atypical, or particularly interesting about it (for example, a failure of a nuclear power plant, or a “whistleblower”
whose courageous persistence saves a state government millions of dollars). As a consequence, it is diffi cult to judge the extent to which the results of a case study have relevance for other cases. Case study researchers should devote serious attention to considering the population of cases to which they may legitimately generalize their results. Unfortunately, researchers and readers alike often are so captivated by the details of an arresting case that they fail to ask the important question: What can be learned from this case to apply to other cases?