Let us consider some of the commonly used experimental designs and determine the extent to which they guard against the seven factors that could contaminate the internal validity of experimental results. The shorter the time span of the experiments, the less the chances are of encountering history, maturation, and mortality effects. Experiments lasting an hour or two do not usually meet with many of these problems. It is only when experiments are spread over an extended period of say, several months, that the possibility of encountering more of the confounding factors increases.
Quasi-Experimental Designs
Some studies expose an experimental group to a treatment and measure its effects. Such an experimental design is the weakest of all designs, and it does
TYPES OF EXPERIMENTAL DESIGNS AND INTERNAL VALIDITY 159 not measure the true cause-and-effect relationship. This is so because there is no comparison between groups, nor any recording of the status of the depen- dent variable as it was prior to the experimental treatment and how it changed after the treatment. In the absence of such control, the study is of no scientific value in determining cause-and-effect relationships. Hence, such a design is referred to as a quasi-experimental design. The following two are quasi-exper- imental designs.
Pretest and Posttest Experimental Group Design
An experimental group (without a control group) may be given a pretest, exposed to a treatment, and then given a posttest to measure the effects of the treatment. This can be diagrammed as in Figure 7.3, where O refers to some process of observation or measurement, X represents the exposure of a group to an experimental treatment, and the X and Os in the row are applied to the same specific group. Here, the effects of the treatment can be obtained by measuring the difference between the posttest and the pretest (O2–O1 ). Note, however, that testing and instrumentation effects might contaminate the internal validity. If the experiment is extended over a period of time, history and maturation effects may also confound the results.
Posttests Only with Experimental and Control Groups
Some experimental designs are set up with an experimental and a control group, the former alone being exposed to a treatment and not the latter. The effects of the treatment are studied by assessing the difference in the out- comes—that is, the posttest scores of the experimental and control groups. This is illustrated in Figure 7.4. Here is a case where the testing effects have been avoided because there is no pretest, only a posttest. Care has to be taken, how- ever, to make sure that the two groups are matched for all the possible conta- minating ―nuisance‖ variables. Otherwise, the true effects of the treatment cannot be determined by merely looking at the difference in the posttest scores of the two groups. Randomization would take care of this problem.
There are at least two possible threats to validity in this design. If the two groups are not matched or randomly assigned, selection biases could contam- inate the results. That is, the differential recruitment of the persons making up the two groups would confound the cause-and-effect relationship. Mortality
Figure 7.3
Pretest and posttest experimental group design.
Group Pretest score Treatment Posttest score
Experimental group O1 X O2
Treatment effect = (O 2 – O 1)
160 EXPERIMENTAL DESIGNS Figure 7.4
Posttest only with experimental and control groups.
Group Treatment Outcome
Experimental group X O 1
Control group O 2
Treatment effect = (O 1 – O 2)
(the dropout of individuals from groups) can also confound the results, and thus pose a threat to internal validity.
True Experimental Designs
Experimental designs, which include both the treatment and control groups and record information both before and after the experimental group is exposed to the treatment, are known as ex post facto experimental designs. These are dis- cussed below.
Pretest and Posttest Experimental and Control Group Designs
This design can be visually depicted as in Figure 7.5. Two groups—one experi- mental and the other control—are both exposed to the pretest and the posttest.
The only difference between the two groups is that the former is exposed to a treatment whereas the latter is not. Measuring the difference between the differ- ences in the post- and pretest scores of the two groups would give the net effects of the treatment. Both groups have been exposed to both the pre- and posttests, and both groups have been randomized; thus we could expect that the history, maturation, testing, and instrumentation effects have been controlled. This is so due to the fact that whatever happened with the experimental group (e.g., mat- uration, history, testing, and instrumentation) also happened with the control group, and in measuring the net effects (the difference in the differences between the pre- and posttest scores) we have controlled these contaminating factors. Through the process of randomization, we have also controlled the effects of selection biases and statistical regression. Mortality could, however, pose a problem in this design. In experiments that take several weeks, as in the
Figure 7.5
Pretest and posttest experimental and control groups.
Group Pretest Treatment Posttest
Experimental group O1 X O2
Control group O3 O4
Treatment effect = [(O2 – O1) – (O4 – O3)]
TYPES OF EXPERIMENTAL DESIGNS AND INTERNAL VALIDITY 161 case of assessing the impact of training on skill development, or measuring the impact of technology advancement on effectiveness, some of the subjects in the experimental group may drop out before the end of the experiment. It is possi- ble that those who drop out are in some way different from those who stay on until the end and take the posttest. If so, mortality could offer a plausible rival explanation for the difference between O2 and O1.
Solomon Four-Group Design
To gain more confidence in internal validity in experimental designs, it is advis- able to set up two experimental groups and two control groups for the experi- ment. One experimental group and one control group can be given both the pretest and the posttest, as shown in Figure 7.6. The other two groups will be given only the posttest. Here the effects of the treatment can be calculated in sev- eral different ways, as indicated in the figure. To the extent that we come up with almost the same results in each of the different calculations, we can attribute the effects to the treatment. This increases the internal validity of the results of the experimental design. This design, known as the Solomon four-group design, is perhaps the most comprehensive and the one with the least number of problems with internal validity.
Solomon Four-Group Design and Threats to Internal Validity
Let us examine how the threats to internal validity are taken care of in the Solomon four-group design. It is important to note that subjects have been ran- domly selected and randomly assigned to groups. This removes the statistical regression and selection biases. Group 2, the control group that was exposed to both the pre- and posttest, helps us to see whether or not history, maturation, testing, instrumentation, regression, or mortality threaten internal validity. If
Figure 7.6
Solomon four-group design.
Group Pretest Treatment Posttest
1. Experimental O 1 X O 2
2. Control O 3 O 4
3. Experimental X O 5
4. Control O 6
Treatment effect (E) could be judged by:
E = (O 2 – O 1) E = (O 2 – O 4 ) E = (O 5 – O 6 ) E = (O 5 – O 3 )
E = [(O 2 – O 1) – (O 4 – O 3)]
If all Es are similar, the cause-and-effect relationship is highly valid.
162 EXPERIMENTAL DESIGNS
scores O3 and O4 (pre- and posttest scores of Group 2) remain the same, then it is established that neither history, nor maturation, nor testing, nor instrumenta- tion, nor statistical regression, nor morality has had an impact. In other words, these have had no impact at all.
Group 3, the experimental group that was not given a pretest, helps to estab- lish whether or not testing effects have affected internal validity in a given exper- iment. The difference, if any, between O2 (the posttest score of Group 1, which was exposed to a treatment and also took a pretest) and O5 (the posttest score of Group 3, which was exposed to a treatment but did not take the pretest), can be attributed to the testing effects. If, however, O2 and O5 are equal, then inter- nal validity has not been thwarted by testing effects.
Group 4 (which has had only the posttest score but not the pretest or expo- sure to any treatment) helps us to see whether or not changes in the posttest scores for our experimental group are a function of the combined effects of his- tory and maturation by comparing O6 (the posttest score of the control group without the pretest) with O1 (the pretest score of the experimental group that was exposed to a pretest) and O3 (the pretest score of the control group that was exposed to a pretest as well). If all three scores are similar, maturation and his- tory effects have not been a problem.
Thus, the Solomon four-group experimental design guarantees the maximum internal validity, ruling out many other rival hypotheses. Where establishing cause-and-effect relationship is critical for the survival of businesses, as for exam- ple pharmaceutical companies, which often face lawsuits for questionable prod- ucts, the Solomon four-group design is eminently useful. However, because of the number of subjects that need to be recruited, the care with which the study has to be designed, the time that needs to be devoted to the experiment, and other reasons, the cost of conducting such an experiment is high. The experi- mental setup shown in Figure 7.4 with one experimental and one control group, exposing both to the posttest only, is a viable alternative since it has many of the advantages of the Solomon four-group design and can do with just half the num- ber of subjects.
Table 7.2 summarizes the threats to internal validity covered by the different experimental designs. If the subjects have all been randomly assigned to the groups, then selection biases and statistical regression are eliminated in all cases.
Double-Blind Studies
When extreme care and rigor are needed in experimental designs as in the case of discovery of new medicines that could impact on human lives, blind studies are conducted to avoid any bias that might creep in. For example, pharmaceuti- cal companies experimenting with the efficacy of newly developed drugs in the prototype stage ensure that the subjects in the experimental and control groups are kept unaware of who is given the drug, and who the placebo. Such studies are called blind studies.
When Aviron tested and announced the Flu-mist vaccine, neither the subjects nor the researchers who administered the vaccine to them were aware of the