validity and vice versa. To ensure both types of validity, researchers usually try first to test the causal relationships in a tightly controlled artificial or lab setting, and once the relationship has been established, they try to test the causal rela- tionship in a field experiment. Lab experimental designs in the management area have thus far been done to assess, among other things, gender differences in leadership styles, managerial aptitudes, and so on. However, gender differences and other factors found in the lab settings are frequently not found in field stud- ies (Osborn & Vicars, 1976). These problems of external validity usually limit the use of lab experiments in the management area. Field experiments are also infre- quently undertaken because of the resultant unintended consequences—person- nel becoming suspicious, rivalries and jealousies being created among departments, and the like.
FACTORS AFFECTING INTERNAL VALIDITY
Even the best designed lab studies could be influenced by factors that might affect the internal validity of the lab experiment. That is, some confounding fac- tors might still be present that could offer rival explanations as to what is caus- ing the dependent variable. These possible confounding factors pose a threat to internal validity. The seven major threats to internal validity are the effects of history, maturation, testing, instrumentation, selection, statistical regression, and mortality and these are explained below with examples.
History Effects
Certain events or factors that would have an impact on the independent vari- able–
dependent variable relationship might unexpectedly occur while the experi- ment is in progress, and this history of events would confound the cause-and-effect relationship between the two variables, thus affecting the internal validity. For example, let us say that the manager of a Dairy Products Division wants to test the effects of the ―buy one, get one free‖ sales promotion on the sale of the com- pany-owned brand of packaged cheese, for a week. She carefully records the sales of the packaged cheese during the previous 2 weeks to assess the effect of the promotion. However, on the very day that her sales promotion goes into effect, the Dairy Farmer‘s Association unexpectedly launches a multimedia adver- tisement on the benefits of consuming dairy products, especially cheese. The sales of all dairy products, including cheese, go up in all the stores, including the one where the experiment had been in progress. Here, because of unexpected advertisement, one cannot be sure how much of the increase in sales of the packaged cheese in question was due to the sales promotion and how much to the advertisement of the Dairy Farmers‘ Association! The effects of history have reduced the internal validity or the faith that can be placed on the conclusion that the sales promotion caused the increase in sales. The history effects in this case are illustrated in Figure 7.1.
To give another example, let us say a bakery is studying the effects of adding to its bread a new ingredient that is expected to enrich it and offer
152 EXPERIMENTAL DESIGNS Figure 7.1
Illustration of history effects in experimental design.
Time: t1 t2 t3
Independent variable Dependent variable
Sales
promotion Sales
Dairy farmers' advertisement
Uncontrolled variable
more nutritional value to children under 14 years of age within 30 days, sub- ject to a certain daily intake. At the start of the experiment the bakery takes a measure of the health of 30 children through some medical yardsticks. There- after, the children are given the prescribed intakes of bread daily. Unfortu- nately, on day 20 of the experiment, a flu virus hits the city in epidemic proportions affecting most of the children studied. This unforeseen and uncontrollable effect of history, flu, has contaminated the cause-and-effect relationship study for the bakery.
Maturation Effects
Cause-and-effect inferences can also be contaminated by the effects of the pas- sage of time—another uncontrollable variable. Such contamination is called mat- uration effects. The maturation effects are a function of the processes—both biological and psychological—operating within the respondents as a result of the passage of time. Examples of maturation processes could include growing older, getting tired, feeling hungry, and getting bored. In other words, there could be a maturation effect on the dependent variable purely because of the passage of time. For instance, let us say that an R & D director contends that increases in the efficiency of workers would result within 3 months‘ time if advanced tech- nology is introduced in the work setting. If at the end of the 3 months increased efficiency is indeed found, it will be difficult to claim that the advanced tech- nology (and it alone) increased the efficiency of workers, because with the pas- sage of time, employees would also have gained experience, resulting in better job performance and therefore in improved efficiency. Thus, the internal valid- ity also gets reduced owing to the effects of maturation inasmuch as it is difficult to pinpoint how much of the increase is attributable to the introduction of the enhanced technology alone. Figure 7.2 illustrates the maturation effects in the above example.
FACTORS AFFECTING INTERNAL VALIDITY 153
Figure 7.2
Illustration of maturation effects on cause-and-effect relationship.
Time: t1 t2 t3
Independent variable Dependent variable
Enhanced technology Efficiency increases
Gaining experience and doing the job faster
Maturation effects
Testing Effects
Frequently, to test the effects of a treatment, subjects are given what is called a pretest (say, a short questionnaire eliciting their feelings and attitudes). That is, first a measure of the dependent variable is taken (the pretest), then the treat- ment given, and after that a second test, called the posttest, administered. The dif- ference between the posttest and the pretest scores is then attributed to the treatment. However, the very fact that respondents were exposed to the pretest might influence their responses on the posttest, which would adversely impact on internal validity.
For example, if a challenging job is expected to cause increases in job satisfac- tion, and a pretest on job satisfaction is administered asking for employees‘ level of satisfaction with their current jobs, it might sensitize people to the issue of job satisfaction. When a challenging job is introduced and a further job satisfaction questionnaire administered subsequently, the respondents might now react and respond to the posttest with a different frame of reference than if they had not orig- inally been sensitized to the issue of job satisfaction through the pretest.
This kind of sensitization through previous testing is called the testing effect, which also affects the internal validity of experimental designs. In the above case, though increases in job satisfaction can legitimately be measured through pre- and posttests, the pretest could confound the cause-and-effect relationship by sensitizing the respondents to the posttest. Thus, testing effects are another threat to internal validity.
Instrumentation Effects
Instrumentation effects are yet another source of threat to internal validity. These might arise because of a change in the measuring instrument between pretest and posttest, and not because of the treatment‘s differential impact at the end (Cook
& Campbell, 1979a). For instance, an observer who is involved in observing a particular pattern of behaviors in respondents before a treatment might start
154 EXPERIMENTAL DESIGNS
concentrating on a different set of behaviors after the treatment. The frame of measurement of behaviors (in a sense, the measuring instrument) has now changed and will not reflect the change in behaviors that can be attributed to the treatment. This is also true in the case of physical measuring instruments like the spring balance or other finely calibrated instruments that might lose their accuracy due to loss of tension with constant use, resulting in erroneous final measurement.
In organizations, instrumentation effects in experimental designs are possible when the pretest is done by the experimenter, treatments are given to the exper- imental groups, and the posttest on measures such as performance is done by dif- ferent managers. One manager might measure performance by the final units of output, a second manager might take into account the number of rejects as well, and a third manager might also take into consideration the amount of resources expended in getting the job done! Here, there are at least three different measur- ing instruments, if we treat each manager as a performance measuring instrument.
Thus, instrumentation effects also pose a threat to internal validity in experi- mental designs.
Selection Bias Effects
The threat to internal validity could also come from improper or unmatched selection of subjects for the experimental and control groups. For example, if a lab experiment is set up to assess the impact of working environment on employees‘ attitudes toward work, and if one of the experimental conditions is to have a group of subjects work for about 2 hours in a room with some mild stench, an ethical researcher might disclose this condition to prospective sub- jects, who may decline participation in the study. However, some volunteers might be lured through incentives (say a payment of $70 for the 2 hours of par- ticipation in the study). The volunteers so selected may be quite different from the others (inasmuch as they may come from an environment of deprivation) and their responses to the treatment might be quite different. Such bias in the selec- tion of the subjects might contaminate the cause-and-effect relationships and pose a threat to internal validity as well. Hence, newcomers, volunteers, and oth- ers who cannot be matched with the control groups would pose a threat to inter- nal validity in certain types of experiments.
Statistical Regression
The effects of statistical regression are brought about when the members chosen for the experimental group have extreme scores on the dependent variable to begin with. For instance, if a manager wants to test if he can increase the ―sales- manship‖ repertoire of the sales personnel through Dale Carnegie–type programs, he should not choose those with extremely low or extremely high abilities for the experiment. This is because we know from the laws of probability that those with very low scores on a variable (in this case, current sales abilities) have a greater probability of showing improvement and scoring closer to the mean on the posttest after being exposed to the treatment. This phenomenon of low scorers