Research design issues: planning research3
Stage 4 Decide the research design
Each stage contains several operations. Box 3.4 clarifies this four stage model, drawing out the various operations contained in each stage. It may be useful for research planners to consider which instruments will be used at which stage of the research and with which sectors of the sam-ple population. Box 3.5 sets out a matrix of these for planning (see also Morrison, 1993:109), for example, of a small-scale piece of research.
A matrix approach such as this enables re-search planners to see at a glance their coverage of the sample and of the instruments used at particular points in time, making omissions clear, and promoting such questions as:
Why are certain instruments used at certain times and not at others?
Why are certain instruments used with certain people and not with others?
Why do certain times in the research use more instruments than other times?
Why is there such a heavy concentration of in-struments at the end of the study?
Why are certain groups involved in more instru-ments than other groups?
Why are some groups apparently neglected (e.g.
parents), e.g. is there a political dimension to the research?
Why are questionnaires the main kinds of instru-ment to be used?
Chapter 3
89 Box 3.4
A planning sequence for research
MANAGING THE PLANNING FOR RESEARCH
Why are some instruments (e.g. observation, test-ing) not used at all?
What makes the five stages separate?
Are documents only held by certain parties (and, if so, might one suspect an ‘institutional line’ to be revealed in them)?
Are some parties more difficult to contact than others (e.g. University teacher educators)?
Are some parties more important to the research than others (e.g. the principals)?
Why are some parties excluded from the sample (e.g. school governors, policy-makers, teachers’ as-sociations and unions)?
What is the difference between the three groups of teachers?
Matrix planning is useful for exposing key fea-tures of the planning of research. Further matri-ces might be constructed to indicate other fea-tures of the research, for example:
• the timing of the identification of the sample;
• the timing of the release of interim reports;
• the timing of the release of the final report;
• the timing of pretests and post-tests (in an experimental style of research);
• the timing of intensive necessary resource support (e.g. reprographics);
• the timing of meetings of interested parties.
These examples cover timings only; other ma-trices might be developed to cover other
combinations, for example: reporting by audi-ences; research team meetings by reporting; in-strumentation by participants etc. They are use-ful summary devices.
Conclusion
This chapter has suggested how a research plan can be formulated and operationalized, moving from general areas of interest, questions and purposes to very specific research questions which can be answered using appropriate sam-pling procedures, methodologies and instru-ments, and with the gathering of relevant data.
The message from this chapter is that, while re-search may not always unfold according to plan, it is important to have thought out the several stages and elements of research so that coher-ence and practicability have been addressed within an ethically defensible context. Such plan-ning can be usefully informed by models of re-search, and, indeed, these are addressed in sev-eral chapters of the book. The planning of re-search begins with the identification of purposes and constraints. With these in mind, the re-searcher can now decide on a research design and strategy that will provide him or her with answers to specific research questions. These in turn will serve more general research purposes and aims. Both the novice and experienced
Box 3.5
A planning matrix for research
Time sample Stage 1 (start) Stage 2 (3 months) Stage 3 (6 months) Stage 4 (9 months) Stage 5 (12 months)
Principal/ Documents Interview Documents Interview Documents
headteacher Interview Questionnaire 2 Interview
Questionnaire 1 Questionnaire 3
Teacher group 1 Questionnaire 1 Questionnaire 2 Questionnaire 3
Teacher group 2 Questionnaire 1 Questionnaire 2 Questionnaire 3
Teacher group 3 Questionnaire 1 Questionnaire 2 Questionnaire 3
Students Questionnaire 2 Interview
Parents Questionnaire 1 Questionnaire 2 Questionnaire 3
University Interview Interview
teacher Documents Documents
educators
Chapter 3
91
researcher alike have to confront the necessity of having a clear plan of action if the research is to have momentum and purpose. The notion of
‘fitness for purpose’ reigns here; the research plan must suit the purposes of the research. If the
reader is left feeling, at the end of this chap ter, that the task of research is complex, then that is an important message, for rigour and thought-ful, thorough planning are necessary if the re-search is to be worthwhile and effective.
CONCLUSION
Introduction
The quality of a piece of research not only stands or falls by the appropriateness of methodology and instrumentation but also by the suitability of the sampling strategy that has been adopted (see also Morrison, 1993:112–17). Questions of sampling arise directly out of the issue of defin-ing the population on which the research will focus. Researchers must take sampling decisions early in the overall planning of a piece of re-search. Factors such as expense, time and acces-sibility frequently prevent researchers from gain-ing information from the whole population.
Therefore they often need to be able to obtain data from a smaller group or subset of the total population in such a way that the knowledge gained is representative of the total population (however defined) under study. This smaller group or subset is the sample. Experienced re-searchers start with the total population and work down to the sample. By contrast, less ex-perienced researchers often work from the bot-tom up; that is, they determine the minimum number of respondents needed to conduct the research (Bailey, 1978). However, unless they identify the total population in advance, it is virtually impossible for them to assess how rep-resentative the sample is that they have drawn.
Suppose that a class teacher has been released from her teaching commitments for one month in order to conduct some research into the abili-ties of 13-year-old students to undertake a set of science experiments and that the research is to draw on three secondary schools which con-tain 300 such students each, a total of 900 stu-dents, and that the method that the teacher has been asked to use for data collection is a
semi-structured interview. Because of the time available to the teacher it would be impossible for her to interview all 900 students (the total population being all the cases). Therefore she has to be selective and to interview fewer than all 900 students. How will she decide that selec-tion; how will she select which students to in-terview?
If she were to interview 200 of the students, would that be too many? If she were to inter-view just twenty of the students would that be too few? If she were to interview just the males or just the females, would that give her a fair picture? If she were to interview just those stu-dents whom the science teachers had decided were ‘good at science’, would that yield a true picture of the total population of 900 students?
Perhaps it would be better for her to interview those students who were experiencing difficulty in science and who did not enjoy science, as well as those who were ‘good at science’. So she turns up on the days of the interviews only to find that those students who do not enjoy science have decided to absent themselves from the sci-ence lesson. How can she reach those students?
Decisions and problems such as these face researchers in deciding the sampling strategy to be used. Judgements have to be made about four key factors in sampling:
1 the sample size;
2 the representativeness and parameters of the sample;
3 access to the sample;
4 the sampling strategy to be used.
The decisions here will determine the sampling strategy to be used.
Chapter 4
93
The sample size
A question that often plagues novice research-ers is just how large their samples for the re-search should be. There is no clear-cut answer, for the correct sample size depends on the pur-pose of the study and the nature of the popula-tion under scrutiny. However it is possible to give some advice on this matter. Thus, a sample size of thirty is held by many to be the mini-mum number of cases if researchers plan to use some form of statistical analysis on their data.
Of more import to researchers is the need to think out in advance of any data collection the sorts of relationships that they wish to explore within subgroups of their eventual sample. The number of variables researchers set out to con-trol in their analysis and the types of statistical tests that they wish to make must inform their decisions about sample size prior to the actual research undertaking.
As well as the requirement of a minimum number of cases in order to examine relation-ships between subgroups, researchers must ob-tain the minimum sample size that will accu-rately represent the population being targeted.
With respect to size, will a large one guarantee representativeness? Surely not! In the example above the researcher could have interviewed a total sample of 450 females and still not have represented the male population. Will a small size guarantee representativeness? Again, surely not! The latter falls into the trap of saying that 50 per cent of those who expressed an opinion said that they enjoyed science, when the 50 per cent was only one student, a researcher having interviewed only two students in all. Further-more, too large a sample might become unwieldy and too small a sample might be unrepresenta-tive (e.g. in the first example, the researcher might have wished to interview 450 students but this would have been unworkable in practice or the researcher might have interviewed only ten students, which would have been unrepresenta-tive of the total population of 900 students).
Where simple random sampling is used, the sample size needed to reflect the population
value of a particular variable depends both on the size of the population and the amount of heterogeneity in the population (Bailey, 1978).
Generally, for populations of equal heterogene-ity, the larger the population, the larger the sam-ple that must be drawn. For populations of equal size, the greater the heterogeneity on a particu-lar variable, the particu-larger the sample that is needed.
To the extent that a sample fails to represent accurately the population involved, there is sam-pling error, discussed below.
Sample size is also determined to some extent by the style of the research. For example, a sur-vey style usually requires a large sample, particu-larly if inferential statistics are to be calculated.
In an ethnographic or qualitative style of research it is more likely that the sample size will be small.
Sample size might also be constrained by cost—
in terms of time, money, stress, administrative support, the number of researchers, and resources.
Borg and Gall (1979:194–5) suggest that corre-lational research requires a sample size of no fewer than thirty cases, that causal-comparative and experimental methodologies require a sample size of no fewer than fifteen cases, and that survey research should have no fewer than 100 cases in each major subgroup and twenty to fifty in each minor subgroup.
They advise (ibid.: 186) that sample size has to begin with an estimation of the smallest number of cases in the smallest subgroup of the sample, and ‘work up’ from that, rather than vice versa. So, for example, if 5 per cent of the sample must be teenage boys, and this sub-sam-ple must be thirty cases (e.g. for correlational research), then the total sample will be 30÷0.05=600; if 15 per cent of the sample must be teenage girls and the sub-sample must be forty-five cases, then the total sample must be 45÷0.15=300 cases.
The size of a probability (random) sample can be determined in two ways, either by the re-searcher exercising prudence and ensuring that the sample represents the wider features of the population with the minimum number of cases or by using a table which, from a mathematical formula, indicates the appropriate size of a
THE SAMPLE SIZE
random sample for a given number of the wider population (Morrison, 1993:117). One such ex-ample is provided by Krejcie and Morgan (1970) in Box 4.1. This suggests that if the researcher were devising a sample from a wider population of thirty or fewer (e.g. a class of students or a group of young children in a class) then she/he would be well advised to include the whole of the wider population as the sample.
The key point to note about the sample size in Box 4.1 is that the smaller the number of cases there are in the wider, whole population, the larger the proportion of that population must be which
appears in the sample; the converse of this is true:
the larger the number of cases there are in the wider, whole population, the smaller the propor-tion of that populapropor-tion can be which appears in the sample. Krejcie and Morgan (1970) note that
‘as the population increases the sample size in-creases at a diminishing rate and remains con-stant at slightly more than 380 cases’ (ibid.: 610).
Hence, for example, a piece of research involv-ing all the children in a small primary or elemen-tary school (up to 100 students in all) might re-quire between 80 per cent and 100 per cent of the school to be included in the sample, whilst a large secondary school of 1,200 students might require a sample of 25 per cent of the school in order to achieve randomness. As a rough guide in a random sample, the larger the sample, the greater is its chance of being representative.
Another approach to determining sample size for a probability sample is in relation to the con-fidence level and sampling error. For example, with confidence levels of 95 per cent and 99 per cent and sampling errors of 5 per cent and 1 per cent respectively, the following can be set as sam-ple sizes (Box 4.2). As with the table from Krejcie and Morgan earlier, we can see that the size of the sample reduces at an increasing rate as the population size increases; generally (but, clearly, not always) the larger the population, the smaller the proportion of the probability sample can be.
Borg and Gall (1979:195) suggest that, as a general rule, sample sizes should be large where:
• there are many variables;
• only small differences or small relationships are expected or predicted;
• the sample will be broken down into sub-groups;
• the sample is heterogeneous in terms of the variables under study;
• reliable measures of the dependent variable are unavailable.
Oppenheim (1992:44) adds to this the view that the nature of the scales to be used also exerts an influence on the sample size. For nominal data the sample sizes may well have to be larger than
Notes
N=population size S=sample size
Source Krejcie and Morgan, 19701 Box 4.1
Determining the size of a random sample
N S N S N S
10 10 220 140 1,200 291
15 14 230 144 1,300 297
20 19 240 148 1,400 302
25 24 250 152 1,500 306
30 28 260 155 1,600 310
35 32 270 159 1,700 313
40 36 280 162 1,800 317
45 40 290 165 1,900 320
50 44 300 169 2,000 322
55 48 320 175 2,200 327
60 52 340 181 2,400 331
65 56 360 186 2,600 335
70 59 380 191 2,800 338
75 63 400 196 3,000 341
80 66 420 201 3,500 346
85 70 440 205 4,000 351
90 73 460 210 4,500 354
95 76 480 214 5,000 357
100 80 500 217 6,000 361
110 86 550 226 7,000 364
120 92 600 234 8,000 367
130 97 650 242 9,000 368
140 103 700 248 10,000 370
150 108 750 254 15,000 375
160 113 800 260 20,000 377
170 118 850 265 30,000 379
180 123 900 269 40,000 380
190 127 950 274 50,000 381
200 132 1,000 278 75,000 382
210 136 1,100 285 100,0000 384
Chapter 4
95
for interval and ratio data (i.e. a variant of the issue of the number of subgroups to be ad-dressed, the greater the number of subgroups or possible categories, the larger the sample will have to be).
Borg and Gall (ibid.) set out a formula-driven approach to determining sample size (see also Moser and Kalton, 1977; Ross and Rust, 1997:427–38), and they also suggest using cor-relational tables for corcor-relational studies—avail-able in most texts on statistics—‘in reverse’ as it were, to determine sample size (p. 201), i.e.
looking at the significance levels of correlation co-efficients and then reading off the sample sizes usually required to demonstrate that level of significance. For example, a correlational sig-nificance level of 0.01 would require a sample size of 10 if the estimated co-efficient of correla-tion is 0.65, or a sample size of 20 if the esti-mated correlation co-efficient is 0.45, and a sample size of 100 if the estimated correlation co-efficient is 0.20. Again, an inverse proportion can be seen—the larger the sample size, the smaller the estimated correlation co-efficient can be to be deemed significant.
With both qualitative and quantitative data, the essential requirement is that the sample is representative of the population from which it is drawn. In a dissertation concerned with a life history (i.e. n=1), the sample is the population!
Qualitative data
In a qualitative study of thirty highly able girls of similar socio-economic background follow-ing an A-level Biology course, a sample of five or six may suffice the researcher who is prepared to obtain additional corroborative data by way of validation.
Where there is heterogeneity in the popula-tion, then a larger sample must be selected on some basis that respects that heterogeneity. Thus, from a staff of sixty secondary school teachers differentiated by gender, age, subject specialism, management or classroom responsibility, etc., it would be insufficient to construct a sample con-sisting of ten female classroom teachers of Arts and Humanities subjects.
Quantitative data
For quantitative data, a precise sample number can be calculated according to the level of accu-racy and the level of probability that the searcher requires in her work. She can then re-port in her study the rationale and the basis of her research decision (Blalock, 1979).
By way of example, suppose a teacher/re-searcher wishes to sample opinions among 1,000 secondary school students. She intends to use a 10-point scale ranging from 1=totally
Box 4.2
Sample size, confidence levels and sampling error
THE SAMPLE SIZE
Sampling error of 5% with a conf idence Sampling error of 1 % with a conf idence
level of 95% level of 99%
Size of total population Size of sample population Size of sample population
(N) (S) (S)
50 44 50
100 79 99
200 132 196
500 217 476
1,000 278 907
2,000 322 1,661
5,000 357 3,311
10,000 370 4,950
20,000 377 6,578
50,000 381 8,195
100,000 383 8,926
1,000,000 384 9,706
unsatisfactory to 10=absolutely fabulous. She already has data from her own class of thirty students and suspects that the responses of other students will be broadly similar. Her own stu-dents rated the activity (an extra-curricular event) as follows: mean score=7.27; standard de-viation=1.98. In other words, her students were pretty much ‘bunched’ about a warm, positive appraisal on the 10-point scale. How many of the 1,000 students does she need to sample in order to gain an accurate (i.e. reliable) assess-ment of what the whole school (n=1,000) thinks of the extra-curricular event?
It all depends on what degree of accuracy and what level of probability she is willing to accept.
A simple calculation from a formula by Blalock (1979:215–18) shows that:
• if she is happy to be within + or - 0.5 of a scale point and accurate 19 times out of 20, then she requires a sample of 60 out of the 1,000;
• if she is happy to be within + or - 0.5 of a scale point and accurate 99 times out of 100, then she requires a sample of 104 out of the 1,000;
• if she is happy to be within + or - 0.5 of a scale point and accurate 999 times out of 1,000, then she requires a sample of 170 out of the 1,000;
• if she is a perfectionist and wishes to be within + or - 0.25 of a scale point and accurate 999 times out of 1,000, then she requires a sam-ple of 679 out of the 1,000.
Determining the size of the sample will also have to take account of attrition and respondent mortality, i.e. that some participants will leave the research or fail to return questionnaires.
Hence it is advisable to overestimate rather than to underestimate the size of the sample required.
It is clear that sample size is a matter of judge-ment as well as mathematical precision; even formula-driven approaches make it clear that there are elements of prediction, standard error
and human judgement involved in determining sample size.
Sampling error
If many samples are taken from the same popula-tion, it is unlikely that they will all have character-istics identical with each other or with the popula-tion; their means will be different. In brief, there will be sampling error (see Cohen and Holliday, 1979; 1996). Sampling error is often taken to be the difference between the sample mean and the population mean. Sampling error is not necessar-ily the result of mistakes made in sampling proce-dures. Rather, variations may occur due to the chance selection of different individuals. For ex-ample, if we take a large number of samples from the population and measure the mean value of each sample, then the sample means will not be identi-cal. Some will be relatively high, some relatively low, and many will cluster around an average or mean value of the samples. We show this diagram-matically in Box 4.3.
Why should this occur? We can explain the phenomenon by reference to the Central Limit Theorem which is derived from the laws of prob-ability. This states that if random large samples of equal size are repeatedly drawn from any population, then the mean of those samples will be approximately normally distributed. The
Box 4.3
Distribution of sample means showing the spread of a se-lection of sample means around the population mean
Source Cohen and Holliday, 1979
Chapter 4
97
distribution of sample means approaches the normal distribution as the size of the sample in-creases, regardless of the shape—normal or oth-erwise—of the parent population (Hopkins, Hopkins and Glass, 1996:159, 388). Moreover, the average or mean of the sample means will be approximately the same as the population mean. The authors demonstrate this (pp. 159–
62) by reporting the use of computer simula-tion to examine the sampling distribusimula-tion of means when computed 10,000 times (a method that we discuss in the final chapter of this book).
Rose and Sullivan (1993:144) remind us that 95 per cent of all sample means fall between plus or minus 1.96 standard errors of the sam-ple and population means, i.e. that we have a 95 per cent chance of having a single sample mean within these limits, that the sample mean will fall within the limits of the population mean.
By drawing a large number of samples of equal size from a population, we create a sam-pling distribution. We can calculate the error involved in such sampling. The standard devia-tion of the theoretical distribudevia-tion of sample means is a measure of sampling error and is called the standard error of the mean (SEM).
Thus,
where
SDS= the standard deviation of the sample and N = the number in the sample.
Strictly speaking, the formula for the standard error of the mean is:
where SDpop=the standard deviation of the population.
However, as we are usually unable to ascertain the SD of the total population, the standard deviation of the sample is used instead. The standard error of the mean provides the best
estimate of the sampling error. Clearly, the sam-pling error depends on the variability (i.e. the heterogeneity) in the population as measured by SDpop as well as the sample size (N) (Rose and Sullivan, 1993:143). The smaller the SDpop the smaller the sampling error; the larger the N, the smaller the sampling error. Where the SDpop is very large, then N needs to be very large to coun-teract it. Where SDpop is very small, then N, too, can be small and still give a reasonably small sampling error. As the sample size increases the sampling error decreases. Hopkins, Hopkins and Glass (1996:159) suggest that, unless there are some very unusual distributions, samples of twenty-five or greater usually yield a normal sampling distribution of the mean. For further analysis of steps that can be taken to cope with the estimation of sampling in surveys we refer the reader to Ross and Wilson (1997).
The standard error of proportions We said earlier that one answer to ‘How big a sample must I obtain?’ is ‘How accurate do I want my results to be?’ This is well illustrated in the following example:
A school principal finds that the 25 students she talks to at random are reasonably in favour of a proposed change in the lunch break hours, 66 per cent being in favour and 34 per cent being against.
How can she be sure that these proportions are truly representative of the whole school of 1,000 students?
A simple calculation of the standard error of pro-portions provides the principal with her answer.
where
P = the percentage in favour Q = 100 per cent–P
N = the sample size
The formula assumes that each sample is drawn on a simple random basis. A small correction
SAMPLE ERROR