In the light of these issues, and some of the problems levelled at survey research, to what extent is it possible to trust the findings of studies which are based upon this
approach? Such studies can be (at least partially) replicated to test whether the results of a particular study are verified in another similar study. But does this necessarily mean that both studies accurately reflect the reality that both aim to measure? For instance, if both studies suggest that people would rather the government increase taxes to improve the education service than reduce income tax, how can we be sure that people really hold such self-less beliefs?
One way in which the results from survey-based research can be tested is to look at the record of opinion polls in forecasting election outcomes – do the results of opinion polls (which aim to measure expected political behaviour) correspond with the way people behave when they actually come to cast their votes in an election? As Nick Moon of NOP, one of the leading British opinion poll companies, explains:
General Election opinion polls represent one of the few occasions when sample surveys are tested against actual figures. (Moon 1997, p.5)
Opinion polls are a form of social survey – they differ only in terms of the subject area that they focus upon. While social surveys are concerned with issues which might broadly be defined as ‘social’, polls investigate people’s political beliefs, attitudes, opinions, and behaviour; in terms of the methods employed and the general approaches followed, opinion polls and social surveys are broadly identical.
Generally, the performance of opinion polls in forecasting the share of votes achieved by the different political parties in British general elections is considered to be good. As a consequence, opinion pollsters have, until recently, been reluctant to consider changing their methods and techniques (Henn 1998, pp.117–35).
In the immediate post-war period, the pollsters achieved a reputation for effectively and accurately estimating electoral outcomes. However, the final polls at the general elections of 1951 and 1970 failed to anticipate the outcome of the electoral contests. In 1951, the three final-day polls suggested a Conservative victory of between 2.5 and 7.1%, when in fact Labour secured a larger share of electoral support by a margin of 0.8%:
Were it not for the fact that the Conservatives won 26 more seats than Labour, despite taking fewer votes, the polls’ failure would have been regarded at the time more seriously than it was. (Crewe 1983, p.11)
In 1970, however, the error on the final pre-election polls was more marked. The electorate eventually voted the Conservatives into office with a lead of 2.4%; 19 of the 20 pre-election polls, however, had put the Labour Party ahead, and four of the five final polls gave Labour a lead of between 2 and 8.7%. To a lesser extent, the performance of the pre-election polls at the two general elections in 1974 were also short of the mark. In February, the polls indicated a comfortable Conservative victory, when in fact a hung parliament was the outcome; in October, a Labour landslide was forecast, yet only a three-seat majority was won.
In 1992, the opinion polls were widely perceived to have failed in their attempts to forecast the British general election accurately. Throughout the 1992 campaign period, there were 50 national opinion polls conducted, which collectively provided a fairly constant pattern of party support, and suggested an average Labour lead over the Conservatives of 1.85%. These findings were widely interpreted as indicating that Labour would be the largest party in a hung parliament.
The final pre-election polls themselves provided a similar summary, with Labour at 40% (±2%), the Conservatives at 38% (±1%), and the Liberal Democrats at 18% (±2%). These results were largely reinforced by the exit polls, which, when adjusted, suggested between 298 and 305 seats for the Conservatives, and 294 and 307 seats for Labour.
However, in the event, the Conservatives won 42.8% of the vote – a 7.6% lead over their nearest rivals, the Labour Party – and were returned to government with 336 seats and an overall majority of 21. The net effect of these results implied the most serious failure on the part of the opinion pollsters in general elections since polling began in Britain (Butler 1994, p.vii). The mean error per party at 2.6% was higher than any recorded in the post-war period, and the mean error on the gap between the Conservatives and Labour at 8.5% was well beyond the threshold of 4.2% which could be explained by sampling error alone with an average sample size of 2102.
As a consequence of the pollsters’ apparent poor performance in terms of both the scale and the direction of the error in their final forecast polls, a series of inquiries were set up to try to identify the factors which could account for the discrepancies. Of particular significance were the 1992 and 1994 reports of the Market Research Society, the former of which suggested that there was:
a prima facie case to support the claim that opinion polls are generally likely to slightly over-estimate Labour support and under-estimate Conservative support.
(1992, p.15)
and had indeed been doing so not only throughout the 1992 campaign, but more seriously at most general elections since 1959.
The polls were again criticised at the European Assembly elections in 1994, and the 1997 general election for failing to forecast accurately the levels of support achieved by the main British political parties (Henn 1998; O’Muircheartaigh 1997).
All of this is not to suggest that opinion polls – and by implication the sample survey method itself – cannot be trusted to measure social and political phenomena necessarily. Instead, that like all research methods, surveys and opinion polls should be used carefully, and the findings derived from them interpreted cautiously.
SUMMARY
This chapter has focused on the main quantitative methods that are used in research – sample surveys and experiments. In particular, we have considered the usefulness and application of sample surveys in research.
In examining these questions, there is an explicit connection with many of the issues that were raised in Chapter 1, where the quantitative–qualitative debate was first encountered. We have seen that the logic of quantitative research is to explainsocial phenomena – why people behave in the way they do, or hold certain views and values – by reference to underlying causes. This emphasis on the search for causal connections between different phenomena (or variables) tends to steer researchers working within this tradition towards favouring highly
structured research approaches and techniques such as experiments and questionnaire-based sample surveys.
Quantitative approaches, while comparatively dominant within the social sciences, have attracted significant criticism at both epistemological and technical levels.
Largely, this criticism is levelled at the very structured nature of such methods, which, it is claimed, prevent any opportunities for respondents to seek clarification over the meaning of questions that are asked, and in turn to communicate their responses fully and clearly. This is said to threaten seriously the validity of any such research project – such critics would therefore dispute the extent to which survey research can provide an accurate assessment of people’s views and behaviour.
In terms of opinion polls, the failure to forecast the outcome of the 1992 British general election has added considerable weight to the arguments of those who have focused upon and criticised the limitations of the sample survey method for measuring public opinion, and who advocate new approaches in terms of method and technique to improve them.
Chapter research task
Design a survey on a topic that you are considering to investigate.
Produce a structured questionnaire of 10–15 (mostly ‘closed’) questions that would elicit consistent information for your chosen topic – this must be a questionnaire suitable for a face-to-face interview. You also need to take into consideration the things you have learnt from your previous reading, such as bias, objectivity, and operationalisation of concepts. In addition you need to (a) outline how you would identify the (target) population, (b) outline a sample population, and (c) outline how you would go about ‘finding’ this.
The final questionnaire should be ‘piloted’ on one other person, and you should then hold a debriefing session where you should discuss:
1. Did you successfully develop a rapport?
2. Did you have a sense the person ‘told the truth’?
3. How did you feel?
4. Did the questionnaire interview professionally? Were you taken seriously?
5. Did you have a sense that there was a hierarchical/power relationship involved?
6. Were the questions (a) clear and (b) unambiguous?
7. Was the data useful? Or was it too limiting?
You may find it useful to remind yourself of the following stages in a survey as you approach this task:
1. Choose the topic to be studied.
2. Review the literature.
3. Form hunches and hypotheses.
4. Identify the population to be surveyed, and choose sample selection strategy.
5. Carry out preparatory investigations and interviews.
6. Draft the questionnaire or interview schedule.
7. Conduct a pilot survey.
8. Finalise the questionnaire.
9. Select a sample of the population.
10. Select and train interviewers (if necessary).
11. Collect the data.
12. Process the data and analyse the results.
13. Write the research report, perhaps in the form of a book.
14. Publish the report.
RECOMMENDED READING
Arber, S. 1993. Designing Samples. In: N. Gilbert (ed.), Researching Social Life. London: Sage. pp.68–92.
De Vaus, D. 1985.Surveys in Social Research. London:
UCL Press. pp.80–105.
Fowler, F. 1993.Survey Research Methods. London:
Routledge. pp.54–68.
Hoinville, G. and Jowell, R. 1978.Survey Research Practice. London: Heinemann.
Marsh, C. 1979. Problems With Surveys: Method or Epistemology? Sociology, 13 (2), pp.293–305.
Moser, C. and Kalton, G. 1971.Survey Methods in Social Investigation. London: Heinemann.
Newell, R. 1993. Questionnaires. In: N. Gilbert (ed.), Researching Social Life. London: Sage.
Punch, K. 1998.Introduction to Social Research:
Quantitative and Qualitative Approaches. London:
Sage. pp.66–87.
Schofield, W. 1993. Sample Surveys. In: M. Hammersley (ed.), Principles of Social and Educational Research:
Block 2. Milton Keynes: The Open University.
pp.75–108.
Tonkiss, F. 1998. The History of the Social Survey. In:
C. Seale (ed.), Researching Society and Culture. London:
Sage. pp.58–71.