When writing surveys, researchers normally follow some simple rules. The question wording needs to be clear, simple, and precise. In other words, questions need to be written so that respondents understand their meaning right away. In contrast, poorly
written questions lead to ambiguity and misunderstandings and can lead to untrust- worthy answers. In particular, vague questions, biased/value-laden questions, threat- ening questions, complex questions, negative questions, and pointless questions should be avoided.
4.4.1 Vague Questions
Vague questions are questions that do not clearly communicate to the respondent what the question is actually all about. For example, a question like“Taken alto- gether, how happy are you with Chancellor Merkel?” is unclear. It is imprecise because the respondent does not know what substantive area the questions is based on. Does the pollster want to know whether the respondent is “happy”with her rhetorical style, her appearance, her leadership style or her government’s record, or all of the above? Also the word happy should be avoided because it is at least somewhat value laden. Hence, the researcher should refine her research question by specifying a policy area and a measurement scale and by using more neutral- and less colloquial language. Hence a better question would be: Overall, how would you rate the performance of Chancellor Merkel in the domain of refugee policy during the refugee crisis in 2015 on a 0–100 scale?
4.4.2 Biased or Value-Laden Questions
Biased or value-laden questions are questions that predispose individuals to answer a question in a certain way. These questions are not formulated in a neutral way.
Rather, they use strong normative words. Consider, for example, the question: On a scale from 0 to 100, how evil do you think the German Christian Democratic Union (CDU) is? This question is clearly inappropriate for a survey as it bears judgement on the question subject. Therefore a better formulation of the same question would be: On scale from 1 to 100 how would you rate the performance of the Christian Democratic Union in the past year? Ideally, the pollster could also add a policy area to this question.
4.4.3 Threatening Questions
Threating questions might render respondents to surveys uneasy and/or make it hard for the respondent to answer to her best capacities. For instance, the question:“Do you have enough knowledge about German politics to recall the political program of the four parties the Christian Democrats, the Social Democrats, the Green Party, and the Party of Democratic Socialism?”might create several forms of uneasiness on the beholder. The person surveyed might question their capability to answer the ques- tion, they might assume that the pollster is judging them, and they might not know how to answer the question, because it is completely unclear what enough
4.4 Getting the Questions Right 39
knowledge means. Also, it might be better to reduce the question to one party, because citizens might know a lot about one party program and relatively few things about another program. A better question would be: On a scale from 0 to 100, how familiar are you with the political programs of the Social Democratic Party for the General Election 2017. (0 means I am not familiar at all and 100 means I am an expert.)
4.4.4 Complex Questions
Researchers should avoid complex questions that ask the polled about various issues at once. For example, a question like this:“On a scale from 1 to 10, please rate, for each of the 12 categories listed below, your level of knowledge, confidence, and experience”should be avoided, as it confuses respondents and makes it impossible for the respondent to answer precisely. Rather than clustering many items at once, the researcher should ask one question per topic.
4.4.5 Negative Questions
The usage of negative questions might induce bias or normative connotation into a question. A question such as“On a scale from 0 to 100 how unfit do you think American President Donald Trump is for the Presidency of the United States?”is more value laden than the same question expressed in positive terms:“On a scale from 0 to 100 how fit do you think American President Donald Trump is for Presidency of the United States?” Using the negative form might also confuse people. Therefore, it is a rule in survey research to avoid the usage of negative wording. This includes the use of words such as“not,” “rarely,” “never,”or words with negative prefixes“in-,” “im-,” “un-.”Therefore, researchers should always ask their questions in a positive fashion.
4.4.6 Pointless Questions
A pointless question is a question that does not allow the researcher or pollster to gain any relevant information. For example, the form I-94 from the US Citizenship and Immigration Services that every foreigner, who enters the United States has to fill out does not make any sense. The form asks any foreigner entering the United States: Have you ever been or are you now involved in espionage or sabotage or in terrorist activities or genocide, or between 1933 and 1945 were you involved, in any way, in persecutions associated with Nazi Germany or its allies? Respondents can circle either yes or no. This question is futile in two ways. First, people involved in any of these illegal activities have no incentive to admit to it; admitting to it would automatically mean that their entry into the United States would be denied. Second, and possibly even more importantly, there remain very few individuals alive who, in
theory, could have been involved in illegal activities during the Nazi era. And furthermore, virtually all of those very few individuals who are still alive are probably too old now to travel to the United States.