Chapter 6 Two Behaviors that Might be Indicative of Shirking
6.2 No Opinion Responding
6.2.6 Incorporating New Difficulty Measures into the Anal- ysis of No Opinion Respondingysis of No Opinion Responding
05.0e-04.001.0015.00205.0e-04.001.0015.002
0 2000 4000 6000 0 2000 4000 6000
Not hard at all Not very hard
Somewhat hard Very hard
Density
Set-Asides Timer
Figure 6.7: Histogram of Response Times by Difficulty
that response times and reported question difficulties are positively correlated. The same relationship is evident for each of the other five public policy questions under consideration. In fact, simple correlations between response times and reported ques- tion difficulty ratings were between 0.22 and 0.39. Hence, it appears reasonable to include response times together with the difficulty measures in the analysis of shirking as identified by “no opinion” response behavior.
6.2.6 Incorporating New Difficulty Measures into the Anal-
influence survey response behavior. For the six policy questions in the MI1998 survey, respondents were not explicitly given a “no opinion” response option but a small fraction of respondents still indicated that they did not have an opinion on each question. Given the randomization design implemented in MI1998, each respondent was only asked four of the six policy questions. Thus the maximum number of times that they could express a “no opinion” response is four. Table 6.12 indicates how frequently respondents provided “no opinion” responses—almost ninety-five percent of respondents always provided an opinion in response to these policy questions. This low “no opinion” response rate is undoubtedly partly attributable to the lack of an explicit “no opinion” response option.40
Table 6.12: No Opinion Response Rates for the Six MI1998 Policy Questions Count
Non-Responses Frequency Percent
Zero 1010 94.66
One 47 4.40
Two 9 0.84
Four 1 0.09
Since this data has such a small fraction of “no opinion” responses and because those who indicated they did not have an opinion on a question were not asked the follow-up question about difficulty, it is not feasible to examine shirking behavior separately for each question.41 Instead, the dependent variable in the subsequent analysis is the number of times a respondent provided a “no opinion” response on the policy questions that they were asked. Respondents who did not provide a difficulty rating for the questions they were asked are re-coded as having rated them as very
40For a discussion of how the inclusion of “no opinion” response options affects response selection see, for example, Krosnick, Holbrook, Berent, Carson, Hanemann, Kopp, Mitchell, Presser, Ruud, Smith, Moody, Green & Conaway (2002).
41Attempts to estimate a logit model of shirking using the stacked “no opinion” response data (one observation for each respondent-question pair) was unsuccessful because there was not enough variation in the “no opinion” response indicator and the difficulty measures.
hard. As suggested in Section 5.4, this type of ordinal data may be viewed as indi- cating different levels of shirking. Thus a simple extension to the probability model expressed in Equation (5.2) implies that an ordered logit is appropriate for testing what factors affect “no opinion” response rates.
Table 6.13 contains the results from several ordered logit regressions of the number of times each respondent indicated that they did not have an opinion. The first model in this table, contained in columns two and three, reports the results from running an ordered logit regression of the number of times respondents indicated
“no opinion”on a variety of demographic characteristics. These results indicate that the only variables that have a significant effect on “no opinion” responding behavior are dummy variables for ages over sixty-five and residence in the northeast or south.
Notice, however, that this model does not do very well at explaining the variance in the “no opinion”count data—the pseudo R-squared is only 0.07. The second model in this table augments the first by adding a subjective difficulty measure. Since this analysis is being done across questions, the difficulty measure included here is the sum of the difficulty measures provided by respondents for each of the questions they were asked.42 This measure of difficulty has a statistically significant positive effect on “no opinion” response behavior across these questions. In other words, the more difficult the respondent finds the questions the more likely he is to provide “no opinion”
responses. Also, the measure of political informedness (developed from responses to seven factual questions) becomes marginally significant in the second model. The final columns in this table contain the results from further expanding the model to include a measure of how long it took respondents to answer these questions. Since this analysis looks at “no opinion” response behavior across all the policy questions, the measure of response time is the log of the sum of the active timers for these questions. This variable also has a significant positive effect on the probability of providing “no opinion” responses. None of the models in Table 6.13 explain the
42Recall that respondents were only asked four out of six of the questions and corresponding difficulty ratings. Including indicators for which policy questions respondents received did not have a significant effect on the analysis.
variance in “no opinion” responding very well, although the pseudo R-squared of the third model is almost three times that of the first model. This may occur because there are relatively few cases in which respondents provided “no opinion” responses.43
Table 6.13: Ordered Logit Analysis of No Opinion Response in the MI1998 Policy Questions
Model 1 Model 2 Model 3
Count Coef. Std. Err. Coef. Std. Err. Coef. Std. Err.
Over 65 Years Old 1.22* 0.32 1.25* 0.33 1.24* 0.33
Some High School 1.20 1.12 0.79 1.13 0.85 1.14
High school Grad 0.64 1.10 0.29 1.11 0.30 1.12
Some College 0.26 1.14 0.08 1.15 0.08 1.16
College Graduate 0.38 1.17 0.12 1.19 0.16 1.20
Graduate Work 1.32 1.17 1.09 1.19 1.16 1.19
Political Informedness -0.14 0.09 -0.15+ 0.09 -0.15 0.09
Northeast -0.99* 0.49 -1.00* 0.49 -1.03* 0.50
Midwest -0.36 0.36 -0.30 0.37 -0.37 0.37
South -0.64+ 0.38 -0.64+ 0.38 -0.69+ 0.38
Democrat -0.89 0.35 -0.94 0.36 -0.95 0.36
Republican -0.43 0.36 -0.36 0.36 -0.39 0.36
Sum of Difficulty Measures · · 0.13* 0.03 0.13* 0.03
Log Trimmed Timer · · · · 0.52* 0.26
Cut 1 2.43 1.11 3.61 1.17 7.40 2.25
Cut 2 4.24 1.15 5.46 1.21 9.25 2.27
Cut 3 6.57 1.49 7.82 1.54 11.61 2.46
Number of Observations 1056 1056 1056
Pseudo R-squared 0.07 0.11 0.12
* Statistically significantly different from zero at the p≤0.05 level.
+Statistically significantly different from zero at the p≤0.10 level.
Unfortunately, if “no opinion” responding is used to identify shirking, there is no way to extend the analysis beyond tests for the impact of respondent and question characteristics on this form of shirking behavior. Also, because “no opinion” respond- ing behavior is so rare in MI1998 policy questions, it is possible that respondents who did not want to exert much effort used some other response strategy when they were
43The infrequency of “no opinion” responding in this data suggest perhaps a more appropriate model for this analysis would be a rare-events logit as suggested by King & Zeng (2001).
shirking. It has been suggested that behaviors such as choosing the first response or randomly selecting response options might be indicative of cognitive shortcutting (Krosnick 1991). Although it may be fruitful to consider how these different behaviors are manifest in surveys, the MI1998 policy questions do not provide the ideal situ- ation for examining these behaviors since they only offer two response options: for or against. Instead, the next section turns attention to identifying shirking behavior through direct measures of response times.