This course will cover the basic elements of designing and evaluating questionnaires. We will review the process of responding to questions, challenges and options for asking questions about behavioral frequencies, practical techniques for evaluating questions, mode specific questionnaire characteristics, and review methods of standardized and conversational interviewing.
Attitude Questions: Don't Know
Loading...
From the course by University of Michigan
Questionnaire Design for Social Surveys
239 ratings
University of Michigan
239 ratings
Course 3 of 7 in the Specialization Survey Data Collection and Analytics
From the lesson
Unit 4: Measuring Attitudes
Context effects in attitude questions; Use of different scales; Offering don’t know options; Response order effects
Meet the Instructors
Frederick Conrad, Ph.D.Research Professor, Survey Methodology
Institute for Social Research
Frauke Kreuter, Ph.D.Professor, Joint Program in Survey Methodology
Adjunct Research Professor, Institute for Social Research
This segment is on "don't know" options.
Always a tricky question,
should you offer respondents a don't know answer
category in the questionnaire or should you not?
Howard Schuman and Stanley Presser had in
their 1981 book, "Questions and Answers in Attitude Surveys", a nice example here.
The question asked respondents on whether they are in
favor or opposed to the Agricultural Act of 1978.
The interesting thing about this question is
that this agriculture act actually didn't exist.
So, most people, as you can see in this graph here, they should say
don't know because there is no favor or opposed to something that doesn't exist.
However, in the first set of bars that you see
here, this piece here, the question was asked, just like this,
"Are you in favor or oppose?"
and there was no filtering out of people that say they have
never heard or they don't know what the Agricultural Act of 1978 is.
In this condition, we had about 70% who were volunteered the answer don't know.
In this other segment here, where first people were asked
whether they do know something about the Agricultural Act of 1978,
almost 90% said they don't and only the remaining set
was asked whether they favor or oppose this particular act.
Now there are various methods for offering don't know options.
You can have a full filter up front, "Do you
have an opinion on that?" or "Do you know about that/"
Or you can have a quasi-filter,
which is, "Do you agree, disagree, or do you not have an opinion on
that?" or "Do you favor or oppose or do you not have an opinion?"
And that would be in a formulation for a
telephone survey because in self administered mode, obviously, you
would have to write down the answer category no
opinion, even if it's not part of the questionnaire.
Now the question is, should you do this or not?
The issue here is, will the respondent be provided with the option "don't know"
or do they have to themselves say, "No I
don't know anything about this", or "I don't have an opinion"?
The views of Converse and Presser is that you should offer to
filter out respondents who don't know much and thus can't have an attitude.
That was sort of in the 80s,
that conclusion from these studies that they did.
You can do this and execute this
kind of recommendation with filters with increasing strength.
So the quasi-filter we talked about, you can have the
blunt filter, but you could also have justified full filters.
For example, "Have you been interested enough
in this to favor one side or another?"
or "Have you thought about this issue?",
or "Have you heard and read about this issue?"
to soften the filtering in case you're worried about that.
Now,
all these methods aside, Jon Krosnick and actually together with Stanley
Presser worked on this later and so more recently the views on
using "don't know" option have changed a bit and they are sort
of grounded in two cognitive models on the "don't know" response option.
The first perception is, we optimize our model
in here with the notion that there's four situations
in which a respondent might want to say "don't
know", along the lines of the response process model.
The first one is, so they interpret the question and realize they
don't know, which means that the meaning of the question is not clear,
right?
The search in the memory can be leading to "don't know", which means
that there's no information found at all in their head on this topic.
The "don't know" can arise, the third step, if
they integrate the information into a judgment but in the
formulation of their judgment, so there might be ambivalence and
conflict among the information that they have in their head
or there's insufficient information to justify an opinion. And the last one would
be in formulating the answer, so translating
the judgment into a response category.
And here, it would be that the meaning of the response alternatives
is not clear, so they can't really match their thinking to an answer category.
And it was only the second step here, the search for memory and
"don't know", that really would mean that if you force people into an answer,
meaning you don't provide a "don't know" option, they
would be giving you a non meaningful answer.
So for all of the reasons, except this one,
pushing people to offer opinions might yield to meaningful responses.
If you take another cognitive model as a baseline here
to evaluate these "don't know" responses, that of a satisfying respondent.
We'll mention this later again but to point out this term satisficing is from
Herbert Simon in the 50s, he described economic behavior with this term,
and it's being adopted by Krosnick and Duane
Alwin and brought us to match behavior and questionnaire design.
So anyway, if we have the satisficer in mind here, who
does just enough to sort of get through the questionnaire, then
some respondents sometimes look for cues in a question to allow
them to skip all interpretation or retrieval to justify the answer.
So, you know, I say "I don't know", because I don't want to go
through these four steps and think, "Do I understand, can I formulate an answer,
can I even retrieve what is in my head?" and the like.
So they just pick an easy answer category.
Most likely, this behavior might be seen when
the respondent's ability to answer that question is low
or the respondent motivation is low
or there are too high of cognitive demands going on here.
So here in this situation, if pushed to
offer opinions, these people would offer meaningful opinions
and, again, that would be an argument to actually do not offer "don't know" options.
There's some experimental evidence out there
comparing offering versus omitting "don't know" options.
McClendon and Duane Alwin in the 90s, as well as Krosnick and Berent and
Poe and others, and I'm sure that there are many more studies of that type,
they did years of experiments. They found no more
unreliability in responses to unfiltered question than to those filtered.
Another set of experiments listed here found that filtering
doesn't strengthen constraint correlations between attitudes and different issues.
McClendon also, in a later, actually earlier study in 1991,
found that filtering did not reduce acquiescence or response order effects.
And Krosnick also did a series of studies that supported the notion that
by and large, one should probably omit
"don't know" options rather than offering them.
So, the Krosnick verdict, as we call it here, summarized in those two
papers is that "don't know"s are mostly not due to complete lack of information.
"Don't know"s are mostly due to ambivalence,
unclear questions, intimidation, self-image protection, and satisficing.
The best questionnaire design strategy appears
to be omitting "don't know" filters and
telling respondents, "I'll note that, but if you had to choose, would you say...?"
So that could be a way for the interviewer
to get substantive response after all, nudging the respondent for
an answer and as a result you will be
collecting informative data from a larger portion of your sample.
Okay and in the next segment we will talk about
scales and response order effects in answer categories and the like.
Explore our Catalog
Join for free and get personalized recommendations, updates and offers.
Coursera provides universal access to the world’s best education,
partnering with top universities and organizations to offer courses online.
© 2018 Coursera Inc. All rights reserved.
