This course will cover the basic elements of designing and evaluating questionnaires. We will review the process of responding to questions, challenges and options for asking questions about behavioral frequencies, practical techniques for evaluating questions, mode specific questionnaire characteristics, and review methods of standardized and conversational interviewing.
Attitude Questions: Order Effects
Loading...
From the course by University of Michigan
Questionnaire Design for Social Surveys
248 ratings
University of Michigan
248 ratings
Course 3 of 7 in the Specialization Survey Data Collection and Analytics
From the lesson
Unit 4: Measuring Attitudes
Context effects in attitude questions; Use of different scales; Offering don’t know options; Response order effects
Meet the Instructors
Frederick Conrad, Ph.D.Research Professor, Survey Methodology
Institute for Social Research
Frauke Kreuter, Ph.D.Professor, Joint Program in Survey Methodology
Adjunct Research Professor, Institute for Social Research
Okay, so this segment here on response order effects fits nicely
into the attitude question unit, but it could also go somewhere else.
These response order effects might appear in factual questions,
though maybe not as often because people might
have a stronger representation of what the answer should be.
Let's take this question as one example.
"Which one of these qualities is the most desirable for a child to have?"
And here comes a long list of qualities and some
of them you might have thought about before, others not.
This is all formulated in the male version, but of course
it should, in this question, relate to both males and females.
Good manners, tries hard to succeed, is honest, neat and clean,
good sense and sound judgment, self-control,
all those characteristics of a child.
And what studies have shown is that if you have a long list, like this,
people, if they self-administer these questionnaires, where you look at this
list, tend to answer more often "yes" to items in the upper segment of this list.
Then, experiments where the order of the list was changed, so half
of the respondent would get the list in the order printed here
and the other half would get them in
the opposite order, version B, all items flipped.
And in a study by Jon Krosnick and Duane Alwin in 1987, they observed
this primacy effect, more items being picked on
the top, in both forward and backward orders.
So, they concluded it's not due to the items.
They had these 13 choices about child
characteristics that I just showed you and the
task for the respondent was to choose three
most important. And among those most important they
found that this primacy effect was actually
only present for respondents with low sophistication, as
they called it, so lower level of education
and lower level on scores on vocabulary tests
has pronounced this kind of effect.
So, they argued that this group most likely answered in a satisficing way,
which means they responded as soon as an acceptable option was encountered.
They also argued that response order effects
of this kind are a function of mode,
because under a visual representation, respondents might
become cluttered after the first few options
and the first few are likely to be endorsed, showing this primacy effect.
But if you hear it in a telephone survey and it's an auditory presentation,
then the earlier options are overwritten by later ones and so
the last few are more likely to be endorsed, which would
lead to a recency effect. And actually that has more to
do with who administers the option rather than the actual mode.
Don't get confused here,
the respondent could request, of course, from the interviewer or the computer
that there's a visual presentation in addition.
So it's not mode within the sense of what the
survey mode in general is, but whether its visual or auditory.
In a meta analysis on response order effects
that Holbrook, Krosnick, Moore, and Tourangeau did in 2007,
they examined over 500 experiments in about 150 Gallup polls
that involved dichotomous scales and looked at response order effects.
And 19.2 showed a significant recency effect
and only 1.8 showed a significant primacy effect.
Mind you, these are telephone polls.
So, they had an average shift of 2.2%,
computed in this way, you know, in the percentage of picking the
option when second minus the percentage of option when it's picked first.
They saw as largest predictor of this effect, question difficulty,
so the average lengths of words, number of
words per sentence, and the number of sentences.
The response option length was also a significant effect
and so what the position within the questionnaire.
Now, just as a quick quiz question here in
between, what is the primacy and what is recency?
That's right, primacy would mean, options that come first are more likely to be endorsed
and rencency effect is, options that come last are more likely to be endorsed.
Now, continuing with that meta-analysis,
question type is another thing that they looked at
more closely and they grouped or categorized questions
in to three types, seemingly open ended questions,
delayed processing questions, and seemingly yes/no questions.
The seemingly open ended questions would be something like
this: "How do you feel about President Bill Clinton?
Is he trustworthy or dangerous?"
And, you know, you pause after Bill Clinton, that of
course doesn't have an answer category there yet, and so "Is he
trustworthy or dangerous?" is a closed ended answer category but
is something that sounds more like a seemingly open ended question.
The delayed processing question would be like this:
"Which of the following describes your view about President Clinton?"
So you know something is coming up and now you get it, "Trustworthy or dangerous?"
And the seemingly yes/no question would be, "Do you
think that President Bill Clinton is trustworthy or dangerous?"
But the "do you think" sort of implies that there will be a
yes/no but in this case the yes/no is converted into "trustworthy or dangerous".
And they found an effective question type here.
The seemingly open ended questions were least
prone to the recency effect and the delayed processing questions
were most prone to recency effects. So that's something
to keep in mind when you consider formulating your questions.
In general, education had the largest effect, for
low education groups compared to higher education groups.
Now, let's look actually, how respondents answer these questions.
I brought with me a few video clips to show to you.
And the first one is a respondent who is considering all answer options and
chooses the best answer, so this is like the perfect respondent that we see here.
This is an eye tracking study done at the University of Maryland
by Mirta Galesic and Roger Tourangeau
and colleagues, meanwhile published but we were,
you know, allowed to use those videos and you can see here, while
I'm talking, that the respondent nicely
reads the question, thinks about the answer.
Now he reread the question, goes back up,
goes back down, back and forth reading the question,
still thinking about the answer options, and then finally
deciding on one and, "Whoo, off to the next screen."
Now, not everybody has that same response pattern.
So here is another example for a response pattern.
You have a respondent,
she reads the question
slowly, carefully uses the mouse as a pointer,
and then clicks on the first option that seems good,
but then keeps reading and reconsidering.
It's like, you know, you have
listen to phone options and you hover your finger over
a particular number and then you might pick another one.
So here, the first option was changed to
the third and then the respondent still reads all
the way down until she's done and realizes there's
no better option than the one she already picked.
And then, finally, we are seeing this satisficing type behavior with this last
respondent that we filmed here. And again, the question is read nicely.
You see the first two, three, four, five answer categories are read again.
And the bubble gets bigger, that means
there's time spent on this particular players reading.
And so far, the last four options have not been read.
And now, here one is picked and off we go to the next screen.
So, you do see that this is a situation where the respondent did
not read all the options and just picked something in the upper half.
So in summary, the response order effects are common.
They can be large.
And Krosnick and Miller actually found them on election ballots,
which is of concern if you don't randomize the order.
The direction depends on the order of processing, so the
mode of presentation, auditory versus visual, we talked about that,
the pace with which the items are read,
the type of the item, whether it's a scale or an ordered list of things.
And it's magnitude depends on the
respondent's side on processing capacity, age,
education, interest in the topic, familiarity
with the topic, item difficulty, fatigue,
motivation in general of the respondent, and some
psychological concepts, like need for cognition, need for closure,
those people who show these personality traits, would
be more likely to go through the entire list,
and fuzziness of the respondents, thinking still about prior
items and prior thoughts that might occur in the questionnaire.
What can you do to mitigate this effect as a questionnaire designer?
Well, ideally you would randomize the order of these items.
It's not always possible in particular, if there's an inherent
structure in this list. Then it might be odd to randomize them
but then you can hope that this inherent
structure will also mitigate some of these effects,
because people do expect a certain set of items to appear later on.
For example, if you have in factual questions
a list of, let's say, the study subjects,
you know, and they're alphabetical and "Computer
science", you know, you'll find that in third place
but if you study, you know, "War history", then
you will look at the very bottom of the list.
But, you know, some things are ordered historically and,
you know, it doesn't necessarily need to be alphabetical, so that can help.
But you can always at least run this list forward and backward
and randomize your respondents to one of the two,
if you can't randomize the entire order of those questionnaires.
That is one way, how you can mitigate this effect, when you design your questionnaire.
Explore our Catalog
Join for free and get personalized recommendations, updates and offers.
Coursera provides universal access to the world’s best education,
partnering with top universities and organizations to offer courses online.
© 2018 Coursera Inc. All rights reserved.
