Okay, so this segment here on response order effects fits nicely into the attitude question unit, but it could also go somewhere else. These response order effects might appear in factual questions, though maybe not as often because people might have a stronger representation of what the answer should be. Let's take this question as one example. "Which one of these qualities is the most desirable for a child to have?" And here comes a long list of qualities and some of them you might have thought about before, others not. This is all formulated in the male version, but of course it should, in this question, relate to both males and females. Good manners, tries hard to succeed, is honest, neat and clean, good sense and sound judgment, self-control, all those characteristics of a child. And what studies have shown is that if you have a long list, like this, people, if they self-administer these questionnaires, where you look at this list, tend to answer more often "yes" to items in the upper segment of this list. Then, experiments where the order of the list was changed, so half of the respondent would get the list in the order printed here and the other half would get them in the opposite order, version B, all items flipped. And in a study by Jon Krosnick and Duane Alwin in 1987, they observed this primacy effect, more items being picked on the top, in both forward and backward orders. So, they concluded it's not due to the items. They had these 13 choices about child characteristics that I just showed you and the task for the respondent was to choose three most important. And among those most important they found that this primacy effect was actually only present for respondents with low sophistication, as they called it, so lower level of education and lower level on scores on vocabulary tests has pronounced this kind of effect. So, they argued that this group most likely answered in a satisficing way, which means they responded as soon as an acceptable option was encountered. They also argued that response order effects of this kind are a function of mode, because under a visual representation, respondents might become cluttered after the first few options and the first few are likely to be endorsed, showing this primacy effect. But if you hear it in a telephone survey and it's an auditory presentation, then the earlier options are overwritten by later ones and so the last few are more likely to be endorsed, which would lead to a recency effect. And actually that has more to do with who administers the option rather than the actual mode. Don't get confused here, the respondent could request, of course, from the interviewer or the computer that there's a visual presentation in addition. So it's not mode within the sense of what the survey mode in general is, but whether its visual or auditory. In a meta analysis on response order effects that Holbrook, Krosnick, Moore, and Tourangeau did in 2007, they examined over 500 experiments in about 150 Gallup polls that involved dichotomous scales and looked at response order effects. And 19.2 showed a significant recency effect and only 1.8 showed a significant primacy effect. Mind you, these are telephone polls. So, they had an average shift of 2.2%, computed in this way, you know, in the percentage of picking the option when second minus the percentage of option when it's picked first. They saw as largest predictor of this effect, question difficulty, so the average lengths of words, number of words per sentence, and the number of sentences. The response option length was also a significant effect and so what the position within the questionnaire. Now, just as a quick quiz question here in between, what is the primacy and what is recency? That's right, primacy would mean, options that come first are more likely to be endorsed and rencency effect is, options that come last are more likely to be endorsed. Now, continuing with that meta-analysis, question type is another thing that they looked at more closely and they grouped or categorized questions in to three types, seemingly open ended questions, delayed processing questions, and seemingly yes/no questions. The seemingly open ended questions would be something like this: "How do you feel about President Bill Clinton? Is he trustworthy or dangerous?" And, you know, you pause after Bill Clinton, that of course doesn't have an answer category there yet, and so "Is he trustworthy or dangerous?" is a closed ended answer category but is something that sounds more like a seemingly open ended question. The delayed processing question would be like this: "Which of the following describes your view about President Clinton?" So you know something is coming up and now you get it, "Trustworthy or dangerous?" And the seemingly yes/no question would be, "Do you think that President Bill Clinton is trustworthy or dangerous?" But the "do you think" sort of implies that there will be a yes/no but in this case the yes/no is converted into "trustworthy or dangerous". And they found an effective question type here. The seemingly open ended questions were least prone to the recency effect and the delayed processing questions were most prone to recency effects. So that's something to keep in mind when you consider formulating your questions. In general, education had the largest effect, for low education groups compared to higher education groups. Now, let's look actually, how respondents answer these questions. I brought with me a few video clips to show to you. And the first one is a respondent who is considering all answer options and chooses the best answer, so this is like the perfect respondent that we see here. This is an eye tracking study done at the University of Maryland by Mirta Galesic and Roger Tourangeau and colleagues, meanwhile published but we were, you know, allowed to use those videos and you can see here, while I'm talking, that the respondent nicely reads the question, thinks about the answer. Now he reread the question, goes back up, goes back down, back and forth reading the question, still thinking about the answer options, and then finally deciding on one and, "Whoo, off to the next screen." Now, not everybody has that same response pattern. So here is another example for a response pattern. You have a respondent, she reads the question slowly, carefully uses the mouse as a pointer, and then clicks on the first option that seems good, but then keeps reading and reconsidering. It's like, you know, you have listen to phone options and you hover your finger over a particular number and then you might pick another one. So here, the first option was changed to the third and then the respondent still reads all the way down until she's done and realizes there's no better option than the one she already picked. And then, finally, we are seeing this satisficing type behavior with this last respondent that we filmed here. And again, the question is read nicely. You see the first two, three, four, five answer categories are read again. And the bubble gets bigger, that means there's time spent on this particular players reading. And so far, the last four options have not been read. And now, here one is picked and off we go to the next screen. So, you do see that this is a situation where the respondent did not read all the options and just picked something in the upper half. So in summary, the response order effects are common. They can be large. And Krosnick and Miller actually found them on election ballots, which is of concern if you don't randomize the order. The direction depends on the order of processing, so the mode of presentation, auditory versus visual, we talked about that, the pace with which the items are read, the type of the item, whether it's a scale or an ordered list of things. And it's magnitude depends on the respondent's side on processing capacity, age, education, interest in the topic, familiarity with the topic, item difficulty, fatigue, motivation in general of the respondent, and some psychological concepts, like need for cognition, need for closure, those people who show these personality traits, would be more likely to go through the entire list, and fuzziness of the respondents, thinking still about prior items and prior thoughts that might occur in the questionnaire. What can you do to mitigate this effect as a questionnaire designer? Well, ideally you would randomize the order of these items. It's not always possible in particular, if there's an inherent structure in this list. Then it might be odd to randomize them but then you can hope that this inherent structure will also mitigate some of these effects, because people do expect a certain set of items to appear later on. For example, if you have in factual questions a list of, let's say, the study subjects, you know, and they're alphabetical and "Computer science", you know, you'll find that in third place but if you study, you know, "War history", then you will look at the very bottom of the list. But, you know, some things are ordered historically and, you know, it doesn't necessarily need to be alphabetical, so that can help. But you can always at least run this list forward and backward and randomize your respondents to one of the two, if you can't randomize the entire order of those questionnaires. That is one way, how you can mitigate this effect, when you design your questionnaire.
