This course presents research conducted to increase our understanding of how data collection decisions affect survey errors. This is not a “how–to-do-it” course on data collection, but instead reviews the literature on survey design decisions and data quality in order to sensitize learners to how alternative survey designs might impact the data obtained from those surveys. The course reviews a range of survey data collection methods that are both interview-based (face-to-face and telephone) and self-administered (paper questionnaires that are mailed and those that are implemented online, i.e. as web surveys). Mixed mode designs are also covered as well as several hybrid modes for collecting sensitive information e.g., self-administering the sensitive questions in what is otherwise a face-to-face interview. The course also covers newer methods such as mobile web and SMS (text message) interviews, and examines alternative data sources such as social media. It concentrates on the impact these techniques have on the quality of survey data, including error from measurement, nonresponse, and coverage, and assesses the tradeoffs between these error sources when researchers choose a mode or survey design.
1.4.3 Sequential (Follow-up) Mixed Mode
Loading...
From the course by University of Michigan
Data Collection: Online, Telephone and Face-to-face
126 ratings
University of Michigan
126 ratings
Course 2 of 7 in the Specialization Survey Data Collection and Analytics
From the lesson
Module 1: Introduction, Classic Modes of Survey Data Collection
In this lesson, you will be introduced to some key concepts about survey data collection methods that we will rely on throughout the course. By the end of this lesson, you should be well acquainted with the major sources of survey error and how these are affected -- usually in the form of tradeoffs -- by the particular mode used to administer questions and capture responses.
Meet the Instructors
Frederick Conrad, Ph.D.Research Professor, Survey Methodology
Institute for Social Research
Hi, in this segment we'll look in more detail at sequential
mixed-mode designs which are also known as mixed-mode follow ups.
Again the goal is to reduce nonresponse.
So by using one mode as a reminder for another mode,
importance of the study may be intensified.
Examples include contacting respondents who were originally
invited to participate in a mail questionnaire by phone.
Contacting by phone to remind them to fill out the paper questionnaire.
And related goal is to reduce the cost of nonresponse follow-up by doing
the initial contact in the least expensive mode possible and
then following up with increasingly expensive modes.
So that the most expensive mode is just used for
the most resistant sample members.
The downside is that the risk of mode effects is really there
because it's really hard to separate out
the effects of mode from the substantive responses in these cases.
This is a study by Dillman in which he followed up via phone or
mail with respondents who were initially invited to take part via mail,
phone, IVR which is interactive voice response.
Which in this case meant that a recorded voice presented the question, and
respondents used a telephone keypad to enter their answers, and web invitations.
If you look at the blue bars and
compare those to the yellow bars, you can see that additional cases,
additional participation was produced by a follow up in a different mode.
So, those who were initially invited by
mail actually responded at quite a high rate, about 75%.
Following up by phone with those who didn't respond increased the response
rate to sightly above 80%.
Similarly those who were invited by phone responded at about 45%.
And inviting the nonrespondents by mail also increased
overall response rates to about 80%.
One thing that's a little unusual here is that the response rates are higher for
mail than for phone.
Dillman and his colleagues attribute that to the fact that the initial mail
invitations contained
a $2 incentive,
whereas phone interviews promise an incentive upon completion.
So that's just not as strong an incentive for respondents.
So the phone initial response rate was lower than
the mail initial response rate having to do with the kind
of an asymmetry in the way the incentive was offered.
But you can see for other combinations of modes,
a follow-up in the different mode also increased response rates.
So the main finding is that offering modes sequentially can improve response rates.
But the authors note that this didn't really resolve demographic biases
due to nonresponse so
that the same kinds of people are not responding across the modes.
And that hints at a kind of disconnect between increased response rates but
no reduction in nonresponse error.
And then finally they note that mixing modes can mix mode effects.
So that the aural modes, phone and interactive voice response,
response in those modes tended to give more positive responses on bipolar
scales than when those scales were presented visually via mail or web.
So that tends to be attributed to mode.
There isn't really any other way to account for it, except for
fairly complex arguments involving differential nonresponse.
So an increase in response rate, probably not a reduction in nonresponse bias,
and introduction of a mode effect are really the outcomes.
A related study by Millar and Dillman attempted to increase
web response rates to the level equivalent to mail response rates,
on the assumption that web has many desirable properties.
And we'll talk about web surveys in more detail in the next lesson.
What they did was they compared mail to a choice of mail or web and
among many other combinations, to offering web with
what they called email augmentation, which they followed up the web invitation
with an email message that included a link to the web survey.
So, what they found was that offering the choice of mail or
web didn't improve response rates, similar to the findings
from the Fulton and Midway meta-analysis that we just mentioned.
And if anything, it reduced response rates.
But they did find that when they invited sample members to participate by web and
then followed up those who didn't respond by email with a link to the web survey,
that the resulting response rate for web respondents was
indistinguishable from the response rate for mail respondents.
So that's an example of how a mixed mode follow-up invited by web,
invited to take part in web with a follow up by email increased response rates for
the web mode, in this case to a level comparable to mail,
that kind of being the gold standard.
We've mentioned incentives a number of times.
They do play a role.
In a study by Beebe and his colleagues, they looked at the role of incentives.
Specifically, they mailed
paper questionnaires to a sample of Medicaid recipients.
This is a type of health insurance in the US.
Half received a $2 incentive in the envelope.
The response rates with the incentive were substantially higher, nine percentage
points higher, 54% versus the group with no incentive whose rates were 45%.
They then focused on nonresponse for minority,
ethnic members of the sample and tried to increase response rates for
those subgroups with a telephone follow-up.
If these respondents had received an incentive in the initial invitation,
then with a telephone follow-up, the response rate was 69%.
But if they hadn't received an incentive in the original
invitation, the follow-up response rate was 64%.
So the effect of the initial incentive was reduced in the follow-up.
The difference was between 54% and 69%,
for those who had received the incentive.
That's a difference of 15%.
But for those who had not received an incentive,
the follow up was that much more effective.
It was about a 20% difference, 19% difference.
So, overall response rate, there was an increased response rate with
a telephone follow up which offset the cost of the incentive.
Just want to say a word or
two about some of the other designs that we mentioned earlier, mixed mode designs.
So one is switching modes within the questionnaire, such as one sample,
one time point but different modes for different parts of the questionnaire.
So, the best example is a face to face interview in which the interviewer
collects information for sensitive topics through self administration.
That is, turns the computing device over to respondents who either see
the question on the screen or hear the questions over the headphones,
so-called CASI or audio-CASI.
And longitudinal mixed mode surveys in which one sample
is measured at multiple time points but
same sample members are measured with different modes at different time points.
So, for example, starting the data collection in one mode and then
switching to generally a cheaper mode once the sample members,
the households have committed to participating in a longitudinal study.
This reduces costs.
So for example if the initial mode is face to face and then for
subsequent waves by telephone, this reduces cost.
Telephone is cheaper than face to face interviews.
But it does confound time and mode effects on measurements.
So for example, if there are differences between the initial interview which
we'll say in this example is conducted face to face, and follow up interviews,
subsequent interviews conducted by phone.
It's hard to know if the differences are due to the passage of time, or
whether they're due to the different modes.
Examples of longitudinal studies that use this kind of mixed-mode design
are the panel study of income dynamics conducted at the University of Michigan.
And the current population survey, the primary labor force survey of the U.S.
conducted by the Bureau of Labor Statistics in the US.
And then parallel or separate samples for being measured with different modes.
Mainly used for comparative studies to accommodate regional survey traditions or
practical constraints, differences in coverage or literacy.
So in one region or country the penetration,
for example, of Internet may be quite low, and quite high in another
country in this multi-national study, that would be a reason to use different modes.
Literacy may vary.
So a visual mode in which questions are presented textually,
might not be acceptable for one country or region but
might be fine in another where literacy is higher.
There may even be different questionnaires
used in the different regions.
So mode effects may result, and it's again, hard to know
if these differences between countries or regions are due to mode or
due to the many other differences between the populations in these regions.
Examples are the international social survey program, the European
social survey, and the behavioral risk factors surveillance system.
They all use different modes in different countries or states.
That concludes our discussion of mixed mode designs.
In the next segment we'll turn to non-response and non-response error, which
is an important source of survey error that we've alluded to a number of times.
And we'll just go into somewhat more detail to kind of set us up for
the rest of the course.
Explore our Catalog
Join for free and get personalized recommendations, updates and offers.
Coursera provides universal access to the world’s best education,
partnering with top universities and organizations to offer courses online.
© 2018 Coursera Inc. All rights reserved.
