Welcome everyone. We're here today with Andrew Caporaso, who's a Survey Methodologist in Westat, Statistics and Evaluation Sciences Unit. Andrew is here today with us to do a demonstration of the importance of what's called cognitive interviewing for assessing the validity of survey questions when one wants to collect design data. Andrews is currently pursuing his PhD from the joint program in survey methodology at the University of Maryland, where he received his master's degree in 2011. His work at Westat involves developing survey questions and data collection strategies for large federal research studies. He has over 10 years of experience conducting cognitive interviews and usability testing for clients including the National Cancer Institute, the Food and Drug Administration, the Centers for Disease Control, and the US Census Bureau. In addition to cognitive interviewing, Andrew's research interests include strategies to improve survey response rates and representation, mixed-mode data collection, eye tracking, and all things statistics and data science. Andrew, welcome. Thank you for joining us today, and maybe you can say a little bit more about the work that you do at Westat, and what got you into cognitive interviewing in the first place. Sure. Thanks, Brady. I was brought onto Westat as an intern to work as a cognitive interviewer, and that's where my work in the survey methods field began. I guess I work most of the time on self-administered survey instruments largely paper and pencil surveys, and also developing data collection strategies. More generally, extending beyond just the survey itself to things like contact strategies and incentives and things like that are important for maximizing participation in federal surveys. I guess my work typically begins when our clients have an idea about the topics that they want to collect data about. They sometimes have questions in mind already. Usually they have questions in mind that they want to put in their survey, and my job often is to review these as questions and give my opinions and expert evaluation about whether I think they make sense, are easy to answer it and whether they'll go on to work, I guess, work well is good survey questions that collect the data that the statisticians are expecting in a way that will facilitate their analysis. Cognitive interviewing is a big component of that because oftentimes researchers have ideas about what they want to measure and write questions that may be understandable to their fellow researchers, but it's important to also make sure that the questions are understandable by the general population and are understood in a consistent way across different groups of people. Otherwise, you'll get survey responses, but it might be that data doesn't really reflect what you expect. That's the area where most of my work revolves, is making sure those questions translate into the concepts of interests for the statistical analysis. Great. Thanks Andrew. Again, this week in the course we're spending a lot of time talking about validity and the importance of trying to establish and assess the validity of survey questions, especially when you're collecting design data. We've talked about how cognitive interviewing is such an important tool for examining the validity of survey questions, so what we're going to do in this demonstration is Andrew is going to be carrying out his role as a cognitive interviewer, and I'm actually going to be an interviewee, so I'm going to be a participant in this cognitive interview just to give everyone a flavor of how this process works and how you can get deep into what a survey question is really getting at and whether or not everybody has a good understanding of what's being asked in that question in the first place, so that's the purpose of our demonstration today. I'm going to turn it back over to Andrew, and he's going to walk me through thinking about some hypothetical survey questions, say that a client is interested in or somebody wants to ask as part of a research project. We'll talk about the parts of cognitive interviewing and how that helps us to understand whether everybody is really getting a good understanding of what a question's asking, so Andrew, I'll turn it back over to you. Brady, I'm going to treat you as if you were a cognitive interviewing participant that we recruited and came into the lab, or we talked to on the phone for evaluating some survey questions that will go on an interviewer administered survey. I'm going to start off with a typical introduction that we would give to a respondent to orient them to the process because cognitive interviewing is about understanding the processes that people go about when answering the questions. It's not always easy to audit your own behavior and understand why you answer in certain ways. I think it's important to share the introduction and go over how we ground the respondent to the process. Brady, you're here to help us out with some research that's being conducted for the Department of Health. The Department of Health is putting together a survey about health behaviors and attitudes that'll be answered by thousands of people around the country. Before the study begins, we need to make sure that the survey questions are easy to answer and that they're working as expected. Today, we're going to ask you to answer some of the questions on the survey and also share your reactions to the questions. We're interested in things like whether the questions are easy or difficult to answer, whether there's anything vague or confusing about them, or whether there's questions that you would prefer not to answer. There's no right or wrong feedback and I didn't write the questions so you won't be hurting my feelings if you have anything negative to say. Your feedback, along with feedback from other interviews like this one, will be used to make sure that the questions are working well before the survey is fielded around the country. In a moment, I'll begin reading the survey questions and I'd like for you to answer the questions as you normally would if you were responding to the survey at home. However, I'd also like for you to think aloud as you answer the questions. That is, I'd like for you to describe out loud the thoughts you have after you hear the question and as you decide on your answer. As we go through the survey, I'll also stop periodically and ask you some follow-up questions about your reactions to the survey questions. Before we get started, do you have any questions for me about the goal of the interview or your role as a research participant? Great. Good, that's what we like to hear. Without further ado, we'll go ahead and jump into the survey questions. Here's the first one, during the past 12 months, not counting times you went to an emergency room, how many times did you go to a doctor, nurse, or other health professional to get care for yourself? For myself. You said not counting ER visits, right? That's right. That's right, okay. I would be thinking about typical trips to urgent care. I'd also be thinking about trips to my primary care provider, and one thing that would be confusing is if online consultation about a medical issue would qualify as one of those trips like if I'm having an online consult about a medical question that I had. If I include those online medical consults, I would say probably 10-15 trips overall. If I include those online consults, if I don't include those online consults, I'd say probably five or six trips, I'd say, to a primary care physician. Thank you. During the past 12 months, did you use any complementary, alternative, or unconventional therapies such as herbal supplements, acupuncture, chiropractic, homeopathy, meditation, yoga, or tai chi? I have actually not used any of those things. If I go on, everything that you said in those examples, I have not done any of that and I can't really think of any other non-conventional therapies that I would have engaged in. I'd have to say no to that question. I'd like to ask you about that last question you just answered. In your own words, what do you think of when you hear the phrase complementary, alternative, or unconventional therapies? It's a good question. I think of those as being therapies that would typically not be recommended or suggested by my primary care provider. Therapies where there may not be as much science behind them. Straight medical science saying that this is the best approach to go, but rather some more anecdotal evidence that that might be a useful therapy to consider. I guess I think of it at black and white in terms of therapies suggested by a doctor would be grounded in science and these other therapies, not necessarily, so I guess that's how I would think about it. In your opinion, are these three things the same or different from each other? Complimentary, alternative, or unconventional therapies. Based on the way you asked the question, I think of those as being different synonyms for the same thing. I didn't get a different reaction when thinking about those three different ways to refer to these therapies. You answered no to the question. Have you ever used any type of complimentary, alternative, or unconventional therapy? I tried yoga once before but I'm not at all flexible, so I pretty much broke myself when I was trying to do it. That was the one time I've tried something like that. I've tried a couple of herbal supplements once or twice but no kind of sustained therapy like you're talking about. I'd say yoga is pretty much the only thing that I've tried before. Yes. Thinking about when you did yoga, did you consider it to be a complimentary, alternative, or unconventional therapy? I think of it more as just a different type of exercise. I'm pretty active in terms of cross fit and I viewed yoga just as another type of workout. I didn't necessarily think of it as a completely alternative therapy or strategy, so I think of that more as a form of stretching and careful breathing and things of that sort. I guess it would be considered alternative to your typical type of workout at the gym, but I don't see it as being that different from a standard exercise program. Can you say in your own words what homeopathy means to you? No. I guess that, that term I again lumped it in just based on the other terms that you were using. I would not be able to give you a definition, no. In overall, how easy or difficult was it for you to think about all the therapies you've used over the past 12 months? It was pretty easy because I've had particular medical conditions that I've needed assistance with, and because of the pandemic, there haven't been that many typical trips to the doctor for having a bad cold or some kind of virus. It's largely been physical, structural issues, more like along the physical therapy types that I've needed assistance with. In some sense, I've been healthier than ever before because we've just been locked at home due to the pandemic. That's minimized the number of standard visits that I would have. Any other thoughts about that question before we move forward? No. I think that's pretty much it. Great. Now we'll keep moving through the survey. The next few questions are about your beliefs about cancer. How much do you agree or disagree with each of the following statements. When I think of cancer, I automatically think of death. Would you say strongly agree, somewhat agree, somewhat disagree, or strongly disagree? I'd say somewhat agree. Cancer is most often caused by a person's behavior or lifestyle. Would you say you strongly agree, somewhat agree, somewhat disagree, or strongly disagree? I'd say somewhat disagree because I think overall, whether or not you get cancer, some of it is hard wired given your family history and given genetics. But I think some of it also is coming from the type of lifestyle that you engage in, you participate in. I think your lifestyle can exacerbate the effects of cancer, the severity of the cancer. I would say it's a mix of those things. That's what leads me to say somewhat disagree. The next question, it seems like everything causes cancer. Would you say strongly agree, somewhat agree, somewhat disagree, or strongly disagree. I'd say strongly disagree. I would strongly disagree that everything has this chance to cause cancer. I think a lot of it is genetic, but there are certain selected things that might exacerbate your propensity. Definitely, it's not that everything causes cancer. I'd like to ask you about the question that you just answered. Cancer is most often caused by a person's behavior or lifestyle, and you chose somewhat disagree. At this point, I would typically say can you say more about how you decided on your answer. During your think-aloud, you explained pretty clearly that you think some of that information is hard-wired or some of your cancer risk is hard-wired, whereas some is related to your behaviors and lifestyle. The think-aloud there was pretty informative, so I'm not sure, I would need to ask a different additional probes after that. How easy or difficult was it for you to answer these questions about cancer beliefs? Pretty easy because I again, I think that cancer comes from many different factors. Saying that it comes entirely from one source or another source, I think that's a little bit misguided. But that could just be because they do a lot of work in medical research and everything like that. I can see how people might have stronger opinions about where it's coming from depending on how much work you've done in that area. For me it was pretty easy just because I've had a lot of discussions about that with collaborators given that I work in academia. But I can see how other people may have a less informed opinion of where that might be coming from or what drives cancer. I can see people coming with a variety of perspectives on that question for sure. Thinking back to that one, cancer is the most often caused by a person's behavior or lifestyle. When you heard the phrase, most often caused by a person's behavior or lifestyle, what were you thinking about? I think about drawing the line at 50 percent when I heard that. When I hear most often that saying that the majority of the time cancer's going to be coming from the lifestyle choices that a person makes. When I was thinking about that 50 percent of the time cut point dropping below that was what came into mind if a lot of it is driven by genetics and you're predisposed to have a particular type of medical profile. I drew the line at that 50 percent and then in my mind, that statement fell far below that 50 percent lying because I think a lot of it can't be coming from genetics. That's what led to my ultimate response. Thank you. Have you ever been diagnosed as having cancer? I have not, but I have a lot of family members who have, especially in terms of prostate cancer. I just recently turned 40 and my risk is going to be elevated definitely as a result of these family members having prostate cancer. It's something that I just have in the back of my mind is again, having something to do with genetics and I'll have to get regularly checked just because of that increased risk. That's definitely in the back of my mind when I think about those questions. How often would you say you look for information about cancer? It's a dark subject. Not unless I'm reading an article about it or reading some news story about it or talking to a family member. Usually that comes from an appointment, a checkup with a physician or something like that. Outside of that, I'm not looking to read more about it recreationally or anything like that unless I absolutely have to. Any other thoughts about that set of questions? No, I don't think so. I think that's pretty much it. Yeah. I wasn't sure how much time we'd have. That's the end of my two examples. That's great. No, that's great. I went through one situation where we're dealing with a behavioral question. Maybe I'll jump back up to that. I asked two questions there, one was during the past 12 months, not counting times you went to an emergency room, how many times did you go to a doctor or nurse or other health care professional for yourself? Then that other question that I asked you about, which was about complimentary alternative and unconventional therapies. I guess my thinking there was, I was really only testing the second question, but I wanted to ask the first one to give some context that we're asking about basically, health care utilization. But as a good survey respondent, you provided some good feedback to that first question. What jumped out to me is that, you immediately double-checked about not counting the ER. That's informative for us that that might have been something that in the question that we need to draw more emphasis on, because it's an important clause that we don't want people to exclude. Otherwise we're going to include those emergency room visits and that would be contrary to the researcher objective because obviously the question implies they don't want to use emergency rooms. Then I pulled these questions from a survey in 2008, I believe. I didn't want to test anything that was too new. Right. But you brought up a good observation that you are way too good of a survey respondent, I have to say. Most people who come into the lab do not think about surveys day in and day out. You basically interviewed yourself. I'd say it was little challenging, but it's okay. You mentioned distinctions between physical trips to health care professionals versus telemedicine, telehealth. You distinguished, you said 10-15 if we include the telehealth and about five or six, if we just include the physical visits. That would be a good observation if someone were to fill this question particularly now and in the midst of a pandemic that if they're really after interactions with health care professionals and not physical trips to the hospital, to the medical office or whatever they should consider augmenting the language or having a follow-up question that asks about telemedicine or else they may get an incomplete picture of what's going on in the respondents, the healthcare utilization. You might want to wonder why I didn't follow up about that question, but it's often that you include more questions in the survey than you're actually testing because survey questions don't exist in a vacuum. You don't want to just ask a question outside of context. Because a question always appears in a survey that has some context in, and so it's best to field it within that context. But that can make it hard to do the cognitive view because respondents don't necessarily know. They know they're there to evaluate a survey, and so when you ask them to give feedback, you want to be broad but not too specific in what you're asking them. Part of the interaction is just sort of taking that information and moving on. If they give you feedback about a question you're not testing or something, and you just need to get through it and get to the information that you need. It's clearly a skill to get through these cognitive interviews and be able to handle all those kinds of situations. You can see the diversity of the types of people who might participate in these kinds of cognitive interviews that people actually conducting these interviews like Andrew, you've got to be prepared for all those different kinds of situations and still gleaning the essential information that you need to potentially refine the survey questions. I'm just one type of participant, but Andrew could interview many different kinds of people with many different kinds of backgrounds, and you have to do that to really get a sense of the different ways that people might react to these questions. That's right. Yeah. We usually do a couple of dozen interviews at most when we're pretesting because cognitive interviewing, it's kind of a labor-intensive process. You get a lot of qualitative feedback that you have to figure out how to synthesize in a way that makes sense and then translate that into findings and survey recommendations. That process really starts in the interview itself, but it really starts when you developed the survey and you review it and you decide how you're going to go about probing the respondent for information. You have to kind of have some idea of what the issues with the question might be before the survey starts so that you kind of probe to investigate those issues. But at the same time, you don't want to give away what you think the issues are to the respondent because then they might just agree with you. We don't bring the respondent in to be the questionnaire designer. We bring them to kind of be a respondent who would answer the survey at home, and we want to get as much information we have. We can from them without influencing what they would do. It's certainly a challenge to balance all that. When the respondent is someone like Brady who knows a lot about surveys, they're going to give you a lot of information that a typical respondent wouldn't provide. Brady understands how questions are supposed to work and can give feedback that I guess is a little bit more precise for us as methodologists. But people that don't work in surveys all the time, will find the process a lot more abstract and so you need to engage with them. You need to always be hearing what they're saying and adjusting your approach to interviewing with them based on what they give you. Because you have some probes that you have scripted in advance, but if the respondent says something totally that you weren't expecting, you also want to investigate that. It's much different than a standardized interview in that way. I think we've really hit on a lot of the main points as it relates to cognitive interviewing and just seeing the examples of the types of questions that are asked I think is really helpful to help learners understand how much work goes into trying to refine survey questions so that you know you're not asking confusing questions and you're getting the information that you want. This is just part of the process in really trying to maximize the validity of these kinds of questions. It's important, all this kind of pretesting, cognitive interviewing, think aloud work is incredibly important for making sure that you're asking high-quality questions to begin with and that you're maximizing that level of quality when you're at that stage of thinking about validity. This has been very valuable. Did you have any other final thoughts, Andrew, that you wanted to say about the cognitive interviewing process? Yeah. The reason that I chose one behavioral question to test and I think behavioral questions do you have a true value theoretically. When I'm interviewing, I'm trying to understand the respondent's perspective and make an assessment about whether their answer was valid or not based on their experience. There is question about these different therapies, there's a couple of aspects of the questions. One had asked about three different things, so you want to know if the respondent is thinking about all three complementary alternative and unconventional therapies and then it gives a list of seven examples and you want to make sure like are they thinking just to these examples? Or are they also thinking beyond these examples and if they are, do those examples qualify? A lot of it is just collecting that information and reporting it to the researchers and assessing like well, they said they take vitamin D, does that count as an unconventional therapy or should that not have been counted. A lot of that is comparing what the respondent tells you back to your research, objectives, and understanding whether they are in line. The second example with the attitudinal question, those questions that you're asking someone to agree or disagree with the statement, there's not really a theoretical true answer. The assessment of validity is a little bit more challenging in that aspect because you need to get their explanation of their answer and decide for yourself subjectively whether you think that that explanation justifies the response they provided. I think you chose somewhat disagree for cancer is most often caused by a person's behavior or lifestyle, and your explanation was, well, there's also hardwired family history and genetic type things that might also play a factor in. I think of issue with that is that some people might not know that dichotomy of nature versus nurture. If they don't understand that what does their answer mean? Cancer is most often caused by something, what else might it be caused by. That can be an issue I'd be worried about more of someone with low education. But you'd want to collect the approaches people take to answering the question and then compare those different approaches across different types of people and see whether differences in interpretation are really driving differences in response patterns, or if it's actually true differences in attitude. You can't get everything from a single interview, you have to do multiple interviews and then summarize at the respondent level or at the interview level and then across all interviews. Then if you do enough, you can look at different subgroups and draw conclusions and say, the question might work differently for one group versus another. That diversity is so important so that you're trying to go through the same process with different types of people like you probably don't want survey methodologists to be your only participants, right? Because then you're going to get a very different picture of the validity of these questions from then if you asked a broader set of individuals. That diversity is incredibly important. Because a lot of these surveys you're trying to collect data from a diverse set of individuals. Very important to think about that at this stage for sure. You definitely purposively sample to get like a group of people that have the important different types of characteristics represented in your population and that's important. If we just surveyed a bunch of survey methodologists, it'd be better than just interviewing health researchers or something, but you would still one that would not give you the full perspective. That's the importance of testing is to get the general populations assessment of the questions and make sure that it's in line with the researcher's objectives. That makes great sense. Well, yeah, Andrew, this has been great. I want to thank you for helping and participating in this. It's been great to get a real professional in this area to demonstrate how this really works. Thank Westat as well for allowing you to do this today and get off of your busy schedule to help with this demonstration for a little bit. We're incredibly grateful. Thank you again and good luck in your continued doctoral studies. Thanks, [inaudible]. Appreciate it. Happy to help.
