We're now going to move on to Section D. Minimizing Bias in the Included Studies: Bias in the Analysis. This is the last area where bias may be important in the studies that you're including in your systematic review and meta-analysis. There's nothing that's really called analysis bias. So, I sort of weaseled around the terminology here by calling it bias in the analysis. I think you'll know what I mean as we go along. The things that you're concerned about and you no doubt learned about in the first year at epidemiology and statistics class. Is that the following things can increase the bias in the analysis of an individual study. If there are losses to follow-up, this can increase bias. For example, maybe those dropping out or everybody who got better, or everybody who got worse. And maybe you tend to get better if you're in the test treatment group. Maybe you tend to get worse if you're in the test treatment group. We often can't tell which direction the bias is supposed to go, but nevertheless, it is a potential bias when there are high losses to follow-up. Similarly, I think it's intuitive that those who don't comply with the treatment they're assigned can bias the analysis. For example, if we have a surgical trial and people are assigned to one type of surgery versus another. And many of those allocated to type A surgery decide not to have the surgery and either to have type B or not to have surgery at all, you can see how that might bias the analysis. That is, if we drop them out entirely. Why didn't they have the surgery? Maybe because there's something about that surgery we need to know or maybe they're analyzed with the other group, which could bias that analysis as well. So noncompliance is a very complicated analysis but we do know that noncompliance can possibly bias the analysis. Similarly, withdrawals. Some people withdraw from a study, and they either stop taking the treatment, or they completely withdraw. They don't come back for any visits. Those people can bias the analysis or lead to bias in the analysis. And then, as I mentioned earlier briefly and we'll talk a little bit more about it here, there can be changes in the outcome measure. That is, as we're going along let's say I measured pain two possible ways using two measures. And I said, I could see that using one of the measures there wasn't going to be a difference between the test treatment and the comparison treatment. And so I switched the pain outcome measure that I was planning on using to one where a different stitch show up. Wasn't a big difference, but it was a statistically significant difference. So this so of post hoc defined outcome can lead to bias in the analysis. So missing data is an increasingly recognized big problem in your analysis. Unfortunately, there's no simple rule. So you might ask, well, if we come across a study and there's been 10% dropout or missing data, what do we do? Should we eliminate that study from our systematic review of meta analysis? No, probably not. What about 25%? Gee, that's not so good but I don't know, I hate dropping studies out. What about 50%? Well now I'm getting a little bit worried. So you have to think about what's the question you're asking, what's the overall proportion of missing data. And what kind of affect are you looking for and how could missing data affect this? Now you have to make all of these decisions before you start. You can't make those decisions on the basis of seeing the data and deciding whether to include or exclude a study because it has a lot of missing data. What we do know is that missing data can influence our findings and that is worrisome. What we don't know is when some missing data is too much missing data. That's the hard part to say, and it's a judgement call, like a lot of what we're talking about. Now some people say well, the amount of missing data was balanced between the treatment group and the comparison group. So, that's probably okay, right? Or, the amount of data was similar and when we looked at the characteristics of patients who had missing data, they were similar so it's probably okay to include these studies and assume there isn't bias. No, you cannot know, ever, if missing data has influenced your findings. You just have to make that judgement call ahead of time. You can certainly look at the data, see if it's balanced, see if the two groups are similar who are missing data, but it doesn't mean that everything's okay. So you might want to do what we call a sensitivity analysis, with and without that study, and see whether your findings hold up based on whether that study is included or not. So this post hoc definition of outcomes is becoming more and more important. We did talk in an early lecture about selective outcome reporting. And I think since 2004, so that's in the last ten years, people are realizing that this might be a real problem. That investigators are changing the outcome, sometimes quite innocently and not realizing what a bad idea it is, and sometimes not so innocently. They're changing the outcomes based on the results that they find. Now I gave you one result of measuring pain in two ways, and selecting one because it had a beneficial effect shown where the other one did not. Another way that outcomes can be changed, is you can say, well we're going to change the time point at which we measure the outcome. That's our primary outcome. So a great deal of emphasis is being put now on what's the primary outcome that was pre-specified before the beginning of the study, and what are the secondary or other outcomes that were pre-specified. Now, I'm not going to get into all the details, but by pre-specifying these outcomes, you are protecting against what we call type 1 error. And by pre-specifying, we're saying this is the main focus of our study. This is the outcome we use to estimate sample size. And type 1 error that is saying that there's a statistically significant difference when there really isn't a difference between the two intervention groups, you do not want to make that kind of error. And, so by switching the outcomes, you're putting yourself at higher risk. You could have a hundred outcomes you're examining, for example, the associate between your intervention and a hundred different outcomes. And, by chance, you're likely to find an association then with five of the outcomes and the exposure or intervention that you're looking at. So, that's why you don't want to look at many different outcomes that are not pre-specified. I'm not going to go into a lot detail. We're happy to talk to you about this further, but this is something from epidemiology 1, and your first course in statistics. So let me give you some examples though, of the problems we're up against, when we look at data either in journal articles, or for example, the data from clinicaltrials.gov. This is from a paper published in 2011 by Deborah Zarin that describes clinicaltrials.gov and the outcomes that are specified in that database. So that database asks the investigators to say what are your primary outcomes and what are your secondary outcomes? Now what we would call an outcome is called a domain. That's the big picture for example. Visual acuity, or all cause mortality, or quality of life. That's your primary outcome domain. And what they found is, that a median of one primary outcome domain was specified per trial. However, the range of primary outcomes specified was 1 to 71. That is, some people specified almost 100, three quarters of 100 primary outcomes. Well, as you know, if you test for an association between your intervention and 71 primary outcomes you're highly likely to find a statistically significant association just by chance. What about secondary outcomes? There's a median of 3 secondary outcome domains specified per trial and the range was 0-122. This is too many. I don't know what the right number of primary outcomes or secondary outcomes is, but it's not 122, and it's not 71. It's a much lower number than that. And there's a lot of debate going on right now about what is that correct number. It's very hard to say, especially now given that we want outcomes that are relavent to doctors, to patients, and they could be measured in the short term as well as over the long term. I'll say a little bit more about outcomes because it is such an area of concern in modern times. There's not much detail that people go into about what their primary outcome is. And this is concerning because you want to know how they measured that outcome. You might be doing a similar study and want to know what outcome measure to use. But in doing a systematic review of meta analysis you definitely want to know whether these data can be combined. So for example, in outcome measure descriptions that were on clinicaltrials.gov, the descriptions went from very vague, such as anxiety to a little less vague. Hamilton anxiety rating scale to even a little less vague than that, Hamilton anxiety rating scale at 12 weeks to something that was much more detailed and much more helpful to be truthful, such as proportion of participants with a change of greater than equal to 11 points at 12 weeks from baseline on the Hamilton Anxiety Rating Scale. Now one still wonders what baseline was, but okay, that's pretty good. So if you're doing a systematic review of meta analysis, you want a lot more detail than anxiety or Hamilton Anxiety Ratings Scale. You want to know when it was measured, what was considered a change if you're looking at a change, it was a change between when and until when. And so you want a lot of detail to help you do your meta analysis. Here's another example. What clinicaltrials.gov does, and now were expecting this for systematic reviewers as well, and you will see something like this in the Cochrane Handbook, is that we're expecting people to talk about the domain. As I mentioned, it could be anxiety, visual acuity, pain, that sort of the big picture of what your outcome is. How it was measured, anxiety could be measured by the Beck Anxiety Inventory, the Hamilton Anxiety Rating scale, a fear questionnaire. What metric they used. In the example I just gave, they use a change from baseline. But, you can also say, time to event, like, time until death you might use, if death is your domain outcome. Some n value, you might not use change, you might use whatever the rating is at that anxiety rating scale at 12 weeks, for example. How you aggregate the data, are they categorical data or are they continuous data. Are you going to look at a proportion or are you going to look at a mean or a median. And in our example we use the proportion with a decrease not of greater than or equal to 50%, but it was, I think, greater than or equal to 11 points. And then you want a time point. And that time point might be one year, it might be six months, it might be one month, it might be eight weeks. And you want a time point. In clinicaltrials.gov, time point is not a separate level. It's not what's called a separate element, but it's an overarching element which each of these elements have to be measured. But I think the movement now is to move it to five separate levels, so that each outcome has to be specified in these five separate ways. This is more detail than I'm sure you're asking for at this point. And that has directly to do with the risk of bias, but I think it helps you to see how one could change an outcome quite easily without anybody knowing, if you don't report very much about when or how you measured your outcome. So again, this is from the same article in the New England Journal of Medicine about clinicaltrials.gov. And what Debra Zarin found in this study was that only about a third of the studies reported on clinicaltrials.gov reported the domain only. That's really not very much. We hope that you get more than domain if you're trying to avoid bias in the analysis. If you really want to know what that outcome was. Was it pre-specified, and exactly how was it pre-specified? About two-thirds, nearly two-thirds, did specify a time point. That's very good, but we didn't know much about how the outcome was measured, the metric that was used, or the method of aggregation. And so I think you can see now why outcomes and selective outcome reporting have become such an interesting question for those that are concerned with the risk of bias in the individual studies included. This is a huge problem and something that people are trying to address in a number of different ways, including open access to clinical trial data. In that way, if this information is not reported in the journal article or in clinicaltrials.gov one could theoretically go into the original data set and find out. Why don't we take a short break and when we return, we'll talk about assessing the risk of bias in observational studies. I've been talking about randomized clinical trials, and we'll move on to observational studies after the break.
