We're going to talk in this section about how to display study quality in your systematic review. And you may remember, I said that there's no such thing, really, as study quality since there's so much that can be measured and unmeasured in a study. And also since we're depending so much on the study report, we don't actually know what happened in the study. Nevertheless, you're going to want to somehow display what you found about study quality, or risk bias, or controlling for minimizing the risk of bias in an individual study, because this is a concern that most readers of systematic reviews and meta-analyses have. That is they're concerned about garbage in, garbage out. And they don't want your systematic review to be full of garbagey studies. So the first thing I want to say, and I'm saying it really loud, and that's why it's in red, is please do not use quality scores. Quality scores were used 20 years ago. You will even still see them used in Cochrane Reviews, which I don't understand at all. But quality scores are not used anymore. It's tempting. It's really nice to assign a score to each study based on what quality you think it is. But the fact is, they don't work, and I'm going to tell you a little bit about why not. There's some quality scores that are what we would call generic. That is, you can use them for all research studies, certainly all research studies of a certain type. That is, you might assign a score to randomized trials, that would say, for example, was there random sequence generation? Was there allocation concealment? Was there masking? And it was an intention to treat analysis done. And then, give it a score of four if the answer is yes to all those questions. The score is what the problem is, not the individual question. You will also see some quality scores that are specific to a particular field like emergency medicine or cardiovascular disease. But again, you don't want to use a score. It's fine to use a particular system for assessing the risk of bias, but not a score. And one of the reason is, how do you find out whether this is a valid score. What does four mean? There's no way to validate that score. All you can do is look at the individual questions that are being asked, and whether the study appears to have addressed that question or problem correctly. You can't assign a number to it. These numbers aren't reliable and they can't be validated. So if we see any quality scores in your systematic reviews, we will not consider that good. This slide shows an example of the Cochrane, risk of bias summary. What it shows along the side is six studies in how each was assessed in terms of the risk of bias in that particular study. And the elements that were assessed are adequate sequence generation, allocation concealment, blinding, in this case, for subjective outcomes and mortality, incomplete outcome data assessed, and short-term outcomes and long-term outcomes, and then two other categories. So the first study, for example, Barry, 1988, was judged to be adequate, based on that plus sign on adequate sequence generation, but not adequate based on the minus sign under allocation concealment. Barry was also adequate based on the plus signs for blinding both for subjective outcomes and mortality, but not adequate based on the minus sign for the other possible elements of risk of bias, and so on. And if you sort of squint and look at this table overall, you can see that all of the studies use adequate sequence generation based on the fact they all use plus signs. That's actually to be expected because in a Cochrane review, often, randomization is a requirement to be included. But for allocation concealment, three of the studies had adequate allocation concealment. Two had question marks, therefore was unclear whether they did. And one, the Barry study did not have adequate concealment. So, this is a good way to demonstrate whether your studies are at risk of bias, and if they are, at what risk of bias. So, I can see, looking quickly, that the Goodwin study faired pretty well. Well, the Cooper study probably didn't fair very well, and Sanders, as well. Now, this is just presenting the data. It's not saying anything about how it's going to be handled in this particular systematic review. I did mention that sensitivity analyses could also be done where you do the first analysis with all the studies that you've included. And then you withdraw studies based on some element. For example, risk of bias, the method of randomization, allocation concealment, masking, intention to treat, and so forth. So, you could withdraw studies and see whether that estimate stays sort of firm, even when the studies that do not meet the minimum risk of bias criteria are withdrawn. So, here are the types of questions that you can ask in a sensitivity analysis. What happens if I exclude studies at a high risk of bias? What if I use different inclusion criteria? What if I use different exclusion criteria? What if unpublished studies were included or not? What if conference abstracts were included or not? And what if a study lost a different proportion of randomized patients to follow-up? So, what if I excluded all studies that lost more than 15% of the randomized patients, what would happen to my results then? So if my conclusion remains robust no matter which studies I withdraw, assuming that each time it's a fairly small number of studies, then that's a good sensitivity analysis. And that says that my conclusion is robust no matter which studies I include and one is reassured. If in fact conclusions change when you take away some of these studies, for example, the unpublished studies, one might be concerned about possible reporting biases that should be taken into consideration. So I said earlier in this lecture that I would mention the CONSORT and STROBE statements. These are statements that are very important to follow these days for proper reporting of clinical trials and observational studies. They don't guarantee that a study was as well done as the report may indicate, but it certainly is a good beginning. You wouldn't choose not to report a study, well, just because of the flaws it might be hiding behind the mask of a good publication. So hopefully the studies that you include in your systematic review will adhere to the CONSORT statement or the STROBE statement. Or some other statement depending on the type of study design that you use. The CONSORT statement, this is from 2001 and it's constantly being updated, but it's stayed pretty much the same over the years, and it's for parallel-group randomized trials. One of the most important parts of the CONSORT statement, is that every randomized trial should include a flow diagram, of how patients progress through the phases of a randomized trial. And this can help you a good deal. So you see how many patients were randomized, how many actually took the treatment they were assigned, how many were lost to follow-up at each stage, and how many were handled in the analysis. This is very, very helpful if you're doing a systematic review. And so I wish you all, if you're doing randomized trials, that the studies you find adhere to the CONSORT statement. The STROBE statement is for observational studies. It's a very long article. This is just the statement itself. But what's called the E and E, the explanation and elaboration, goes into why each element is there, and is very, very helpful for learning. So this is guidelines for reporting observational studies. So in summary, what I've talked about here is that what we all want to know, which is that the systematic review includes studies that were generally of a high quality cannot be assessed. That is, there is no way to really say that a study is of high quality, of low quality. We can't assess a risk of bias. And particularly in randomized trials, this is pretty well worked out by the different elements of risk of bias that I've talked to you about. With observational studies, it's more difficult to assess the risk of bias, and has a lot to do with how the exposed groups are defined and selected. And how the outcomes are defined and determined. And it's much harder then to assess personal bias in an observational study. But we will certainly help you however we can. So the quality of a study is related to its internal validity. An internal validity is how well the biases have been avoided, how well the study is designed and carried out. We don't know a lot about how a study is carried out. We can only determine what is written about it either in the journal article or in ClinicalTrials.gov. But nevertheless we do have clues in these design elements, in its internal validity. We've learned that quality scores and scales have problems, mainly because they can't be validated. There's no way of saying what a particular score means. Nor that anybody else would assign the same score. And instead, what we use today is reporting of the components. That is, how was assignment to treatment done, was that assignment concealed until the moment that it was revealed. Were patients and doctors and outcome assessors masked to the intervention that a patient got? How was the comparison group chosen? How is a case group chosen? How is exposure defined? How was the outcome defined? So there's no question that even though we have a hard time defining quality, and we use risk of bias as our way of measuring quality, that this has to be considered in interpreting the results. This is what everybody is concerned with in a systematic review, a meta-analysis. We do not want garbage in, garbage out. And so, what we do to try to use a quality filter is that we use a measure of risk of bias, either to decide whether to include a trial in a systematic review and meta analysis, or to exclude it. And we also perform sensitivity analyses, where we do our analyses with and without a study where we might have some questions about bias. So in that way, we account for individual study quality in a meta analysis. That's all I'm going to say in this lecture. I think it's a lot to take in. We are concerned about bias in individual studies, and how that can affect your meta analysis in particular. There's a lot being written and a lot being learned all the time. So I would advise you, if you're interested in this area, to keep up with it. It's a very important area in terms of the legitimacy and validity of systematic reviews and meta analysis. [MUSIC]
