In the final section of this lecture, I'm going to talk about preparing data monitoring reports for randomized clinical trials. As I mentioned, this is not talked about a lot. It's sort of a secretive process partly because of the nature of it where monitoring the data from a trial as it accumulates and we don't want to let that information out. But because it's such an important component of conducting trials ethically, it's important that we discuss it more. So I'm really happy to give this presentation. As I mentioned before, it's an ethical imperative to monitor an experiment in humans. Once we have definitive evidence of an inferior intervention, we want to stop the study or we may want to stop the study or modify the study for other reasons, but we have to be looking at the data. I'm talking about it again from the perspective of a multicenter randomized clinical trial where we typically organize a group of independent experts which is called the data and safety monitoring board, DSMB. But regardless of the size of your trial, even if it's a phase 1 trial, all trials need monitoring for safety as they're conducted. The degree of monitoring can vary and depends on the actual details of the trial, but all trials need some type of monitoring. You're doing an experiment in people. For multicenter trials, the DSMB usually has a charter that outlines the frequency of the meetings, how often they'll be reviewing the data. A common for it to be twice a year in a major study. It also has the monitoring plan, what are they going to look at at these interim looks at the data, including any statistical plans for an interim analysis and the plans for the final analysis. The data monitoring report is usually divided into two parts, an open and closed books. In the open part of the report, there is data on the performance of the trial, things like how are we doing on recruitment and data collection. In the closed part, the data are displayed by treatment assignment and we look at outcomes. These books are mostly tables and figures, and I'll be showing you lots of examples of tables and figures. What should be included in the report? We want information on the study progress. The Achilles heel of most trials is recruitment. That is the major reason trials fail, so we need to be looking at that and whether we are able to retain the participants and follow up. The data quality in terms of completeness of the data and other metrics and other kind of features, whether they're adhering to the protocol and whether there's protocol adherence, and also looking in the individual records to see if the data are making sense and hanging together. The primary reason for monitoring, though, is to assess the benefit and risk profile. Are they the same as when we started the trial when we were at a point of equipoise, where we thought we didn't really know what was the better treatment? In monitoring that, we need to look at risks associated with one or more of the interventions. Sometimes there may be specific concern. We also have to look at serious adverse events which is a regulatory term that indicates any event that caused someone to be hospitalized, or died, or otherwise was significant. In terms of benefits, we look at the interim analyses of our primary outcome. Just as we designed the trial to see if one treatment was better than the other, we're looking along the way to see if indeed we prove that already or it looks plausible that there will be that benefit. There can be special issues unique to a study. It might be because of the population that's involved in the study, that it's children, or there may be a particular safety concern. For example, I've been involved in trials of drug for COPD that had some signal of causing depression and suicidal ideation. So during the trial when we were using the COPD drug in a group of people with asthma, we had to be very vigilant in ensuring that there was no signal with regard to an increase rate of depression or suicidal ideation in the group randomized to receive the drug. There's wide variability in reports. Sometimes they're just long listings of events with narratives that occurred during the trial, and sometimes they, as I will show you in this presentation, are organized tables that help you interpret the data. I have to say, this is one of the places that publicly funded trials have really led the way in safety monitoring. Pharmaceutical companies have adopted the model that was created by publicly funded trials conducted by the National Institute of Health. These books are prepared to be reviewed at a meeting. That's why you can get away with just tables and charts mostly, but you have to look at what's the agenda for the meeting to know what you need to put in the book, which should reflect the things that we should be monitoring for. Let me point out that, first, these meetings can have four parts, that they may have an executive session in the beginning for the members to get together and see what they think is important after they have reviewed the materials individually and now we're coming together to review it, an open session that is open to the investigators conducting the trial and other staff members that is going over the status of the trial and things like enrollment and retention, protocol deviations and data quality, and then a closed session that is restricted to the membership of the board, a statistician, maybe some other people managing the data but no one who was in direct contact with patients or makes any treatment decisions. This is where we review data by treatment assignment. After that, the DSMB then usually excuses all the other people from the meeting and has a meeting among themselves to discuss what they think of the data and come up with any recommendations they have for the investigators in terms of continuing the trial. One of the most important things that the DSMB does is that the end of their meeting, if they've reviewed everything and based upon that review, they take a vote as to whether the trials can continue as designed or it can continue but needs to be modified or it should be stopped. Very powerful group. Matching that agenda is a notebook. Here's an example of the table of contents for an open book. I just want to point out that it starts out with the agenda. We also have the minutes from the last meeting. If at the end of the last meeting the DSMB had some recommendations, we have to tell them how we responded to those recommendations, what changes we made, or if we felt that the recommendation wasn't able to be implemented, why not. We also have some material about the design of the study basically because these folks come together twice a year to review a study that they aren't involved in. We want them to remember what the study is about. They're very busy people. They may be on several DSMBs. They may be involved in several of their own studies. It's a good idea just to have those facts very accessible to them. They'll be updated about any changes since the last DSMB meeting, and then the remaining sections are related to performance, recruitment, and data quality. As I warned you, I'm going to be showing you lots of tables and graphs because that's what we put into these reports. I've got four graphs on this slide that depicts different features of recruitment. The first graph that's labeled milestones, we've got a graph that has the number of participants and calendar time along the x-axis, participants on the y-axis. So the blue line indicates how we were doing in terms of recruitment, and the red dots are the milestones, what we had planned. We had planned to end this trial quite a bit before we actually ended the trial. If we continue, this next graph is the number of participants on the y-axis and week on the x-axis, and this shows you recruitments per week. So it gives you an idea of the pace of the trial. How are you doing? Are you speeding up? If you're way behind in recruitment, do you have a trend going upward? If we go where it's labeled by site and stratum, the graph has enrollment by site and by stratum. The most important thing is the overall height of the bars to show you how many people have been enrolled at each site, and you can see there's quite a bit of variability. The stratum is depicted by using two different colors and having a stacked bar so that we get the total number of people who are enrolled, but also can see the distribution of people in each stratum of the trial, which, depending on the design of the trial, may have more or less importance. Finally, we have a chart that shows you site activity. This is different in that the y-axis is not number of participants; it's number of days. The x-axis is the clinical sites. Each site like the bar chart beside it has its own line. The y-axis indicates days since the last enrollment. If you have a short bar for what we call these lollipops, but if you have a short lollipop, that means you have been active recently. If you have a longer lollipop, you've enrolled someone but it's been quite a bit of time. Using these four graphs supplemented with some tables, we can get a pretty good overview of how recruitment for the trial is going. Another figure we often include in DSMB reports is a figure that depicts the information progress. A little complicated, but on the y-axis here is the number of individuals to be enrolled in the trial. So the sample size for this trial is 267 individuals. That's how many people we hope to recruit. Along the x-axis is time but divided into visits. So here's the baseline visit you can see at zero weeks. Here's the second visit, that is at four weeks, all the way to the final visit at 24 weeks. Each line that goes across this graph represents a person enrolled in the trial. When we accumulate all these lines together in this graph, we can see how much information we've collected. Ideally, we'd get to 100 percent. That means that we would see 267 patients for all of these visits. But as it climbs, ideally, we would get to 100 percent, that we would have complete follow-up on all 267 participants. We can see right now that we're probably somewhere about 60 percent. We've got a little bit more than 40 percent of participants who have completed the trial, but quite a few participants that are in the early stages and certainly some unenrolled participants. The y-axis on the right side of the trial is the percent of information that we have collected. It can be a very informative look at how follow-up is going in a particular trial. We want to see a lot of gray, not too much white. A lot of the darker tone, shaded areas and not too many clear areas. We also look at participant follow-up by site and tables as well to get a more quantitative evaluation of how follow-up is going. You can see in this table, there is a row for each site and it shows how many individuals they screen for the trial, how many were eligible. They actually ended up randomizing 12 people. We can see that there are no active people in the trial. They've completed follow-up on 11 people, but there was one person who withdrew early. We can look at individual sites to see if there are any problems in terms of follow-up, and we can also look at the total numbers. Another depiction of follow-up here is a missed visit table. Again, it's by clinic. It has a column that says the number of visits that were missed versus expected, so we can get a missed visit rate for each site and an overall rate for the trial. Then we sometimes divide it into overall in the last six months because looking at the last six months may help us identify any trends in the data. For example, here we seem to have somewhat of an increase in missed visit rate from 3.4 percent to 4.3 percent. It may not be significant, but it incorporates the idea that the kind of pace of the trial and the data accumulation is not constant and can change over the course of the trial, and the investigators and the DSMB have to be alert to those type of changes. This is another table and the purpose of this table is to track data submission for centralized analysis. Again, we have it by clinic, and then we have different images. This is from an ophthalmology trial, so this is an OCT image, this is a fluorescein angiography image, a fundus reflex image. We just want to look to make sure that sites are submitting all of the photos we need to evaluate the outcomes and they're doing so in a timely manner. We can see that in the last column is the median days to receive an image from the day it's taken. Ideally, it to be low and we may want to address the issue at a specific clinic like the one in the top row that is taking almost half a year to submit photos. We also look at data quality based on edits of the data. These can either be done by if paper forms have been completed to check the paper forms against the electronic database, or we can be looking in the electronic database for inconsistencies in the data. Because now as we move more and more to direct data entry or direct data capture, we have less forms that we can do audit so on. But this gives you an idea. Again, we present it by clinic and try to give the DSMB an overview of what we're finding in terms of errors on forms and indicate which clinics are the most problematic by shading their errors differently and also looking at outstanding problems. This helps us pinpoint particular clinics that we may have to address some issues with, and it's very helpful sometimes when you have to deliver bad news to a site and ask them to improve their performance to have a DSMB recommendation as the cause for you intervening. We also look at protocol deviations. This is all under the guise of performance, so we're looking at how well the protocol was followed on the left side. We just have the numbers of protocol deviations. In this particular trial, we were particularly concerned about deviations that had to do with HRCT scans, so we separated them out. Because we saw in the beginning of the trial that a number of sites were having issues with doing the scans, so we wanted to follow them very closely. We also include more narrative listing of all those events so that the DSMB can actually determine what happened and evaluate the significance of particular deviations. Now we come to the closed book, and the closed book is the part where we're really trying to take a look at the benefits and risks and to see if it is still ethical and appropriate to continue the trial. The closed book may also have response to DSMB comments from the last meeting, depending on if those comments were about data from the closed book. It also has many of the characteristics that we are collecting data on, including the outcome by treatment group. I'm just going to show you some tables. It starts off with another table one so that we can tell the DSMB who were enrolling in the trial. No different from the table one that you might have in a paper. We can also look at the completeness of data collection by treatment group to see if there's any indication that we're having a differential and missing datas. This isn't actually looking at the values of the data, but it ensures that we're collecting information in both groups. Serious adverse events. This is a very important part of the DSMB job. To look at events that occur during the trial that either require hospitalization, some kind of medical or surgical intervention, long-term disability, life-threatening, or death, or a congenital anomaly. This is a regulatory criteria for what is a serious adverse event, at least the basics. We may add other events that are specific to a particular trial. For example, often in an eye trial, we'll add as a serious adverse event that has to be reported right away is significant vision loss. We have the columns again, we get the control group and the experimental group, the number of participants. We see overall that 18 participants in the control group have had one or more adverse events, none have had three, and they've had 20 events. Then we have the same information for the experimental group. Like at the protocol deviations, these tables of serious adverse events are backed up with summary tables describing the event and what's the status of the event in terms of hospitalization, whether it was related to the study drug, is the participants still taking the study drug, and what was the outcome as we know it. They'll also be detailed listings that really go into the individual details that were reported for each event. Finally, we sometimes have a formal interim look at the efficacy data. This is the point where we have a statistical plan for looking at our primary outcome data to see if it is still justifiable to continue the trial. I happened to pick an interim look from a trial that we stopped. A little confusing, but I'll try to walk you through it. On the left-hand side, we have a graph and this is a graph of the outcome data, which was percent change in retinal thickness. It goes from minus 100 to 100, meaning it can either increase or decrease. This is for people with macular edema, so preferably what we're looking for is for the decrease. We also have time from randomization at the different visits that they were looked at. In the small table below, it actually has the numbers that are included at each look, and then we have a legend here that describes the different groups. So each treatment group gets a different color, but they're always ordered the same way and we could also use patterns. But in this study, we happen to use colors. Basically, we're getting box plots at different time points looking at the retinal thickness. The lines because these are the same people in the first box at week 4 as the first box in week 8, we can connect those to see if those lines help us interpret. The table on the right provides the data the DSMB needs to interpret this graph and to see if indeed a stopping guideline has been met, which it had been since the two-sided stopping guideline for this trial was a p-value of 0.00132. You can see we had quite a few p-values that were at different time points that were less than the stopping guideline, indicating that the DSMB should consider a recommendation to stop early, which indeed they did. Some final thoughts on preparing data monitoring books. It is a very intensive review of the data. It's actually a great thing to do because you look at the data in great detail and it's very good practice for your final analysis that you'll put in the journal article. There's a lot of variability and procedures as I mentioned before and the types of summaries, whether the report is mass or blinded, and what is monitored. Is it just a safety review, and when is efficacy a measure of safety? That's why on the right, I have this cartoon of a coin that you see two different aspects of the coin. If you look at an efficacy outcome, well, on one side, it's whether it's beneficial, is this treatment beneficial? But if the treatment doesn't work as well as the control, it really then becomes a safety outcome because the treatment isn't working. It's really the perspective that you look at it, and I think that this is one of the emerging issues that we need to deal with in data monitoring. Thank you for your attention.
