There are many factors you have to consider when performing an analysis. The first step in any data analysis is the exploratory data analysis, often referred to as EDA. With the EDA, you get a feel for the data and identify issues that may influence the analysis such as identifying outliers, evaluating distributional assumptions, looking for potential transformations of the data that are required, and also doing a data quality assessment. The methods used for this analysis and are generally summary statistics, plots, distributional tests; not formal modeling. Once you get to the modeling, there are a number of factors to consider. But always keep in mind that your model must answer the question of interest. The examples I give here apply not only to clinical trials, but other forms of analysis. You need to think about the comparison that you want to make. Is that an absolute comparison? Are you measuring things on the relative scale? This is like choosing between a difference of the raw value and a ratio between values. You need to think about your outcome type. Is it continuous? Is it categorical? Are you looking at events that occur after a certain time, either a single event or an event that could be repeated often? What are the assumptions you're making with the modeling and test that you perform? Are they reasonable? Do you need to consider a non-parametric option that doesn't rely on assumptions? Or how do you choose between two different model types? For example, both Poisson and negative binomial regression can be used for repeated events. What is the frequency and type of missing data? One question that is especially important when analyzing trials is whether or not to include baseline factors in your model. This may include a stratification variable, which is used to help balance the characteristics between the two treatment groups or other clinically relevant factors. The goal is to reduce the variability of your model. For example, if you include the baseline value of the outcome, then the change from baseline will have less variability and need a smaller sample size. Another factor to consider is the relationship between the intervention and the outcomes. What is the shape? Does it follow a line or is it quadratic and follows a curve? Or are you interested in the visit by visit estimate? Finally, you should always be aware of the potential for correlation between your measurements. This could include repeated measurements over time, or paired measurements within an individual. For example, the right and the left eye from an individual. There could also be correlation between the outcomes that you are measuring. All these factors have to be taken into account both when you're choosing your analysis plan and when you're executing it. When you're actually executing your analysis techniques, which you need to pre-specify, don't just run the model, create plots, and perform tests to verify that the model you chose was appropriate. This could include things like residual plots, tests of proportional hazards. Be sure to present the whole scope of the analysis. This includes the estimate, your measure of uncertainty, like a confidence interval, as well as your hypothesis testing, such as a p-value or posterior probability. Now we tend to focus a little too much on p-values in the exclusion of these other factors. But all of them are important for understanding your results. It's important to consider the scientific relevance as well as the statistical significance. For example, with a large enough sample size, we could detect a difference in survival of one day. This might be statistically significant, but scientifically it's not very meaningful and probably wouldn't matter to a patient. Graphical techniques can be an extremely valuable tool. They're often easier to interpret than a large table of values. You can have a plot over time instead of a summary of the means over time. You can also summarize patterns that are not easily quantifiable. However, it's important to choose the right tool. The example shows two different ways of summarizing the longitudinal pattern of visual acuity for 15 eyes. The spaghetti plot on the left shows the individual VA over time for each eye. As you can see, it's a bit of a mess. The lasagna plot on the right has a line for each eye. Each visit is shaded according to the visual acuity of that eye. It ranges from dark green, which is very good vision 20/40 or better, down to gray, which is light perception or hand motion, and has red letters indicating adverse events. It is sorted so that eyes with better vision are at the top and eyes with worse vision are at the bottom. This pattern is much more clear than the spaghetti plot on the left. You can see that a large portion of individuals have very good vision, with just a few having very poor vision from the start of the trial. Missing data is always a problem in analysis, especially in clinical trials which due to their longitudinal nature are more prone to miss visits or missed follow up. The FDA created a panel on handling missing data in clinical trials to develop guidelines and the results of this panel's findings were published in 2010. Originally, the panel was going to discuss analytic techniques. But as they met, they considered that as important were methods of prevention. Now, these methods of prevention can include the trial design selection, things like target population, the outcome to be used, whether or not rescue medication was allowed , trial strategies, the choice of sites, how much burden the participant faces, training and monitoring. Now of course they did want to address analysis issues and their concerns were bias, loss of power, and the inference population. There are two factors to consider. The first is the analytic approach. Tools such as multiple imputation can be used. But the second and equally important is the missing data mechanism. The relationship between why something's missing and the outcome and treatment. The underlying pattern of missing data will determine whether or not your analytic approach is appropriate. They came out with some key recommendations. First, to include specifications in the study protocol and analysis plan for how missing data will be handled and what assumptions are made about the patterns of missing this. Second, to consider sensitivity analysis, which use many different methods for analyzing the missing data and assumptions to see if they're all consistent. Another issue in clinical trials is concern about multiple testing. The more tests you do the more likely you are to get a false positive. The plot on the right compares the number of tests performed versus the probability of having false positives. There are three lines. These represent the chance of one false positive on top, the chance of two false positives in the middle, and the chance of three or more false positives at the bottom. As you can see the risk of one or more false positives increases rapidly with the number of tests that are performed. There are methods to control the chance of a false positive statistically. I've given examples of a couple of the more common ones below. Perhaps the most well-known is the Bonferroni test, and it's very simple. You simply take your Type 1 error and divide it by the number of tests. That way, the total Type 1 error at the end is the level that you wanted it to be. The intersection-union tests and the gatekeeping procedures are less well known. The intersection-union test tests all of the hypotheses at the same level, but we'll only declare them to be significant if all of them are significant. The gatekeeping procedure requires that you order your hypothesis tests and perform them in that order stopping once you have the first nonsignificant tests. Now, these methods can seem very clunky and mechanistic and depend very much on how many tests you perform. There are other things you may want to do either in place of such control or in addition to it. First and most importantly, is that you should always report the number of tests that you perform and give an estimate of how many false positives you would expect. Do not just report tests that are statistically significant. Focus on parameter estimates, variability, as well as p-values to get the whole scope of what you're looking at. Be cautious in interpreting the results. It's not so much whether you do the test or not but how you report it. Control the Type 1 error on those tests that are meant to be definitive and give you the final answer. [MUSIC].
