[MUSIC] Okay, so a third potential explanation of this decline effect is exceedingly simple, but exceedingly common. And this may be the most important one to learn to apply in your own work when you're doing statistical analysis. Okay, and so this is the idea of Multiple Hypothesis Testing. So the problem here is that if you perform experiments over and over and over again, you're bound to find something, right? That's sort of the definition of that. If you keep rolling dice long enough, you can't just keep rolling dice and then yell Yahtzee, right? You get one shot at it. And the same is true with these experimental design. Okay, and so this is related to the publication bias problem, in that you're only showing your positive results, but it's a little bit different. Because here, you're talking about the same sample, and you're testing different hypotheses, over the same data. And so in these situations, either you shouldn't do it at all or if you do have to it for various reasons, you need to adjust the significance level down. That means you do not settle for 0.05 as the threshold. You need to do something much much lower. Okay. So, to understand why, consider something pretty basic. Over completely random data. And we set the threshold at 0.05, it's alpha at 0.05. So the probability of detecting an effect where there is none is 0.05. And the probability of detecting an effect when it exists is 1 minus alpha. Then a probability of detecting an effect when it exists on every experiment you do at experiments is 1 minus alpha times 1 minus alpha times 1 miles alpha times 1 minus alpha, assuming that they're independent. It's okay to multiply probabilities together if those probabilities are independent. Finally the probability of detecting an effect where there is none on at least one experiment is one minus that total, right? So first build up the probability of being perfect. And then 1 minus that is the probability of not being perfect without making at least 1 mistake. So, if you plot these numbers what you get is, on the X axis here is the number of tests and the Y axis is the probability of at least one spurious finding. Right? And making at least one mistake. Well, it goes up like this. As you get up to 50 hypothesis tests, you're up at the 90% chance of at least one spurious finding. Okay, and so controlling this is known as controlling the Familywise Error Rate. This is the Familywise Error Rate of at least one mistake. So this is a pretty stringent constraint. So how do we correct for that, how do we control the Familywise Error Rate? Well, one solution is the Bonferroni Correction, which is just to divide by the number of hypotheses. Okay, so if your significance level is alpha, 0.05, and you do 20 experiments, you're testing 20 hypotheses, you just divide 0.05 by 20. So another correction is the Sidak Correction, which has this extra condition where the tests need to be independent. So even though we talked about in the last slide that in order to make that plot, we were assuming that they were independent, but the Bonferonni Correction in general does not need to assume that. Okay, so if you're doing hypotheses test that are related to each other, you can still do the Bonferroni Correction. However, to the derive this Sidak Correction, we're gonna rely on the fact that we're gonna multiply the probabilities together. Whenever you see probabilities being multiplied together, that means that you're assuming they're independent. Okay, so let's see if we can build this up. [MUSIC]