0:00
The P-value is the most widely used statistic in the entire world
including for inference and for everything else.
Its so popular that if it was cited every time that it was used it would have at
least three million citations, making it the most highly cited paper ever created.
So the p-value is a very important statistic and since its such an important
statistic there are lots of people that hate the p-value because it's so popular.
And so part of the reason why people hate it,
is because people consistently miss interpret the p-value.
And so the p-value is defined as the probability of observing a statistic that
you've calculated.
That is extreme as you observed it, if the null hypotheses is true.
So a couple of the things that p-value is not and
that will make statisticians see red is if you say that the p-values
the probability that the null hypothesis is true it's not equal to that.
It's also not the probability that the alternative is true.
And in some sense it's not necessarily a measure of statistical evidence.
That's a philosophical term that people will worry about but in this case,
you need to interpret it very narrowly.
As the probability of observing a statistic as or more extreme than the one
you observed in the data if you would observe the null hypothesis to be true.
So here we're going to use that example again with the responders and
the not responders to illustrate what's going on.
So again, we have responders and not responders, now we're looking at say, for
gene one, a statistic that compares the responders to the not responders.
So we might calculate the T statistics to take the average expression level among
the responders, and subtract the average expression level among the non-responders.
And then standardize that by some measure of the variability, in this case,
the average variability in each of the two groups.
So in a previous lecture we learned that one way that you could try to
quantify a null hypothesis.
The null hypothesis that the distributions are exactly the same among
the responders and the non responders, is to permute the sample labels.
So when you permute the sample labels, you leave the relationship among the genes
unchanged, but you can look at the, you can break the relationship between each
gene and the responder non-responder label.
So if I recompute the statistic, after I do that, I get a distribution
under the permutations and then I have the original statistic that I calculated.
And so the p-value that I can calculate could be the number of
permutation statistics I observed to be larger than
the statistic that I originally calculated.
And I do that in absolute value since in general the null hypothesis is
that the value is equal to zero.
That there's no difference between the two groups.
But the alternative could be that it's either more or it's positive or
it's negative.
And so I have to look in both directions, whether it's positive or negative.
And so I just count up the number of statistics that are more extreme in
each direction, and I divide by the total number of permutations.
So I basically average the number of times I observed the statistic as or
more extreme under this null hypothesis as the statistic I originally calculated and
that gives me the p-value.
So this p-value is often used as a measure, but
in general it's basically used as a hypothesis testing tool to be able to say,
if that p-value is small, you're going to reject the null hypothesis.
Because the statistic is very extreme
compared to the distribution that you would have got under the the null.
So this is what p-value distributions look like for
genomic experiments that are done well.
So typically, you see a distribution like this where there's a spike near zero and
then there's a flat distribution as you move out here towards one.
So if you actually look at this and break it down into the different parts,
this part near zero, these p-values that are really small,
those are really the P-values that are coming from the alternative distribution.
Because remember,
the p-value is measuring the probability of observing a statistic more extreme
under the permutations than the statistic that you got when you observed it.
So if you observe a statistic that's very, very extreme, the number of null or
the number of permuted statistics that will be larger than that is very small,
and you'll get a small p-value.
3:59
So this is the sort of the p-values that you expect to be coming from
the cases that are not from the null distribution.
And then under the null, these are the p-values you get,
you get a flat distribution that goes out here to the right hand side.
So turns out that a particular property of the p-value is that it's
uniformly distributed, it's equally likely to be any value between zero and
one if the null hypothesis is true.
What does that mean in general?
It means that even if you get a small p-value,
it might be from the null distribution because there's an equal chance that
it'll be any value between zero and one if the null is true.
So this actually is a useful set of properties that can be used to
estimate things like the false discovery rate that we we'll talk about when
we talk about multiple testing.
But the basic idea is that this distribution is a mixture of two
distributions.
There's a mixture of the p-values that come from the null hypotheses, and
the p-values that come from the alternative hypotheses.
And the null hypothesis p-values are supposed to be uniformly distributed.
And the alternative ones should be pushed up towards zero.
They should be skewed away from one.
5:03
And so the p-values almost always go to zero with the sample size.
That's another common misinterpretation of the p-value.
Just because you got a really small p-value,
it doesn't mean that the difference is huge.
It could just be that your sample size is really large, and so
the variability is small.
Even if you have any difference at all, as the sample size gets big,
the p-value will get small.
The usual cut off that people use for calling p-values significant is 0.05.
This is if you're doing only a single hypothesis test, but
that number is basically just a made up number.
So it could be any other threshold could also be used.
I mean it's useful to have a standard, but don't treat this as sort of religious
truth that 0.05 is the right way to tell if your p-value significant.
And you should always report p-values in conjunctions with estimates and
variances on the scale that's scientifically meaningful.
P-values can be useful as a complement to that,
as a way to sort of quantify statistical significance, as long as you pay attention
to the properties of the p-values and interpret them correctly.
Jeff Leek, PhDAssociate Professor, Biostatistics
