[SOUND] This module, we're going to talk about the basics of what we can and can't say with FMRI data. And we're going to do this through the lens of crises in psychology and neuroscience. There are many surprising claims that have been made in the literature, claims like this one, from the New York Times. In this study, which is written up as a New York Times op-ed piece, the authors claim to have scanned a number of brains of Democrats and Republicans, and they claim to be able to make inferences about what somebody's attitudes are, depending on what their brain scan show. So for example, they write, the two areas in the brain associated with anxiety and disgust, the amygdala and the insula, were especially active when men viewed Republican, and based on that, they're inferring that the voters sense peril and threat and disgust, and feel disgust. And also emotions about Hilary Clinton are mixed, based on activation in the amygdala and the nucleus accumbens, which is supposed to be related to reward, and so on. Here's another surprising claim. In this article, Martin Lindstrom scanned people viewing their iPhones. And what he writes is most striking of all was the flurry of activation in the insular cortex of the brain, which is associated with feelings of love and compassion. It was discussed in the previous article. Subjects brains responded to the sound of their phones as they would respond to the presence or proximity of a girlfriend, boyfriend, or family member. The subjects love their iPhones. Many of these claims are demonstrably false, others are unlikely to be true, and this isn't limited to brain imaging. This is an endemic problem, in psychology, psychiatry, neuroscience, genetics, biology and other fields. And these are illustrations of some of the more extreme examples of what I would call the replicability crisis. This is one of three crises in neuroscience that we're going to talk about today, with a view towards how to avoid them, and how to educate ourselves to make better and smarter inferences about brain images. The replicability crisis is fueled by findings that are not true and effects that might be true, but are not meaningfully large or not reproducible from laboratory to laboratory. The interpretability crisis concerns findings that might be true, but they're not meaningful in terms of underlying neuroscience and what we know about brains. And finally, the translation crisis refers to this idea that we've done many many studies, but it's very difficult to take brain imaging research, like many kinds of research, genetics research and other research, and bring that from science into the practical domain, where we actually have commercial and clinical applications. So this [LAUGH], yeah. But what we'll see next is a funny example of one of the studies recently that's really fueled this debate. And it's a study of extra sensory perception published in one of the most prestigious journals in psychology, General Personality and Social Psychology. >> According to a study by Cornell Psychology Professor, Daryl Bem, soon to be published in the Journal of Personality and Social Psychology, there is quote, strong evidence for extrasensory perception, the ability to sense future events. I know you're thinking, Stephen, that's bulls [BLEEP]. >> [LAUGH] >> But on the other hand, I know you're thinking, Stephen, that's bulls [BLEEP]. >> So many of these surprising claims are not true. And how do we know that they're not all true? Any one of them could actually be a surprising but true finding. And one way that we can know is to look at studies across different fields, across the sciences. So here you see plot from studies from the space sciences, and geo sciences through to social sciences, immunology, climatology, psychiatry, psychology down at the bottom. And what we're looking at, is the proportion of papers that claim to support the hypothesis they tested. And these claims, well, so what's the base rate of having a hypothesis that you end up supporting? It's probably low, but here most studies, and in fact in psychology and psychiatry, nearly all studies end up supporting the claims that they tested. And one explanation is that they're choosing the claims to support, to fit the data. So this is related to what's called the studies publications bias or the File Drawer Problem, which is a problem throughout sciences. And the idea is that studies with negative results go in the file drawer, you never see them, but studies that happen to find significant results, even if they're not true or not replicable, end up getting published. And because of this false findings are like rumors, they're easy to spread, they're hard to dispel. And one positive finding can cause a ripple that ends up influencing the field for years to come. And if those findings are false, this is obviously a dangerous practice. One of the most famous examples of this is recent controversy over whether autism is caused by vaccines. And there was one publication that claimed to had find this. It was picked up by the media and by several celebrities and spread very widely. And in spite of many, many large scale studies failing to find such associations, it continues to be an idea that's prevalent among certain sectors of the community, and there's a lot of harm that's caused by that. The backlash against these replicability issues has been quite strong and it's not always very specific. So in this article, Ingfei Chen is saying that the vast majority of brain research is drowning in uncertainty. Not just a few isolated studies, but many, many areas of neuroscience, and while there's some truth to this, there's some danger in this also. Here are some of the important papers that look at sample size issues, that look at issues of finding interactions and complex effects where it might be difficult to demonstrate such effects, and really critique then of all kinds of research by John Ioannidis. What you see here, Why Most Published Research Findings are False. So, these claims about replicability issues have been applied really broadly across fields. And this is one of the papers that's been influential in the brain imaging field. It was originally called Voodoo Correlations in Social Neuroscience, which was a targeted attack really on replicability and social neuroscience studies that look at correlations between brain activity and measure of personality, or attitudes, or behavior. And they subserviently changed the title to Puzzlingly High Correlations in Social Neuroscience, but it really caused a big stir. And Sharon Begley was one of the early advocates of brain imaging research in the news who covered a lot of these stories, and here she writes the neuroscience blogosphere is crackling with- so far- glee over this paper. And it really it made a big stir. One question is whether this is caused, that these problems are caused, by a few bad apples. So one example recently is Diederik Stapel, who had a series of very high profile findings in science and other journals, and subsequently was found to have committed fraud and admitted to fabricating data from a number of papers. And this is a terrible thing, but it's unfortunate that it happens once in a while, and we can ask, are all these problems really just caused by a few scientists who are overtly unethical? And I think this is not the case, this isn't just caused by isolated breaches of scientific integrity. It's caused by really a widespread bias even by well-meaning scientists. So, we have to educate ourselves on what the sources of bias are, and then how to work so that we minimize those sources of bias and can overcome them. There are many reasons for this, but let's look at what Ioannidis says about why most published research findings are false, and what are the factors involved. So here's some risk factors for false positives. One is smaller studies, and when we apply this to neuroimaging studies specifically, neuroimaging studies are very expensive to collect and analyze, and so they have been quite small by comparison to other studies. The low prior probability of the effect being true, or at least very large, the ESP example that we looked at from Bem's research is one such example. Some people believe in ESP, other people don't, but there's no known mechanism and even Bem says he doesn't know how it works. And if there's no known physical mechanism, then it's less likely that any phenomenon is true a priori, and that makes findings that they're surprising in this way. That find things with no mechanism less likely to be true. So we need more evidence to demonstrate that they are, if indeed they are. Three is the number of tests conducted. We'll talk more about this in this lecture and the next lecture, but in neuroimaging, we have many voxels that we test, sometimes hundreds of thousands of voxels. We can test multiple contrasts, multiple maps and so the risk of false positives is very great. And in addition, when we pick out the winners, the effect sizes of those voxels look much larger than they actually are. The next factor is flexibility in design, analysis choices, and outcomes. And in neuroimaging there are many analysis pathways and options. And we'll look at some of the impact of that and false positive findings as well. Next, is financial interest. So, it's financial interest and also prestige interest. We all have a stake in supporting the things that we want to be true. And many of us have a stake, our status, and maybe our self-worth, is on the line when we test our scientific findings and we want our favorite effect to pan out. So we really have to guard against this. It's costly to be wrong in many cases, even if we don't think we're biased and wehave the best intentions, and we really have to keep that in mind. And one of the interesting related findings related to this is that there was a meta-analysis published that showed that significant predictor of how large a drug effect is in a clinical trial, is whether a drug company funded that study. And finally, the competition to publish. In hot fields, there tend to be more false positives, and possibly in more prestigious journals as well. So don't we all want to publish in Nature and Science. Well, to publish in Nature and Science, I don't know how to publish in Nature and Science, but if I did, I might say that you have to have something that's both very surprising. You have to have a very surprising finding. Let's look now at selection bias in more detail, and this is going to be the core of a number of problems that we have to deal with in neuroimaging, and they're resolvable. And so first of all, studies and tests with positive findings are reported and those without positive findings are ignored. This is called the file drawer problem. But there are multiple types of selection biases, and they include the publication bias that I talked about a minute ago. Also biases in experiments, inflexibility in which subjects to include, how many subjects to run, when you stop running subjects, and all those sources of flexibility, if you're choosing them after you observe the data, are sources of false positives and increase inflated effect sizes. Flexibility in the model, which outcome do you choose, which covariance do you include, and which modeling procedures do you employ? And finally, the problem of voxel selection bias. So which voxels do I choose out of the many voxels that I tested to actually look at and interpret. So this is the interpretability crisis now, we'll talk a little bit about this. And this relates to findings that might be true but are not meaningful. Here are a couple of the classic papers that deal with inferences about cognitive neuroscience, and what we can infer about function from looking at brain activity. And the first one is a really old paper now by Martin Sarter and Berntson and Cacioppo called Toward Strong Inference in Attributing Function to Structure, and they discuss some of the difficulties in this inferential process. Foreshadowing a lot of these debates later on. And secondly, this is an article by Max Coltheart, which spawned a series of responses, and what he asked is, perhaps functional neuroimaging hasn't told us anything about the mind so far. He challenged the field to come up with an example where brain imaging supported a cognitive theory, or was critical evidence in deciding whether a cognitive theory is true or false, or which theory is true. And there were number of responses, and there was a redux of this later by Mara Mather and colleagues. And I invite you to go and look up some of those papers for many interesting answers, and we'll talk about a couple examples later in the course.
