This course covers the analysis of Functional Magnetic Resonance Imaging (fMRI) data. It is a continuation of the course “Principles of fMRI, Part 1”
Module 1A: Crises in neuroscience and psychology: Problems and solutions
Loading...
From the course by Johns Hopkins University
Principles of fMRI 2
85 ratings
From the lesson
Week 1
This week we will discuss psychological and behavioral inference, as well as advanced experimental design.
Meet the Instructors
Martin Lindquist, PhD, MScProfessor, Biostatistics
Bloomberg School of Public Health | Johns Hopkins University
Tor WagerPhD
Department of Psychology and Neuroscience, The Institute of Cognitive Science | University of Colorado at Boulder
[SOUND] This module, we're going to
talk about the basics of what we can and
can't say with FMRI data.
And we're going to do this through the lens of crises in psychology and
neuroscience.
There are many surprising claims that have been made in the literature,
claims like this one, from the New York Times.
In this study, which is written up as a New York Times op-ed piece,
the authors claim to have scanned a number of brains of Democrats and
Republicans, and they claim to be able to make inferences about
what somebody's attitudes are, depending on what their brain scan show.
So for example, they write, the two areas in the brain associated with anxiety and
disgust, the amygdala and the insula, were especially active when men viewed
Republican, and based on that, they're inferring that the voters sense peril and
threat and disgust, and feel disgust.
And also emotions about Hilary Clinton are mixed,
based on activation in the amygdala and the nucleus accumbens,
which is supposed to be related to reward, and so on.
Here's another surprising claim.
In this article, Martin Lindstrom scanned people viewing their iPhones.
And what he writes is most striking of all was the flurry of activation
in the insular cortex of the brain, which is associated with feelings of love and
compassion.
It was discussed in the previous article.
Subjects brains responded to the sound of their phones as they would respond to
the presence or proximity of a girlfriend, boyfriend, or family member.
The subjects love their iPhones.
Many of these claims are demonstrably false, others are unlikely to be true, and
this isn't limited to brain imaging.
This is an endemic problem, in psychology, psychiatry, neuroscience, genetics,
biology and other fields.
And these are illustrations of some of the more extreme examples of what I would call
the replicability crisis.
This is one of three crises in neuroscience that we're going to talk
about today, with a view towards how to avoid them, and
how to educate ourselves to make better and smarter inferences about brain images.
The replicability crisis is fueled by findings that are not true and
effects that might be true, but are not meaningfully large or
not reproducible from laboratory to laboratory.
The interpretability crisis concerns findings that might be true,
but they're not meaningful in terms of underlying neuroscience and
what we know about brains.
And finally, the translation crisis refers to this idea that we've done
many many studies, but it's very difficult to take brain imaging research,
like many kinds of research, genetics research and other research,
and bring that from science into the practical domain,
where we actually have commercial and clinical applications.
So this [LAUGH], yeah.
But what we'll see next is a funny example of one of
the studies recently that's really fueled this debate.
And it's a study of extra sensory perception published in one of the most
prestigious journals in psychology, General Personality and Social Psychology.
>> According to a study by Cornell Psychology Professor,
Daryl Bem, soon to be published in the Journal of Personality and
Social Psychology, there is quote, strong evidence for
extrasensory perception, the ability to sense future events.
I know you're thinking, Stephen, that's bulls [BLEEP].
>> [LAUGH] >> But on the other hand,
I know you're thinking, Stephen, that's bulls [BLEEP].
>> So many of these surprising claims are not true.
And how do we know that they're not all true?
Any one of them could actually be a surprising but true finding.
And one way that we can know is to look at studies across different fields,
across the sciences.
So here you see plot from studies from the space sciences,
and geo sciences through to social sciences, immunology,
climatology, psychiatry, psychology down at the bottom.
And what we're looking at,
is the proportion of papers that claim to support the hypothesis they tested.
And these claims, well, so what's the base rate
of having a hypothesis that you end up supporting?
It's probably low, but here most studies, and in fact in psychology and
psychiatry, nearly all studies end up supporting the claims that they tested.
And one explanation is that they're choosing the claims to support,
to fit the data.
So this is related to what's called the studies publications bias or
the File Drawer Problem, which is a problem throughout sciences.
And the idea is that studies with negative results go in the file drawer,
you never see them, but studies that happen to find significant results,
even if they're not true or not replicable, end up getting published.
And because of this false findings are like rumors,
they're easy to spread, they're hard to dispel.
And one positive finding can cause a ripple that ends
up influencing the field for years to come.
And if those findings are false, this is obviously a dangerous practice.
One of the most famous examples of this is recent
controversy over whether autism is caused by vaccines.
And there was one publication that claimed to had find this.
It was picked up by the media and by several celebrities and
spread very widely.
And in spite of many, many large scale studies failing to find such associations,
it continues to be an idea that's prevalent among certain sectors
of the community, and there's a lot of harm that's caused by that.
The backlash against these replicability issues has been quite strong and
it's not always very specific.
So in this article, Ingfei Chen is saying that the vast majority of brain research
is drowning in uncertainty.
Not just a few isolated studies, but many, many areas of neuroscience,
and while there's some truth to this, there's some danger in this also.
Here are some of the important papers that look at sample size issues,
that look at issues of finding interactions and
complex effects where it might be difficult to demonstrate such effects, and
really critique then of all kinds of research by John Ioannidis.
What you see here, Why Most Published Research Findings are False.
So, these claims about replicability issues have been applied
really broadly across fields.
And this is one of the papers that's been influential in the brain imaging field.
It was originally called Voodoo Correlations in Social Neuroscience,
which was a targeted attack really on replicability and
social neuroscience studies that look at correlations between brain activity and
measure of personality, or attitudes, or behavior.
And they subserviently changed the title to Puzzlingly High Correlations in
Social Neuroscience, but it really caused a big stir.
And Sharon Begley was one of the early advocates of brain imaging research in
the news who covered a lot of these stories, and here she writes
the neuroscience blogosphere is crackling with- so far- glee over this paper.
And it really it made a big stir.
One question is whether this is caused, that these problems are caused,
by a few bad apples.
So one example recently is Diederik Stapel,
who had a series of very high profile findings in science and
other journals, and subsequently was found to have committed fraud and
admitted to fabricating data from a number of papers.
And this is a terrible thing, but
it's unfortunate that it happens once in a while, and we can ask, are all these
problems really just caused by a few scientists who are overtly unethical?
And I think this is not the case,
this isn't just caused by isolated breaches of scientific integrity.
It's caused by really a widespread bias even by well-meaning scientists.
So, we have to educate ourselves on what the sources of bias are, and
then how to work so that we minimize those sources of bias and can overcome them.
There are many reasons for this, but let's look at what Ioannidis says about
why most published research findings are false, and what are the factors involved.
So here's some risk factors for false positives.
One is smaller studies, and when we apply this to neuroimaging studies specifically,
neuroimaging studies are very expensive to collect and analyze,
and so they have been quite small by comparison to other studies.
The low prior probability of the effect being true, or at least very large,
the ESP example that we looked at from Bem's research is one such example.
Some people believe in ESP, other people don't, but there's no known mechanism and
even Bem says he doesn't know how it works.
And if there's no known physical mechanism,
then it's less likely that any phenomenon is true a priori, and
that makes findings that they're surprising in this way.
That find things with no mechanism less likely to be true.
So we need more evidence to demonstrate that they are, if indeed they are.
Three is the number of tests conducted.
We'll talk more about this in this lecture and the next lecture, but in neuroimaging,
we have many voxels that we test, sometimes hundreds of thousands of voxels.
We can test multiple contrasts, multiple maps and
so the risk of false positives is very great.
And in addition, when we pick out the winners,
the effect sizes of those voxels look much larger than they actually are.
The next factor is flexibility in design, analysis choices, and outcomes.
And in neuroimaging there are many analysis pathways and options.
And we'll look at some of the impact of that and false positive findings as well.
Next, is financial interest.
So, it's financial interest and also prestige interest.
We all have a stake in supporting the things that we want to be true.
And many of us have a stake, our status, and maybe our self-worth, is on the line
when we test our scientific findings and we want our favorite effect to pan out.
So we really have to guard against this.
It's costly to be wrong in many cases, even if we don't think we're biased and
wehave the best intentions, and we really have to keep that in mind.
And one of the interesting related findings related to this is that there
was a meta-analysis published that showed that significant predictor of how large
a drug effect is in a clinical trial, is whether a drug company funded that study.
And finally, the competition to publish.
In hot fields, there tend to be more false positives, and
possibly in more prestigious journals as well.
So don't we all want to publish in Nature and Science.
Well, to publish in Nature and Science,
I don't know how to publish in Nature and Science, but if I did,
I might say that you have to have something that's both very surprising.
You have to have a very surprising finding.
Let's look now at selection bias in more detail, and this is going to be the core
of a number of problems that we have to deal with in neuroimaging, and
they're resolvable.
And so first of all, studies and tests with positive findings are reported and
those without positive findings are ignored.
This is called the file drawer problem.
But there are multiple types of selection biases, and
they include the publication bias that I talked about a minute ago.
Also biases in experiments, inflexibility in which subjects to include,
how many subjects to run, when you stop running subjects, and all those
sources of flexibility, if you're choosing them after you observe the data,
are sources of false positives and increase inflated effect sizes.
Flexibility in the model, which outcome do you choose,
which covariance do you include, and which modeling procedures do you employ?
And finally, the problem of voxel selection bias.
So which voxels do I choose out of the many voxels that I tested to actually look
at and interpret.
So this is the interpretability crisis now, we'll talk a little bit about this.
And this relates to findings that might be true but are not meaningful.
Here are a couple of the classic papers that deal with inferences about cognitive
neuroscience, and what we can infer about function from looking at brain activity.
And the first one is a really old paper now by Martin Sarter and Berntson and
Cacioppo called Toward Strong Inference in Attributing Function to Structure, and
they discuss some of the difficulties in this inferential process.
Foreshadowing a lot of these debates later on.
And secondly, this is an article by Max Coltheart,
which spawned a series of responses, and what he asked is,
perhaps functional neuroimaging hasn't told us anything about the mind so far.
He challenged the field to come up with an example where brain imaging
supported a cognitive theory, or was critical evidence
in deciding whether a cognitive theory is true or false, or which theory is true.
And there were number of responses, and
there was a redux of this later by Mara Mather and colleagues.
And I invite you to go and look up some of those papers for many interesting answers,
and we'll talk about a couple examples later in the course.
Explore our Catalog
Join for free and get personalized recommendations, updates and offers.
Coursera provides universal access to the world’s best education,
partnering with top universities and organizations to offer courses online.
© 2018 Coursera Inc. All rights reserved.
