[MUSIC] Hi, welcome to this course where we'll try to improve your statistical inferences. My name is Daniel Lakens, I'm an experimental psychologist working at the Human-Technology Interaction group at Eindhoven University of Technology. Now, some years after I completed my Ph.D, I realized that my understanding of statistics was actually not good enough to design a proper study. I joined the Reproducibility Project which has the goal to reproduce 100 studies in psychology. And when I started to perform the replication study that I would do, I realized that I had to calculate an effect size to perform an a priori power analysis. Now, this is a very basic first step when you design a new study, whether it's a completely new study or a replication study, it doesn't matter. I realized I didn't really know how to do this. And then I wondered, well, if I don't know to do this very first, basic thing, how can I ever collect informative data? Since then, I've been looking into how to improve the way that we design studies and draw inferences from data. I published some papers on this, and I've realized that improving the statistical understanding of how you do research really makes you a more efficient and better researcher. So I'll share some of the insights I had, and I'll try to prevent some confusion that you might have about statistical tools that we use. The goal of this is to try to prevent you from fooling yourself. As Feynman reminds us, the first principle in doing science is that you must not fool yourself - and you are the easiest person to fool. So I see a better understanding of statistics as a very good way to try to prevent you from fooling yourself. Now, as an experimental researcher, the goal might not be to do statistical tests, itself. What you always want to do is to say something about a theory. So the goal of data collection is to first draw statistical inferences and then to draw theoretical inferences. That's what you really want to know. But, of course, you need to draw good and valid statistical inferences because otherwise the next step doesn't really work. So let's try to improve this step. There's a reason why psychologists, maybe scientists more generally, are now really interested in improving statistical inferences. And one of the first reasons for starting on this improvement cycle was a study published by Daryl Bem on precognition. Precognition is the idea that we can predict the future. Now, you might believe that this is possible or not, but as a scientist we should always keep an open mind and look at the evidence. In this case, there was a set of nine published studies that all showed support for the idea that people could predict the future. There were very simple basic tasks that were performed that are widely used in experimental psychology, but the tasks were always reversed. So instead of first seeing a picture and then pressing a button to categorize the picture, the task was presented in the reversed order. People first had to press a button, and only then the picture would emerge. So, of course, you don't really know what to press unless you can predict the future. And in this set of studies, people were able to perform above guessing average in most of these studies. So after this, maybe some people thought precognition must be real. But other people thought, well, if this is going to be published in one of the top journals in our field then maybe we are not drawing the correct statistical inferences from data because it seemed unlikely that this was true. There has been more criticism in the scientific literature on how we draw inferences from data. Now, there are some people who even go as far as to say that most published research findings are false. Is this true? What do you think? Do you think that most of the scientific literature is false or not? I think the answer is we don't really know. We can try to estimate it, but we don't really have an accurate number. It's more important, probably, to try to improve the way that we do research to prevent this from continuing on, so that in the future we know that the data that's generated and the conclusions that are drawn are most likely to be correct. There are some problems in the scientific literature at the moment. We know that many of the sample sizes we collect are too small, so we have underpowered studies. We know that people are very flexible in how they analyze data. So this leads to a lot of flukes, random findings, that are actually not true, but that people interpret as a true effect. And we know that there is strong publication bias. People share findings that show an effect, but they don't share findings that do not show an effect. And all these things make it very difficult to conclude whether published findings are reliable or not. And the goal in this course is to try to give you some tools to try to answer this question better, given that there's publication bias, flexibility in the data analysis and low-powered studies. So, first of all, we'll try to make sure that you can draw better statistical inferences. To achieve this, we'll talk about p-values and how to interpret them, report and calculate effect sizes and confidence intervals, but also discuss alternative ways and frequentist statistics such as Bayes Factors or likelihoods. The second goal is to enable you to design more informative experiments. To achieve this, we'll talk about how you can control error rates when you design studies, how you can efficiently run high-powered studies. But we'll also focus a little bit on theory construction. Another goal is to evaluate the evidence in the literature, given that there is publication bias. We'll talk about p-curve analysis, and we'll discuss publication bias and some ways to identify this in more detail. Finally, an important goal is to facilitate cumulative science. Of course, we try to improve our knowledge. How can we do this the best way? We'll talk about how you can perform replication studies, the importance of pre-registration and the benefits of using pre-registration, and the importance of adhering to open science principles in your research. Accompanying these lectures are assignments, and these assignments are really hands-on and in-depth knowledge. The goal is to learn mainly through simulations. I'm not a big fan of teaching through formulas, myself. If you're interested in that, you can easily look up the literature and I'll provide references where these formulas are discussed. But in these assignments, we'll mainly rely on performing simulation studies, and just seeing what happens in the long run when you would perform hundreds of thousands of studies. Personally, I've found that really informative when I tried to understand statistics myself. We'll use R, which is free software. You don't need to have a very good understanding of how R works. We'll typically use very simple examples where you can just run the code, maybe change one or two numbers. That's really all you need to know about how R works. Now, personally, as I started to get a better understanding of statistics, I felt much less uncertain and much more able to draw correct inferences from the scientific literature, and to design studies that would yield informative results. I really think that if you want to be a good researcher, it's important to invest some time in learning how to draw better inferences from your data. And it can be remarkable fun, getting this better understanding. If you want to reach out to me, you can contact me at Twitter with any questions or comments you might have. For now, let's get started. [MUSIC]
