So, what does research say about how you should study? I'm going to give a brief summary of this paper that was produced in 2005 by Nunes and Karpicke, where they do a great review of the history and the most recent state of research on what's called retrieval-based learning. Retrieving, in this sense, means the process of trying to remember something in an effortful way to test yourself on it, to retrieve some knowledge that you studied before and to bring it out into the world. Now, retrieval knowledge is an the idea that trying to retrieve information from your memory is good for helping you remember it, that's not a new concept. They even bring up back in the 1600s, Francis Bacon, who if you go look him up on Wikipedia, you'll see was really famous guy in the area of empiricism where people were really trying to empirically study things and the scientific method was really coming into play. And here's what he had to say about this idea of retrieving or studying. He said, you could read something 20 times but you do a lot better if instead, you read it just ten times and every once in a while you'd stop and try and recite what it was you were remembering rather than reading it, and then check yourself to see if you've got it right. So, this idea of instead just passively or letting your eyes go over the words and re-reading them, that you actually effortfully read and you stop and you try to actually remember what's there, and then if you can't get it, then you go and read it. Now, let's jump ahead to a somewhat more modern times. One of the studies that is discussed in this paper comes from 2006 and this happens to be one of the most biggest news stories in science from 2006. So, we've come a long way since Francis Bacon. So, what happened in 2006? Roediger and Karpicke, who are two of leading researchers in this area, they did almost a replication study of something that had been done many hundreds of years before. They gave students the exact same amount of time, that's however many minutes, and they either gave them that time to sort of study in four different time periods. So, maybe 10 minutes studying, 10 minutes studying, 10 minutes studying, 10 minutes studying. Or they gave people three periods in which they would study and then the fourth period, they asked them to retrieve or recall as much information as they could. And then they had a third condition where they actually only got to study once but then spent the remaining three periods recalling information. So, it's important to note that, again, in any of the conditions, where a row was a condition in this case, students had the exact same amount of time that they were working on this. It's matter a how did they spend that time. Did they spend studying? That is, reading over the information which, I think in this case, was something like one of those SAT passages about where you want something about turtles or something like that. Or do you spend some of the time just re-reading it versus trying to remember and retrieve the information that you got? So, I don't know what you think would work the best. Well, why don't you take a guess and see what you think? Well, before we reveal the answers, let's look at the graph that we're going to be studying. We're going to be looking at performance. And in this case, higher is better because on the Y-axis, we have the proportion of things from that reading passage that people were able to recall when they finally tested. And on the X-axis on the bottom, we're looking at our three conditions. On the left is the just study all the time, the middle is three study periods with the retrieval period, and then the right is a study with the three retrieval periods. So, let's see what students did. Wow, that might be surprising to you. Certainly, was to me when I first saw it. So it turns out that when you test people on something but all they've done is had the genes to read over it a while, people don't remember very much of what they've read. In this case, it was only 40 percent. But if instead, you only give them a quarter of the time in that study but they then had the opportunity three different times to try to remember what it was that they had read, then they did much better. And, study once and then retrieve three times? I think most of us or most students would think that might be a real waste of your time. Why shouldn't you be spending more time studying? Surely, you're going to get more by reading it over again, right? Not what this found. In fact, we know they even study this also in the study that that is how students most people believe. So, they asked people after they've gotten done, how confident were they in how well they did? And the student believed some of these are wrong. So again, here in this graph, why higher is better students judgment of learning how well did they think they learned it. Those students who studied all four periods, they were pretty confident about how well they learned it. Whereas those people who studied once and retrieved and then for the three other periods, they had pretty low confidence in their ability that they actually knew it. So, in direct opposition to their actual performance, the beliefs of the students were that they would do better just studying. Now, it's important that we discuss what we mean by retrieval. And there have actually been a number of studies that Roediger and Karpicke have been involved in where they looked at different methods of retrieval. How do you retrieve information from the brain? In the study we just looked at, people retrieved information by writing down everything that they could remember, all the facts that they could remember from the passage that they read. Another option, which maybe is of interest to us with technology and education, is giving people multiple choice quiz. So, the question here is, is one of these better or not? And the answer is, it depends. So, it happens to depend on how much you're actually able to remember from the passage. If you remember a lot about the passage that you read or whatever it is you're studying, one technique is better if you're a member less than another is better. It turns out that if you remember a lot, then writing this down is a better technique for remembering in the long term and the rationale they believe is behind this because it's really effortful retrieval. Like your brain has to work really hard. If you're staring at a blank piece of paper, let me think of everything I can remember and that it made me this effort in the retrieval that's really important. Multiple choice, obviously, is less effortful. You've got your options right there in front of you. All you have to do is recognize and check them. However, this actually ends up being more effective for helping people remember later if they weren't going to do so well at remembering everything anyway. And why is this? That's because with the multiple choice options, they were often can be reminded of the best answer. So, they get feedback in a way even if you don't tell them which ones they got right or wrong, although that's a really good idea. But multiple choice, it provides the options in front of you and so, if you weren't going to remember it, you're going to have a better chance because you;re going to see those words again. So, I think the takeaway here is, and there's not completely compelling evidence, not everyone believes this, I think that as in a regular sort of practitioner setting over trying to figure out as educators, maybe we're not perfectly knowledgeable about exactly what all our students know, that we'd want to use the multiple choice option to engage our students in retrieval practice. I think there's a variety of reasons for this. One, I think students are more likely to do it and this is something that is known informally in the MOOC world that if you provide an open answer question on a quiz, many people will just skip it. Whereas, if you provide multiple choice or check all that apply, people are more likely to at least try to answer those questions. And the other thing is, if you're not going to know in advance how well people are going to do on it, then having them do the multiple choice where you can give them feedback about which ones they got correct or not, that covers the basic case in case they didn't actually know that much about it. So, it's a safer bet. Now, you might say, okay, but they were reading things about turtles in a city. But what about more complex learning? What about conceptual knowledge or knowledge at the university level? Well, they got on to study about that as well. What you're looking at here is something called a concept map. And in particular, the study the wording encryptic, did recently was in chemistry, university level chemistry. And concept mapping is a technique that is sometimes taught in chemistry classes to help people see the bigger picture, to put all the concepts about various chemistry knowledge together in a way that goes far beyond just memorizing details. And so, concept mapping has been a very popular thing in education, is considered to be a really great way to build people's knowledge up. So, they did a study where they compared having people retrieve their knowledge by building a concept map, versus retrieving their knowledge through the standard old sort of multiple choice questions. And here's what we've got. Similar study structure. So higher is better and this is your performance on the final and I've highlighted here the two parts because they had study, they had repeated study, then they had study with the concept mapping for retrieval, and then they had the regular retrieval practice. And what you see is that retrieval practice was better than building up this concept map in terms of how well people could perform on the final test. And they actually interestingly had different tests which involved building concept maps and doing multiple choice questions. And if you want to read more about it, you're welcome to. And just so you know, yet again, students' metacognitive predictions, their predictions about how much they would be able to remember or how well they learned it by doing these things, were again backwards. That is, students who did the concept mapping felt like they learned more. But in fact, we know from their final performance, they did not. So now, we've looked at how you should study. Let's go on to when should you study. The research into this area has these particular terms attached with it. They call it the study of Massed Practice or Massed Study, that is cramming like many hours or many minutes in whatever case, versus Spaced Practice, that is spacing your study or your retrieval out over time. So here, you could imagine a massed retrieval would be, say, study and retrieve it all in one day, and a spaced retrieval would be study it and then practice retrieving it maybe three days in a week, et cetera. So, what does the research say about this? Well, it turns out that the answer depends. It depends on when you need to know this information that you're studying. That is, how for in the future will you be tested on it. And in particular, because these are cognitive science studies and psychology studies that are done in research labs, they've tended to test two retrieval periods. Do you need to be tested one day later or one week later? Now, we can talk later about whether or not we think those are even reasonable things because I certainly hope that most of the things you learn in school, you care about knowing more than even just one week later. But right now, a week serves as a proxy for longer term remembering. So, this is an image from a relatively old experiment, I think, back in the 60's. And we're looking at two things. In the bottom, the two sets of columns, we've got Massed on the left and Distributed or Spaced Learning on the right. And we're just going to look at the dark green bars here which are when people were tested on their learning just one day after their last session. Okay, so if you did Massed Study, you studied and retrieved all in one day, and you were just tested out one day later. And you can see already, well, you can see maybe surprisingly that the people who've studied spaced out over time didn't do as well on that test, this was words that would be recalled one day later. So, that seems to have been plain contrast to what we've said. But if we switch and we look at people who were tested one week later, so this is people who were tested one week after the last time they retrieved the knowledge, oh my gosh, more than double the amount of remembering, double their score on this test, if they needed to know that knowledge more than 24 hours later. But this was a fairly old study. And in fact, while there have been other studies on it over the time, I'll talk to you about one of the more recent ones, where Jeff Karpicke here, in 2011, wanted to replicate this study from 1964, but go even further and say, well, what kind of spacing works best? Is it tight spacing, one day after day? Is it medium spacing, couple days? Is it longer spacing? Et cetera. And I'll stay at the very high level because basically, between the difference of when there was no spacing, which is far on the left, people did very poorly, okay? When that spacing, short, medium or long happened, and then also they had these even more complicated things about whether the spacing expanded over time or was equal over time or contracted over time, none of these were statistically significantly different. I know the bars look like they're different heights, but there was no statistically significant difference in the amount people were able to recall later based on the spacing of their retrieval. So, the only thing, the key thing to come out of this is, it's you should space out your retrieval practice, your studying and retrieval practice. You shouldn't do it all in one day if you want to remember more than but at least a week later. So, what are our final takeaways from this? So, if you care about studying and care about studying in a way that you can remember for a long period of time, then you need to study, yes, but then you need to spend a lot of time not really studying but retrieving or attempting to retrieve and getting feedback on what it is you're trying to learn. And you shouldn't just retrieve at one times like, wasn't study, study, study, retrieve. You should retrieve that many times, study, retrieve, retrieve, retrieve. And this goes against what most students think is good for their memory. So, it's far more important to try to retrieve even if you fail, there's usually a feedback on it. And this retrieval needs to be spaced out over time. Cramming, seriously, doesn't work if you need to know it more than 24 hours long. And while maybe cramming for a test, obviously, in fact, we had the data showed that that would be valuable, if it's something you actually care about knowing beyond the exam, then you need to space your retrieval practice out over time. And the other thing is it's really important to effortfully try to retrieve things. That was the idea of multiple choice versus writing it down. But the most important part, besides effortfully retrieving, is making sure that you review the right answer. And if in that case, a multiple choice type item makes that more effective for people, then that's probably what we should choose.
