[MUSIC] >> Now, we've talked a lot about purposes of assessment, and when we come down to it, what you deal with a lot in the third and fourth weeks is about multiple choice, for example. >> Mm-hm. >> Now, is there not a danger in multiple choice, that it's really kind of a dumbing down? >> Well, yes and no. Fundamentally, if you write them badly, it will automatically dumb them down. The challenge is to write questions that reflect what you really value in a curriculum. It's too easy to write multiple choice questions, even well-constructed ones, about the easy-to-test, easy-to-measure stuff in a curriculum. And every curriculum has to have easy stuff, and obvious stuff, as well as deep and complicated stuff. And if you are testing and you only use the multiple choice technique, and you only use it to test the easy-to-test stuff, then of course, you're reducing the curriculum greatly. The challenge that we had in New Zealand, is that we said, regardless of the format, at least 30 to 40%, had to involve deep cognitive processing. And not just surface recall, remembering, regurgitating. And that forces you to think quite differently. You could ask questions, factual recall questions in multiple choice format, but we found our mathematics teachers were not interested in that. They just said, "Look, we know the answer is seven, no matter how they write it, it's going to be pretty easy for us to judge that, so we don't need multiple choice." But, the English teachers said, "Well, actually to understand a possible theme of this paragraph, it's much better to give them four different options of themes, that are from the realm of all possible themes, to say well, just think about these four ones. Which one's best?" And for them, they thought multiple choice was an ideal technique for getting at that deeper, cognitive task. So, depending on how you understand what your curriculum goal is, and what kind of skill you want out of them, you craft the question to meet those requirements. And, multiple choice can target deep, cognitive, higher-order thinking. >> A lot of people would disagree with you, I suppose, wouldn't they? >> I think the problem is it's easy not to do it, but all the standard texts on writing test questions and assessment that are used in education, always suggest that multiple choice can be used for deeper thinking. The problem with consigning multiple choice to surface and trivial, is then your only option becomes a human judgement scoring system for deep processing and I think you know as well as I do, that you can walk into an essay exam with your essay memorised, and just tweak a few start words to make it fit the question that was asked, and just regurgitate your memorised essay. There's nothing magical about an open ended task necessarily being a created, complex, cognitive act. They can be regurgitation too. >> Sounds like, there where you're talking about getting your essay all prepared, sounds like you're speaking from personal experience? >> Oh, of course! I mean, you know, I did a Masters Degree in education, one course had a three hour exam - do three topics, each one, one hour and you went in and they tell you in general terms what the topics will be, so you prepare your thinking and the first 15 minutes you, work out your plan from what you recall and then you write to the plan and you time yourself and you finish on time and it's time to start the next essay, you know, and away you go. So, yes, you can do that. I was a high school English teacher in New Zealand, and we used to teach students: "Okay, now in the exam There's going to be a question about giving a debate. Now, we actually never got to giving a debate in class. But here's what you write, when they ask you about giving a debate", and you teach them. And they, you know, if the kid has an ounce of brains, he knows that intro sentence didn't quite fit this question but most of the answer did, and they'll get a pass. So, yeah, you can teach to write essay exams, just as much as you can teach people to do multiple choice questions. So, there's no perfect assessment method and there's no assessment method that isn't free from efforts to maximize scores. >> Mm-hm. >> And the other thing I think that regardless of whether you're going to use a human judgement system or an objectively scored technique, is that every assessment has error. The teacher interacts with the child during the school year and they will make errors in judgments about what Johnny can and can't do. And it may take weeks before they realize, "Oh, I misunderstood you. You're actually better at this than I thought." Or, "Oh my goodness. You really missed out on all of that key stuff that I thought you knew." So, we always make errors in our judgments and the problem is with formal assessments, those errors are more obvious and perhaps, more consequential, but every assessment contains error. Let's just relax, take a deep breath and accept, I'm going to make mistakes, when I judge, when I interact, when I mark and therefore, it's an imperfect indicator, but I need indicators. >> Yeah. >> I still have to monitor what's happening. >> Yeah, in this course, one of the things we've gone for, Gavin, is multiple choice. And you've constructed, what is it? One for every-- ten questions for every week. Thinking about the kind of things you said about the vigour of multiple choice, as against essays, for example. What would you say about that as a way of assessing this course that you're currently talking about? >> There's always a tension. You can cover a breadth of content much more efficiently and systematically with lots of short questions that are easy to mark. But, that would leave you with a potential threat of atomistic and somehow inchoate or incoherent understanding. The real acid test in a course on assessment is how well do you assess your own students, and how well do you interpret the data, how well do you collect the data, how well do you do it and so, the real integrated test is, can you go beyond this? However, I suspect that in terms of scaffolding competence, I like people to know that they know the correct answers about the content before we start throwing them in the deep end of practice. So, as a strategy of self-testing, do I really understand what I've just read and listened to? I think it's perfectly good starter, but the real acid comes when they try to implement this in classroom practice, and that's having a strong reference library and a reference library of experts, who can tell you, "This is how we work this out." Teaching practice is not the same as knowing your sums, it's a much more complex field. But, I'm a great believer that you can't do deep and higher-order stuff until you know some surface stuff. >> I mean, when I'm talking about the Coursera quizzes, one of the things that strikes me is how, kind of, sophisticated they are and that takes a lot of thought. I mean, teachers just can't fall into doing that kind of assessment, can they? It's very high level skill. >> Absolutely. When we were developing the assessment tools for teaching and learning in New Zealand, our first strategy was, "Well, let's get panels of teachers together and ask them to write questions." And we soon very quickly realised that, you'd end up with, from every panel, probably about 30% duplication of questions because people write the things they know. And we ended up, instead, seeking out expert teachers, people that were highly regarded in the teaching community for teaching high school reading, or high school writing, or middle school math, and hiring them to come and sit with us for six months. We'd teach them how to do assessments and they would bring their classroom and curriculum knowledge. And, we would say, "We don't know what the content should be, you work that out. And then, we'll work with you until we improve the questions." And then we would take the questions to pannels of teachers to say, "This question is supposed to do A, B, C, D. If it doesn't do that, fix the question, so it still does A, B, C, D." And teachers are tremendously good critics. Not so strong on the creativity, but amongst a community of teachers, there are some creatives that you can train to do test questions. But, frankly the only way to learn, my students learn, is you write some and you give them to people and you go, "Do them. Answer my questions, and if you get it wrong, it's probably because I did something wrong not because you did something wrong." >> Yeah. Good market, kind of, testing there, isn't it? >> Absolutely. >> It's a big, and expensive process, isn't it? To get people to that level of expertise. >> At our university, there is a software tool called Pure Wise, which is, as I understand it, a free open access software, where you can write test questions, or get your students to write test questions and then the peers go in and critique and criticize, and it was created to help, say chemistry students write chemistry questions about chemistry and other chemistry students take the questions, answer it, and then comment on its quality, as a way of them learning from each other by testing each other. Certainly one of the most powerful learning strategies that was ever developed in the post war period was the SQ3R, where you survey, question, read, recall, and revise. And, frankly, self-questioning is powerful. So, writing questions about your own learning is a good way to teach yourself. So, it's not a bad thing. >> Okay, going back to one of the discussions we had before, but as a final question - could we come back, always again to these kind of cultural differences, and we talked about New Zealand, we talked about Hong Kong, but again in this whole area, do you see important cultural differences in using things like multiple choice? >> Education has become so globalized that I suspect that where the differences lie for minority groups is not so much in the standard, conventional, academic subjects, but much more in the culturally valued learning areas. So, it's interesting. New Zealand has a television station, Maori TV, and occasionally I turn on to it because they play a lot of foreign art films as well, but one of the things I keep bumping across is these competitive games that teams of people are playing and from the subtitles, it looks like what they're playing is a game to complete phrases and sayings and explain things and they are doing debates, and they are very competitive in their environment about testing each other in a competitive way on improving their language. And so, it seems to me that if we move away from a sense of competition, we might be disadvantaging minority cultures where there's a strong emphasis on compete with your brothers and sisters, but you're competing. It's still competition. And the things that we can do in the standard academic curriculum, aren't necessarily the things that minorities value. And it's difficult in plural societies like New Zealand and Canada, we allow people to create their own after school schools or their special schools, so we deal with some of these differences by allowing and providing state funding for additional schools or alternative schools that allow them to practice their own culture and language and still get schooling. I suppose the challenge in developing nations is the resource base to do that. >> As you say, I mean, I think you come back again and again to the kind of the global agenda, the globalization. So, everything is becoming more similar and it's policy driven, isn't it? We're all within that global policy context. So, there's a great deal of homogenization. So, things that might be a bit foreign to people like multiple choice testing become more gradually accepted into the norm. >> Yes. I think the bigger problem with the large scale international testing systems is they actually remove all the cultural indicators out of test questions to try to make them as culturally neutral as possible. I would suggest that the disadvantaged children in any society, probably benefit from questions that are extremely culturally laden, because they recognise and understand it, and if you strip that out of it, One: it's boring, and two: why should I care? And three: it's got nothing to do with me, and why should I try? So, I'm not convinced that de-biasing on international comparisons is actually productive. >> Yeah. >> And at the same time, my colleague and I are running an analysis of PISA reading comprehension data, and what we're finding is despite all the efforts to make things as biased-free as possible, our analysis suggests that Australia, New Zealand, U.K., Ireland, performances are statistically similar. And then, as you move away from that group of nations, the responses are no longer statistically similar. Even America and Canada are not identical to Australia and New Zealand. And then, you start getting the other languages, and if you look at the distribution of performance for Spanish speaking languages, what you see is the richer the Spanish speaking country is the more like Australia they are, and the poorer they are the less like Australia they are. And you kind of go "Well, there's an interaction of both socio-economic development, as well as perhaps some sort of Anglo commonwealth, child-centered practice dimension involved in making test questions similar across nations." The PISA tests are not stably invariant across societies. >> I think there's been a lot of debate and discussion around PISA and cultural bias and "Why does Finland do so well?", etc. etc. But, those are issues we can come back to, so thanks, Gavin. >> Not a problem. >> We'll go onto that in weeks four weeks five and six in which we'll pick up some of these issues again. >> Sure. [MUSIC]
