[MUSIC] Welcome back. Again, in week four we're looking at objective testing techniques. And today we're looking at the rest of what goes into a multiple choice question. Wrong answers are the art of multiple choice questions. A wrong answer, or distractor, has to be plausible. You can't ask questions about mathematics and put the word potato in as a distractor just because you can't think of a plausible wrong answer. By choosing something so silly, no student is going to be distracted by that option, and it's wasted time and space. To develop plausible wrong answers, it helps if you can connect it through your experience with students in classroom to things that you know they commonly misunderstand or where they can over-generalize. My teaching area used to be in English reading and when people read, they sometimes make a connection to the wrong thing or they over-generalize or they narrow it too much. These are skills and faults that people make in learning a content area, and distractors should be connected to those kinds of mistakes. But to avoid testwiseness we try to structure the wrong answers so that they're similar to each other in how long they are. So, if one answer is twenty words and all the rest are only ten words a lot of students will pick the long one just because it's different. Following the same grammar, if the correct answer starts with verb "ing", than all the wrong answers should have verb "ing", they should all follow the same grammatical and syntactic style. What we're looking for in a wrong answer, is an answer that somebody who has a partial understanding or a misunderstanding is actually attracted to choosing. We want to be able to separate those who really can do this from those who only have partial understanding. And that's why wrong answers have to look believable, except to the student who truly understands the skill or the domain. If answers are arranged in sorry-- When arranging your answers, if they are numbers or words or times, please put them in ascending or descending order. Don't jumble them up. Don't add an extra layer of complication. And very rarely are words like "never" and "always" the correct answer. There's not much that's always true or never true, and so they're not very helpful to you in most testing situations. On the screen now you can see a simple math question. "What is the value of two plus three?" Now, this question you might use in grade one or grade two. And the answers are certainly not suitable for a grade one or grade two test question. But we're using this to illustrate, the kind of thinking that went into choosing the wrong answers. We know that two plus three is five. But if you had subtracted instead of added, you would have got the answer minus one. If you had multiplied instead of added, you would have got the answer six. If you had given the cube of two, you would have got eight. And if you just ignore the plus sign altogether, you would have got 2-3, 23. So, each of these wrong answers has been based on a plausible misunderstanding of that squiggle shape called the plus sign between the two numbers. Now, it's highly unlikely that anybody in grade one or two being asked this question would make those kinds of mistakes. So, the point is to illustrate the kind of thinking that you need to do, in order to arrive at good distractiors. How many wrong answers do you need? Well, the research says three is good enough and any number between three and five is fine. Certainly in our work here in New Zealand, we kept our questions to three wrong answers and a correct answer, often because for some questions I could find four wrong answers. And a lot of times four wrong answers is just one wrong answer too many. But let's not kid ourselves, sometimes, there's only three answers anyway. For example, an acre is larger, smaller, or equal to a hectare? There's only three possible conditions, so that has to be a multiple choice question with three answers, one of which is correct. And certainly we would discourage you from stretching to five answers by using "all of the above" or "none of the above" simply because if one turns out to be correct or one turns out to be wrong, then "all of the above" or "none of the above" is automatically wrong. So, it's not a very useful question. Here's another example, perhaps a little more challenging and suitable for junior high school social studies curriculum. "Which year is associated with the early European exploration of New Zealand?" And the answers are: 1215, 1492, 1642, and 1852. And you can see on screen there, the reasons we chose those answers all had to do with something historically important either in Europe or part of European exploration. And as a bonus, each answer has a two in it, to avoid the kid who can go, "I know it had something to do with a two." And then, "Darn, all four of them have twos, so I can't use that as a clue." So, it's important that when designing these kinds of questions that you think through carefully. What you'll find is designing good multiple choice questions is hard work, but marking them is very easy. Multiple choice questions can be used for a range of higher order thinking skills. On the screen now you can see two examples from a creativity test. "In each of the following sets, choose only one idea from each set which you think is the most creative use for tin cans." And in each of these two questions, one answer certainly requires more of the skills associated with creativity. That is, changing an object in such a way to make it do something it wasn't originally intended to do. And, that is the notion of creativity. And, this is a good multiple choice test that requires deep thinking, to puzzle out which of these options is most creative. I'll leave you to figure out what you think the right answers are. Multiple choice questions can be used to even test careful thinking. On the screen now you see a question, "With which of the following statements would the Chief Meteorologist most likely agree?" And you could puzzle out reasons for why the chief meteorologist of any country might agree or disagree with each of these statements. And again, I'll let you decide which one you think is the right answer. There's a possibility of variation in style, we don't have to stick to the list of answers labeled A to D or E. We can embed them inside sentences like for example, "Choose the best word for each sentence: he [went or gone] to the store" and so on. Choosing the best word when it's embedded in a sentence is still a multiple choice question, even though it doesn't look like a multiple choice question. And in the last example, on this screen, you can see a task which requires underlining letters where the capital letters are needed. And what you can see here is, in a sense, each word is a choice. And the student has to choose "Should I circle it? Should I highlight it?", whatever the instructions are. Finally, multiple choice questions are an efficient way to sample lots of knowledge and understanding. This is part of the reason Coursera itself and this course, through the Commonwealth Education Trust, we've relied on multiple-choice questioning each week to just explore the range and depth of your understanding. We hope we write them as well as expected, because it takes great skill and practice to write good questions. And the best technique for writing good questions: have another teacher read it and answer it. And if they don't get the same answer that you got, it's probably because you didn't write it well. It takes practice. You have to think about what you really want the student to know and make sure you test and ask for that. Remember, the goal is that students who get the questions right did so because they actually know this material not because we wrote the questions badly. In the next session, we will look at a couple of other techniques beyond the multiple choice question, that will allow you to more objectively assess a wide range of skill, knowledge and ability. [MUSIC]
