[MUSIC] Welcome to this last session of week four on objective testing techniques. Again, we're going to look at two alternatives to multiple choice questions, or perhaps two add-ons, complements to multiple choice questions to do with sequencing and sorting. Again, in our curriculum map, we're looking at test-like, formal teacher-centric formats of assessment. Sequencing. This is a really important cognitive and curriculum ability to put things in the correct order. And putting things in the correct order is part of the early childhood curriculum as well as high school curriculum. Being able to understand how a process actually happens; the internal combustion engine; how icebergs are formed; how human digestion takes place - all of these processes are part of important learning in many curriculum areas. And the advantage of using a sequencing task is that students have to think about the process, put things in the correct order, and normally this doesn't require much writing, because we can set it up to be a selection process, where they only have to sort and select which one goes first and then next and then next. The problem is not designing these tasks, the problem is figuring out how to score it. Take for example the following sequence or ordering task. Write the numbers one to six in the first column to show the correct order for developing a multimedia project. Note, that this is rather like a matching list, except that this time they're equal, and it's numbers. Now, even if you haven't studied computer programming or multimedia projects, you could probably get this right. But it's possible that you would get say, just two of these items out of order, jumbled, backwards. Number three and four reversed. So, the question is how do I score it when I get four in the correct order but two in the wrong order? What if I have everything in the right order but just started with the wrong one, and therefore everything is out by one? This creates a real challenge for us in scoring. So, there is a technique for handling these kinds of wrong sequencing. So, what I've written here on the screen shows the correct order as the key. So, the correct order for write codes is four, test and debug five, analyze one, plan number two, publish number six, and develop storyboards, number three. So, that's the correct order. So, what we have to do first is determine what's the maximum wrong you could do. So, the maximum wrong you could do for number one is to call it number six - so, that's a distance of five, it would be out of place by five. And you can see in the equation above the key that I've worked out what the maximum distances could be. And, as you work it through if number one has become number six then the furthest wrong number two can be is number five, which is a distance of three. And we can add up what the maximum distances could be. And in this case, it's sixteen is the maximum you could be wrong if you got everything wrong to the most. In this case, in the green, we see that in the hypothetical answer, number one and number two were correctly answered, number four was given a three, number five was given a six, number six was given a five and number three was given a four. So, they were all out by one. And when you do the, "How far from the correct are they?", three is distant by one, six is out by one, five is out by one, and four is out by one. So, there's four wrong out of a possible sixteen wrong. So, in that case, this student actually got 12 out of 16 correct sequencing, which is 75%. Now, how many marks should this item actually be worth? It's probably not worth sixteen marks. But you could say divided by four or divided by some other number, and say the total weight for this sequencing problem is this other number. So, you can see there is a little bit of math to work out to determine, what's the maximum wrong and how far from the completely correct is each answer, but that's the only way to give due credit to the kid who gets most of it right or gets it wrong only by a little bit. Another approach to the sequence is where instead of we ask people to put into a sequence, we ask them to complete a sequence. So, some teaching objectives can be fulfilled by "Here's a series of sequential steps", like mathematics, working out math problems, or chemical equations, where we can give them the beginning and the end, and some steps in the middle, and ask them to complete the middle steps. This takes advantage of cognitive load theory, where we give students partial sequences and they use their knowledge to complete the missing steps and this reduces the cognitive load and allows learning. Because if it's too hard to figure out what's the right step and have I done it correct then there get's to be a lot of paralysis. So, in this example of mathematics, there are seven steps in the sequence and they're asked to complete steps three and five and they're scored perhaps two marks. You get step three right and step five right and so this thing is really two marks. Notice we're not testing "Do you know the answer?", we're testing "Do you know the process?" And that's an important curriculum objective, especially in a classroom assessment program where you want to know can they solve these problems, not just do they get the right answer? Another important intellectual skill, as well as a curriculum skill, is sorting or classifying into categories. Now, this is commonly used in intelligence tests, but it's not just an intelligence test. It's also an important skill in most curriculum areas. For example, if I'm teaching in primary school in the early years, I've read to them and taught them the story of Little Red Riding Hood. And I ask them, now take all the characters in this story and put them into two groups, and give a name to each group. One student might say "Well, let's put the women together and the men." But if I say, "Put them into good characters and bad characters", this could turn into a really interesting debate, especially perhaps with middle school students who might actually think perhaps the mother wasn't so good sending her child out into the woods alone when she knew their was a wolf in the woods. And that becomes are interesting discussion as to "Was the mother a good mother?". Nonetheless, you can make these more objective and less debate oriented in a test situation. So, here on the left, you see, sort these assessments into either paper and pencil or spoken types. So, here's a good test of your knowledge of assessment. Examination, spelling bee, homework bee, in-class questions, parent-teacher interview. I suspect that the only thing that might be doubtful is examination, because there are oral exams. Come and do a doctorate and you'll get an oral examination. And, so you might need some qualification around that, but it becomes quite easy to sort out things into categories. But much more challenging is if there's three or more categories, where things can fit into different categories. So, this is limited by your understanding of the content that you're teaching. And again, the challenge is scoring. How many points, if you have ten items and put into three categories, and they get four in category one correct but they add in one that was wrong - how should you score that? Again, what we're suggesting is you give a score for every item being classified and then divide it by the number of categories. So, there's three categories, four were correct in this category, three correct in this category, two correct in this category, so two plus three plus four makes nine, divide by three and it's worth three marks, so there was only one wrong. The take home message in objective testing: questions don't have to be multiple choice to be objective and to test a wide variety of curriculum achievement objectives. Since there are many types of objectives, we don't have to always use multiple choice. Objectively scored questions like sequencing, sorting, matching, and binary and multiple binary as well as multiple choice, make assessment relatively efficient. We can cover a lot of material, in a relatively short period of time and we can score it relatively accurately. And therefore, we can get information back to teachers and parents and students relatively quickly about their strengths and weaknesses. The constant challenge with these objective test items, is to make sure they're high quality. If they're not high quality, if we write them badly or we score them badly, then it interferes with our ability to infer that students who get a high score actually have a good command of the learning material. Next week, we're going to look at more open-ended, subjectively scored items to complement the objective test approach. [MUSIC]
