[SOUND]. This week we're going to look at a variety of ways for evaluating systems or prototypes of systems. And one class of evaluations is qualitative evaluation, that's contrasted with quantitative evaluation where we're taking measurements and collecting numbers so we can do numerical mathematical analysis. Qualitative evaluation is more designed to give you a sense of the usability or to put you in the place of the user. There's a bunch of different techniques for doing qualitative evaluation and we'll look at a handful of those in this video lecture. First lets talk about the cognitive walkthrough. This is something that you can do with an interface or a prototype of an interface. You come up with a description of tasks. You come up with a list of actions that the user has to go through to complete that task, and you develop an idea of a user's background. In this lecture, we're going to have a couple running examples of interfaces and prototypes that we'll evaluate. And one of them will be a low tech prototype for a gesture based authentication system for a mobile device. In this case, what are the requirements? Well, we have a description or a prototype of this interface. Ours will be paper based. The description of the task is simply to authenticate and log in. The actions the user has to take to complete the task is to input a gesture. And our user background is actually going to be a pretty general set of users from, very familiar with technology to those who don't have a lot of background with it. So, there's our prototype. When we're working with it, we'll look to see if users know how to perform the action, if they can see the interface control we've given them, if they know that it'll do what they want, and if they understand the feedback they get. So, let's look at a video of doing a cognitive walkthrough with this interface. So here's our paper prototype, and you can see I've got two screens developed here. I have this overlay, which allows me to have the setup for when a person's initially creating a gesture to log in. So it says, create a distinctive gesture to log in, could be a signature or a sketch. Just make it memorable. To test this, we'd actually have someone pretend that they were interacting with the interface. So they would do their gesture in the middle and click Save. Then me as someone running the paper prototype would remove this. Now we've saved the gesture, and this is our log in screen. It says, enter your distinctive gesture to log in. I would do the gesture here, and then we can confirm that the user is logged in. I didn't create a separate page that shows the user is logged in, but that would be a common thing to do with a paper prototype. Now, to do a cognitive walkthrough, we need a description or prototype of the interface. That's what we have here, and that I've just described. We need a task description that's basically to log in. Normally, logging in isn't a great task, because it's not the user's goal to simply log in. They want to log in to do something else. But since we're just checking the log in system here, we'll make that our task. The list of actions to complete the task would be to click the on button, which I've labeled on the side here, I've drawn it out big like that because if you have embedded buttons in an interface, you obviously can't see that on a two dimensional drawing. So if the user wants to log in, they would click this on button, they would get this log in screen coming up, enter your distinctive gesture, I would enter it, and then the user would get logged in. Now, we want to look for in this cognitive walkthrough is, will users know how to perform the action? Will users see the control, well they know the control does what they want, and will they understand the feedback? Well we've got it labeled that they know the control to turn it on, and there's a pretty good label to indicate they should enter a gesture. I think the box is a pretty reasonable indicator that the gesture should be entered there, and we've already shown them this version where they would create a distinctive gesture to log in, so they hopefully know to enter it in the box. Will they know what the control does? Again, hopefully from creating the gesture in the first place, they will know. Will they understand the feedback? Now that's a problem and that's something that I haven't worked in here. So if I enter my gesture, presumably when I complete the gesture, if it's correct, I'll get logged in. But I haven't added anything in here to handle the case where a gesture fails. So what I might want to do is actually add a new button. We're doing a quick and easy prototype, so I'll actually bring some paper, and put a message, log in failed, try again. This is just a paper prototype, so I will just rip that off. [SOUND] So now if the user enters their gesture and it fails, I have a feedback mechanism to show them. That's something that I didn't realize until I did this cognitive walkthrough, but now the feedback makes sense. So there we go. That's a basic cognitive walkthrough of this interface. It actually helped us find a problem with the interface, and we've been able to adapt our prototype to give better feedback. The next type of analysis that can be done is a heuristic analysis. When you're doing this, you follow rules of thumb or suggestions about good design. And you go through the interface with a set of what you think are representative tasks of what a user would do, and you check to see if those rules of thumb are followed. This can be done by experts or designers, and it's a fast and easy way to evaluate an interface. But, it can miss problems that users might catch or identify if they were actually trying to use it themselves. What are those rules of thumb? Well, Nielsen has a list of ten heuristics. Those include, having simple and natural dialogue, speaking the users' language, which means not using technical jargon if you have users who won't understand it. Minimizing the users' memory load, so they don't have to remember too much. Being consistent, and that means consistent within your own application, but also consistent with what users are used to interacting with. Giving good feedback. Providing clearly marked exits so if the user doesn't know what they want to do, they can get out. Having shortcuts for expert users. Preventing errors. Giving good error messages, and providing help in documentation. So let's take this list of heuristics and go through and apply that to our gesture based authentication prototype. Now let's look at Nielsen's heuristics for analyzing this interface. Do we have a simple and natural dialogue? That's the first question. Well, we can start off with this appearance, create a distinctive gesture to login, could be a signature or a sketch, just make it memorable. That word gesture might not be familiar to users, but since we then go on to say a signature or sketch, that's pretty natural. The save button here is pretty natural. And if you remember, we created this error message. If they try to login and it fails, login failed, try it again. That's pretty clear. This also speaks the user's language, which is the second heuristic. We want to minimize the user's memory load. That's our third heuristic. Well, the only thing the user needs to remember is their gesture, and this dialogue that we have at the beginning, when they're creating it, offers some suggestions. A signature is something people remember pretty easy. And we make sure to tell them to make it memorable. We don't require them to remember much besides that because we give them instructions to enter that gesture when they come back to login, so they don't need to remember that's what they need to do. Consistency isn't much of an issue here. The only consistency that I can think of is that we have a login screen which is something that is pretty consistent across mobile devices. We need to give good feedback. Now that's something that we picked up in the previous cognitive walkthrough. We didn't have feedback for when the user failed with their login. So I've added this feedback here, and I think that that's pretty successful. The sixth item on Nielsen's heuristics is clearly marked exits, and that actually is probably a problem. If we go back to our initial screen where we can create a login, there's no option for a user to get out of this. So, if they don't want to create this gesture, or if they start creating one and they make a mistake, I don't have any option for them to start over. So, that's a problem. So what I could do is create some extra buttons. I could create a little cancel button. If the user wants to actually get out of this and not create the gesture at all. So I could put that up here in the dialogue. Make that a button. And then I also want to have the option for users to get out of having created a bad gesture, so in addition to save, I might create a button that says, start over. And put that at the bottom. So, then they have an option to get out of this window if they've created a problem. So again, this isn't an error that I made so we could find it going through. It's something that I actually didn't think of when I sketched out this simple interface. So Nielsen's heuristics helped us catch that problem with clearly marked exits. Shortcuts probably don't apply here. This gesture is a sort of shortcut way to login instead of having to enter a password. We want to prevent errors. Again this is a pretty simple thing that users are doing. Hopefully we have a good gesture recognition system so people don't enter their gesture and our system fails to recognize it. We need good error messages. That's something that again we created here. That's a pretty good error message. We may want to create some if the user is creating their gesture initially and it doesn't meet our requirements, but for now, we'll say any gesture works. And we want to provide help and documentation. That's not something you generally do on a login screen like this, and so we may want to make sure that we have documentation, either once they're in the phone or in the setup process. So there you go. A heuristic evaluation of our gesture based log in system. Another way of doing qualitative evaluation is with personas. In this method you come up with fictitious users that represent a class of users. They become a reference point for design and analysis, so that prevents designers, programmers and other people who are thinking about an application, from referring to themselves too much. Your persona has a goal or goals that they want to accomplish either in general or within your system. Here's an example of a persona that we've put together. This guy is called Frank. His goal is that he wants encryption, but in simple, low effort way. His overall goals is that he wants to use email and social media tools that are encrypted to protect his privacy. So we know what the wants within a system, and then we've got some background about him. So clearly he's a male. He's an undergraduate student who's 21 years old. He's a psychology major and a political activist. He's a savvy computer user but not an expert. And then we have a little biographical paragraph about him. Frank's an undergraduate at the University of Maryland. He enjoys playing baseball, swimming and watching movies. He carries an iPhone and he has a laptop and a tablet. He's constantly connected and he's a frequent social media user. But he's also a political activist and he worries about his communications being monitored. That part of his background clearly plays into the goals that he has. When you're designing a system and you're using a persona, you want to think about who this person is and then look at your system and ask, does this person have the skills necessary to use the system? Does our system meet his goals, would it be something he's able to do? And having him as a reference point, along with a set of personas that are developed to represent all the different classes of users you might run into, allows you to evaluate a system from the user's perspective, even without bringing users in. Let's use Frank's persona to analyze our login system. Now, even though his goals are for encryption, we can still use his background as one that will be useful to understand our login system. So finally, let's do a persona analysis. We have our revamped screen to create an initial log in, and we're going to see how this would work for Frank. So he's an undergraduate student, he's a psychology major, political activist, and he's a savvy computer user but not an expert. Would he know how to do this? Coming to this, he may not know what that word gesture means, but again, since we said that it could be a signature or a sketch, he probably would be able to figure out as a savvy computer user that he would draw with this finger in here. We know that he has an iPhone and an iPad from the persona, so he is used to a touch base interface. So, likely he would figure out that he could draw in here. However, talking through this, it makes me think that some users might not understand to do their gesture in the box. So we could put a little prompt message that says, draw gesture here. And that could be a message that starts off on the inside of this box when the user is creating their gesture. So, before they start, it's a little message to let them know to enter it there, and once they start gesturing, that message actually disappears, and they're allowed to finish. So that's something that could make it easier for users to understand what to do. Now, his goals in the persona are to have encryption and security to protect his email and his social media. That doesn't really apply here but he's likely security conscious about his devices too. So, as long as this is a secure way of logging in, it probably goes and meets his goals. So there we go. That's a persona based analysis. We have a pretty simple interface here so there's not a lot to discuss, but looking at those insights from Frank's background did give us a little bit to think about in terms of the design. So in conclusion, qualitative evaluation can provide insights into the usability of a system without needing to do measurements or timing. There are various levels of complexity. Some of them, like a heuristic evaluation, can be pretty straightforward, but developing good personas can be something that takes hours to do, and a team may spend a lot of time building the personas and doing the analysis. Some types of qualitative evaluation can be quick and inexpensive, but they may miss insights that users provide. There are also types of qualitative evaluation that include users. We are not talking about those in this lecture, but when we move on to talking about usability studies, we will look at studies that involve users, and those can be done both in the qualitative or a quantitative way.
