When conducting a user test, there are often some practical challenges that you will need to think through. In this lecture, I'll talk about some of these challenges and give you some ideas about how you can deal with them. One of the big challenges is system variability or ways that the system can vary for different users such that they have a different experience. So, most systems have data that they're providing access to, and that data can change during the course of a user test. So, to take Amazon as the example, the data that we'd be talking about are the products that are available for sale, and those can change over time. Certain things can go out of stock, certain things can be added, and so on and so forth, and there's a chance that the data that is available to one user will be different than data that's available to another user. Many systems require users to have an account in order to access most of the functionality that they provide. So, you'll need to, first of all, make sure that you have a way of giving people access to the system by providing them with accounts or having them sign up for accounts, and you'll also need to make sure that the accounts that they use have the same access and the same privileges as each other, so that one user is accessing different functionality than other users. Related to this, many systems have settings that users can configure to change the way that the system operates. So, to just look at the example from Amazon, there are a couple of dozen different settings that can change the way that the system operates, ranging from 1-Click settings, login and security settings, Prime membership, and so on and so forth. More and more, we see systems that changed the way that they operate based on prior actions that users have taken using that system. So, the systems maintain state, most relevantly, the history of prior actions, and use that to change the way that the system behaves. So, for example, Amazon keeps track of your browsing history and will actually offer different products based on what they perceive your preferences to be from your prior actions. Another example of how the history of prior actions can influence the user experience is in how browsers will change the presentation of links that have been previously visited. So, imagine that you're running several user test sessions in a row and test subject number three might see traces of the previous two subjects and might see that in the way that visited links are presented. This could give that user a clue about what direction to take in order to satisfy the task, and that's probably not something that you want to have happen. Another example of how history could influence the user experience is through autocomplete. Many modern systems offer autocomplete and text fields, and they'll populate the options for the autocomplete from things that have previously been entered. So, again, if you're running several test subjects, you need to be careful that the things that earlier subjects entered into that text field don't show up as answers that are available for subsequent test subjects, because they're going to get a clue about how to proceed that they wouldn't get if they were using that system naturally or in the wild. Another tricky thing that comes up sometimes in user testing is dealing with purchasing. If you're working in e-commerce environment, a lot of times, you're interested in seeing how people navigate the purchasing experience. But having people actually buy stuff during your user test can be tricky and often something that you don't really want to have them do. If possible, if you have access to the engineers that are building the system, you could work with them to try to create a sandbox system where purchasing actions don't actually result in credit cards being charged or inventory being depleted. Whether that's possible or not, you also want to think about providing users with the form of payment that they'll be using. That could be in the form of a gift card or a pre-paid debit card or something where you could provide it to them rather than having them use their own funds which might make them uncomfortable. Especially in cases where creating a sandbox system isn't possible, you might want to arrange to cancel the order after the test, and that's going to be something that you're going to need to remember to do so that you could undo the actions that were taken during the test, especially if they have consequences like funds being transferred. If none of those things are realistic or possible, what you're going to want to do is design your tasks so that payment isn't part of the tasks. You can have people accomplish everything up to the point where they actually enter their credit card number or finalize the order, and still learn a lot about how they navigate the purchasing process up to that point. The most important thing is that all of the people who participate in your test should have the same experience to the extent that you're able to control that. They should ideally see the same data. If the data is going to be variable, you need to understand how it's going to vary, and design your tasks accordingly. You want to make sure that they all have the same experience from the perspective of creating accounts, logging in, and so on and so forth. So, you either want to have everybody create and use a personal account or have nobody create and use a personal account. Most of the time, giving people pre-made accounts is going to be a better option unless part of what you want to learn about is what it's like for them to create an account in the first place. You want to make sure that all of your users have the same settings or at least the settings don't vary in ways that will affect the way that the tasks unfold. You want to make sure that all your users have the same history of prior actions or no history of prior actions. You want to make sure that all of your users are using the same payment method and details of payment, if payment is something that you're going to include in your test. What a lot of this means, from a practical perspective, is that many tests that you conduct will require resetting the system in some way between test sessions. So, you may need to do things like clear the browser cache, or clear the search history, or clear other traces of prior actions that would affect the way that the system behaves going forward. You may need to undo user history in the system itself, which may require access to resetting the database or creating new accounts or things like that. A lot of times, you're going to need to remove data that was created by test participants if they've entered information into the system and changed the data that's available in the system. You're going to need to undo that and roll it back, and all of this means that, where possible, you may need to work with an engineer or developer to help set up the test environment so that you can control all of these different things. A lot of times, it can be really difficult to anticipate the way that the system is going to behave after repeated uses, especially if it's a complex system, and this is yet another reason why pilot testing is so important. The stuff we've been talking about in this lecture suggests that you need to do more than one pilot test so you can see what happens after repeated uses of the system and make sure that traces of earlier sessions don't carry forward into later sessions. It also suggests that you want to make sure that your pilot test or at least some of your pilot tests are done in conditions that are as close as possible to the actual test environment, so you can see how the system is actually going to respond to the tasks that people perform. These are just a few of the things that you often need to think about when planning a user test. You can take the time upfront to make sure you've nailed everything down. It's going to pay off in the end because you're going to have a solid, consistent set of data that will tell you how well your system performs and what you need to do to make your system better.
