Field Study Case Example: ShareTable Deployment

Loading...

From the course by University of Minnesota

Evaluating User Interfaces

81 ratings

University of Minnesota

Evaluating User Interfaces

81 ratings

Course 4 of 5 in the Specialization User Interface Design

In this course you will learn and practice several techniques for user interface evaluation. First we start with techniques that can be applied alone or in a design team, including action analysis, walkthroughs, and heuristic evaluation. Then we move on to user testing, including learning from a series of usability tests carried out in a real usability lab, and techniques to carry out your own tests even without a lab. Finally, we wrap up the discussion of evaluation--and of UI Design in the specialization as a whole--by looking at the question of how to set and measure usability goals, and in turn, when a design is usable enough to release it.

From the lesson

Evaluation with Users (Part 2)

We continue our discussion of with-user evaluation techniques (Part 2 of 3)

Ethics in User Evaluation10:07

Drafting a User Test Plan17:44

Field Study Case Example: ShareTable Deployment22:18

Assignment Video: User Test Plan7:14

Meet the Instructors

Loren Terveen
Professor
Computer Science and Engineering
Haiyi Zhu
Assistant Professor
Computer Science and Engineering
Lana Yarosh
Assistant Professor
Computer Science and Engineering
Dr. Brent Hecht
Assistant Professor
Computer Science and Engineering
Joseph A Konstan
Distinguished McKnight Professor and Distinguished University Teaching Professor
Computer Science and Engineering

Hello, and welcome back.

So we've been talking a lot about usability studies that occur in the lab or

another controlled setting.

But there are many contexts where you may actually want to run your study in a field

setting.

So I'll go through the process and what's involved in planning a field study.

And I'll actually do this by example.

I'll use one of the systems that I've built that I've talked about in this

course before, the ShareTable system.

And I'll talk about the deployment, the field study of the system as one example

of how you might want to go about this work.

So as I mentioned, I've already spoken about this ShareTable before, but if

you're looking at videos out of order or you're just joining us now, the ShareTable

is a system that helps parents and kids that little parts stay connected.

So commonly parents and kids may be living apart because of divorce,

separation, military, travel.

And the idea of it is it's actually kind of a piece of furniture

because the hope is to make it as easy to use as possible.

There's no buttons or keyboard in this system.

To start the connection you just open a set of cabinet doors.

That makes the table on the other side ring and if somebody there opens it,

it initiates the system.

Now what it does is it combines something like video chat, so on the monitor you'd

see the other person's face, with a ShareTable top space.

The way the ShareTable top space is created is that there's a camera and

a projector above each table.

It captures what's happening on one table and projects it on top of the other.

So in the image here, you see a paper board game being placed on one side

is being projected on the other side, and

as long as both people have something to use as tokens on their sides,

the tokens will be projected back on to the paper board game.

So, this is meant to support a lot of activities between parents and

kids, because it's kind of boring to just talk to each other, and

it's a lot more fun to do something together.

So whether it's playing a board game like the example here or

if a child puts a book on the table, the parent can see it.

Or a worksheet from school, they can help with homework.

Or if they want to have a tea party,

they can grab some cups from the kitchen and now they're having a tea party.

So we built this system, and we really wanted to see how it would be

used by families that are actually living apart, that are actually facing divorce.

So we wanted to run a field study and we had quite a few goals,

some of these were summative and some of these are formative.

On the summative side, we wanted to see whether parents and children would

spend more more time communicating when they had the ShareTable versus a baseline

comparison, versus what they did before we gave them the ShareTable system.

We also wanted to see whether children initiated more connections with

the ShareTable vs baseline because one of our goals was to make the system so easy

to use that it didn't require this sort of sophisticated scheduling or setting

settings or dealing buddy lists that young children may not be comfortable with.

So we really were hoping to see that they would initiate more of these connections.

We also wanted to use a few different validated metrics,

validated questionnaires that other people have developed to see how the ShareTable

compared with baseline in terms of emotional costs and benefits.

And in terms of how it actually affected the relationship between different family

members, both between the parents and children, but

also between the different parents in say divorced family.

We also had a few formative goals,

because we really wanted to know where to go next with the system.

We wanted to see what kinds of activities the parents and

children do with the ShareTable.

We had some ideas in mind.

We had done a brief lab study to understand kind of

how well it would work for different activities.

But we really didn't know what would happen once we would leave the system at

home with parents and kids and

just let them do whatever they wanted to do with it.

We also wanted to know what worked and didn't work, generally for the family,

so we could work to improve the system in the future.

And lastly, since I'm coming at this from a research perspective,

not necessarily from an industry perspective.

I also wanted to see, what were the promising next research challenges?

So it could actually inform my program of research, my program of study,

in the future.

So the first thing we did is we wanted to recruit some users.

And we really only had two prototypes of the system.

So we could only run two households, or one family at a time.

And we wanted to be very careful with our recruiting.

So in our recruitment calls, which we kind of had a flyer.

We had a few posts online about these.

We were really looking for families that were divorced or

separated with at least one child between the ages of seven and eleven.

So really, older than that kids can deal with regular Skype and

it's not as much of a problem, and

younger than that we weren't sure if it was going to work, though

we did have a few kind of younger siblings end up using it and it worked fairly well.

But we weren't sure in the process.

We also wanted to be able to interview the children about their experience.

And I don't know if you've ever tried interviewing a child younger than seven,

but it's definitely more of a challenge.

The other requirement for us was that the homes,

both the households needed to be at most a two hour drive from Atlanta,

because we actually had to deliver the system and set it up in their home.

And we were going back every week to do interviews.

And so it would really have been unreasonable for

the research team to drive more than two hours to visit the families.

But we still wanted the families to be at least one hour drive from each other.

Because if they, let's say, live on the same street already,

we're not really sure if the ShareTable really adds anything.

They can see each other in person almost every day.

And lastly,

was a requirement that we added after we had a few families reach out to us

who were interested but we couldn't actually set up the system in their home.

The last requirement was that high-speed internet had to be available in that area.

Now the home didn't have to high-speed internet,

we actually provided it for them.

But if there was no availability for it at all our system just wouldn't work.

And there were in fact a few rural areas and the study was done in Georgia.

A few rural areas in Georgia where a high-speed internet was

just not available at all.

Now in addition to these criteria that we actually stated up front,

there were a few kind of self-selection criteria.

So basically,

the families actually had to be willing to participate in a fairly significant study.

Because it was an 8-week study that involves significant data collection,

so it can be kind of a bit of a privacy violation.

And it included weekly interviews, so

it was quite a bit of a time commitment on the participant's part.

Even though we compensated, really for lots of busy families it's really hard to

find the time to dedicate to this kind of a study.

And the other self-selection criteria was that the families had to be low-conflict

enough that both the parents actually could agree to participate in the study.

So we weren't really looking at families where like every form of contact had

to be litigated by a judge.

I think it would be too difficult to try to deploy this kind of a system in

that setting.

So in lots of cases the families just kind of self-selected, nobody would volunteer

for the study if they didn't meet these two self-selection criteria.

But, these were quite restrictive in terms of users.

There was quite a challenge to actually recruit users for this study.

And in the end we actually ended up having to go through a professional recruitment

firm to do the recruiting for us.

Because it was just, so many constrains, so

many restraints on the kinds of families that could be in the study.

But through this process we were able to recruit two families, so four

different households to use this system for the amount of time that we wanted.

So the setting as I mentioned this was done in the the field.

So the ShareTables were actually deployed in families homes, so you see them here.

In some cases, the parents choose to put them in kids room,

in some cases they put them in kind of in a more communal family space.

So the one labeled B is in the living room and

as you see the cat has already appropriated the top shelf of the system.

And the bottom one D of is kind of in the den, the hang out spot in the house.

So it was really up to the families where they wanted to put the system.

And we made sure that we brought enough extension cables and

set up the internet in such a way that they could put it wherever they wanted.

Now to make sure that our system worked well we did provide business class

internet for each of our families.

And we did use our own router just so

we can do a little bit of computer science magic, make sure everything worked well.

And we really needed business class internet because of the upload speeds.

So typically download speeds were fine, but upload speeds are really restricted

for residential internet and it just wouldn't work as well for us.

Now, in terms of the actual methods.

So what we wanted to do is a field what's called A-B-A deployment.

So, A-B-A means that You measure some sort of a baseline for some period of time,

that's the A part.

Then you do your intervention, and you keep it there for some period of time,

that's the B part.

And then, after you remove the intervention,

you do some measures as well, so the A part.

And if your intervention is the thing that led to the change,

what you would expect is that there would be a change during the B condition, but

then that things would return to the way they were in the second A condition.

In this case,

our first A was a pre-deployment two-week baseline where we asked the parents and

kids to just do what they usually do except keep communication diaries about

any time that they used the telephone or Skype or texted or anything like that.

Then we deployed the ShareTable for four weeks and we collected a bunch of measures

during that, and finally we did a post-deployment two week contact, again,

to see what happened after the ShareTable was removed.

Now, in one of the two families, that was actually abbreviated, they had a new baby,

a new addition to the family, and so it just became really hard for

them to also do communication diaries, and this does happen in field studies.

Over a long period of time there may be major changes in the families, or

the homes, or the participants that you're studying and

you need to know that there may be some attrition.

So in terms of tasks and prompts, because we actually wanted them to use, they would

use the system if they had bought it if nobody was actually telling them how to

use it, we did not provide them with any tasks, though we did give them a manual,

which had some ideas for some ways they might use the system.

However, the families were asked to fill out short diaries after any

communication session.

So whether it's phone, text, video chat, ShareTable,

any time that they communicated during any of the sessions in our study, AB or

A, they had to fill out a short diary about it.

In terms of the metrics we collected, so during the pre-deployment two week

baseline, we collected baseline measures of relationship quality.

So we used the NRI inventory for that, it's a validated psychological inventory.

We collected these communication diaries that I just mentioned from both

the parents and the children and

we did weekly interviews just to kind of see what their experience was like.

Then during the four week ShareTable deployment, we continued collecting

communication diaries, but now that we've actually put our own system in place,

we were also able to collect text logs any time the system was in use, so

person A tried to call person B but person B didn't pick up,

that would be something we could record in the logs.

And anytime the system was actually actively in use, so both the people were

actually trying to chat, we would record video of the system use.

Now, to make sure to protect participant privacy, all of that video was actually

stored locally on the machine, and when we visited the families on the communication

areas they could note, I don't want you to watch this video,

can you delete this without watching it, and in front of them, we would delete it.

Now that only happened in a couple of cases, so generally we have video records

of everything the families did with the systems.

And again,

we did weekly interviews to understand what the experience was like for them.

And then after the deployment, we continued to collect

the communication diaries, we did weekly interviews, now we did the post measure of

relationship quality to see if the system had actually changed their relationships.

And we also asked them to fill out a validated questionnaire called effective

benefits and costs of communication technologies,

which focused on comparing different systems based on the emotional costs.

The emotional costs might be something like a feeling of obligation or

loss of privacy.

And emotional benefits, which may be things like feeling closer or having that

sense of the other person being there for you, even if you're not there in person.

So we collect all of these things, and

what I'm showing on the image here on the right is kind of the folder that I

have with all of the different scripts and protocols and

questionnaires that we had them fill out and diaries and all of the stuff.

So as you see, a lot actually goes into a field study,

you have to have a lot of these things prepared ahead of time.

You have to be pretty organized,

you want to make sure that you know what's going to happen every single week, so

every time you would go there to do a weekly interview you would have

a different script, and you would want to make sure to bring the right one.

You want to make sure that you have them fill out the right

questionnaires at the right time because if too much time passes, for

example, they might forget what it's like to use the system.

So, overall, quite a lot of work went into actually organizing this kind of study.

And lastly, I just wanted to show you an example of the diary,

the communication diaries that we used.

So we actually had two different versions, so

one was a version that we used with the parents, and

one was the version we used with the kids, I think you can guess which one is which.

So basically the kids had to circle when they talked, how they talked, so

in this case, the child has circled the phone.

They could also draw something that was different from that,

they have to say how they felt after the talking.

So in this case, the child drew, so I know it's a little bit hard to read, but

that's why we also did interviews so we could interpret these.

That's excited, that's what it says, and

the child also circled the topics that they talked about.

So in this case, they talked about how they felt, and

they also talked about earrings, so

I think the girl had just gotten her ears pierced, so she drew the topic.

And then the parents had something similar,

though they probably did less drawing, so

they said the time that they communicated the approximate length of the session,

the date, how they communicated, and what it was about, and how they felt about it.

And also, as you see here, there's this check mark box for if they didn't want us

to actually view the recording videos of that session, they could mark that, so

we got through quite a few of these diaries throughout the study.

Now, the point of this is to really just give you an example of what goes into

the field study, so I don't want to go too much into the results,

but I feel like if I didn't talk a little bit about the results,

it would just be kind of too much of a cliffhanger.

So let me give you just a little bit about what happened once we deployed the system.

So this is the results of the pre-deployment, so what you're seeing is

that for both of the families, there's generally fairly little talking.

So in a week, they were averaging, one family was at five and

the other one was about at 11 or 12 minutes of communication per week.

So this was mostly very, very short phone calls, and

none of them really used video chat regularly, it was just too hard to set up,

and so, telephone was really the technology they used the most.

And, with the pie charts, what you see is what proportion

of the sessions were actually initiated by the child.

So in the first family, it was actually all initiated by the parent,

the child didn't initiate any of the sessions.

In the second family, I think that pie slice represents one of the sessions

was initiated by the child, the rest were initiated by the parents.

And so, then we compared that to what happened after we deployed the ShareTable,

what happened with the ShareTable, and what we see is that the amount of time

spent communicating each week actually doubled, more than doubled, actually, for

both the families, and the children were initiating a lot more of the communication

sessions, though just because of kind of the social practices in the first family,

it was still very parent driven.

Their rule was that they kind of had to coordinate by phone before they could use

the ShareTable,

and so, it was still mostly the parents doing the communication,

but in the second family, it actually shifted so the kids were actually doing

more initiating than the parents were and that was interesting to see.

Now the other thing I want you to note on here is if you look at week three for

family one and week four for family two, these were the weeks that we actually

deployed the ShareTable system, and you see that there's kind of, it looks like

a pretty tall spike for those weeks, and this is what we call the novelty effect.

So we just gave them this cool new toy and they wanted to use it every day, and

they wanted to figure out how it works and

they were doing all these exciting things with it.

But then you see kind of a leveling off, so, in fact, this dip that we saw in week

four and five, and for one family, week five, for the other family,

is them trying to figure out, okay, well, we tried all the kind of cool,

quirky things that we can do with the system, how do we actually now make it

work for our family, make it work for our relationships?

And that took a little bit of figuring out for both families.

We actually see a dip in use, and then you see a plateauing as they figure it out,

they now have their practices around it,

they decide that they're going to use it once a week for a certain amount of time,

or whatever practice they come up with, and you see that leveling off again.

And this is really cool because you can't see this in a lab study.

You could only see this if you deploy a system in the field for

a longer period of time.

And so, the last element that I want to talk about a little bit is the researcher

rules, because it's also part of a user test plan.

And in this case it took a fairly large team to actually run this deployment.

So there was a research lead, in this case, this was me.

The research lead was in charge of doing things like recruitment, and

getting consent from the people that were recruited, the weekly interviews with

children, and just served as a point of contact for all the issues.

If something didn't work, if there were some questions that the participants had,

those went to the research lead.

There was also a research apprentice.

In this case, my wonderful Masters student at the time, Sanica.

And you can see more of her work in the paper that I'll reference later.

So, the research apprentice handled things like the weekly interviews with parents,

so that we could actually conduct all the interviews in parallel, so

we didn't have to have a really long visit.

And also collecting and

cleaning all the log data that was coming through the system.

Tech support lead, very important, so

doing things like doing setting up the share table,

making sure that if there was some sort of a problem that it was addressed.

Also, just kind of keeping an eye on the logs, and if they see something like there

were a lot of sessions where somebody tried calling and

nobody picked up on the other side.

Was that in fact,

because nobody was picking up on the other side, or was there some sort of

a technical problem that actually needs to be addressed with the system?

And this is a research prototype, so it was actually not that robust, and

we had to do quite a bit of babysitting to make sure that

the system would work when participants needs it.

And there were still quite a few situations where there were tech troubles

that we had to address kind of on the fly.

And lastly, but not least, we also had to have helpers to actually just set up,

take down and transport the system because it was actually a piece of furniture.

And so, every time we needed to bring it somewhere,

we needed to rent a pick up truck and actually get it there.

So we had two, three helpers from the lab who also assisted us with those tasks.

So that's a fairly significant research team just to run a study with

two families, four households.

However, it's still worth it to do these field studies, and

I think the main reason is that you really want to get beyond those novelty effects.

I question every study which gives a new technology to people and

ask them how much they like it.

Well yeah, of course they like it, it's new, it's interesting.

But are they going to still find it as fascinating

four weeks from now where they've had a change to use it for awhile and

they're seeing all the ways that it may or may not work for them?

It also leads you to have just more ecologically valid use and comparisons.

So the families weren't using these for tasks that we made up for them.

They were using these in ways that force them to find value in the system for

themselves.

And so we can actually make a very clear comparison between the use of the share

table system during the deployment,

versus the use of phone system before the deployment.

And lastly, and I think this is also really important,

at the end it led to very well informed feedback from our users.

This wasn't just the system that they saw a picture of, or

that they kind of played around with for five minutes.

This was something that was part of their lives for a whole month.

And so they were really able to tell us what worked, what didn't work, and

what they would do differently with it.

So it led to a lot of interesting research questions for me to explore in the future,

and I think that was a really a valuable side of running a field study.

Now of course, it's not all positive, so there's cautions as well.

And, one of these I've kind of been going through the entire time is cost,

time, recruitment struggles.

It's hard to do studies that last this long.

They cost a lot of money,

they might have to have a very large research team actually working on it.

I also kind of mentioned system robustness.

So the system has to work pretty well if participants are using it without

you there.

It has to work pretty much every time.

It is a research system, so there's some patience on a participant's part if it

doesn't work, but mostly you want it to be a fairly independent thing that can be

done without you being there to hand hold the system.

And the last one is called demand characteristics.

So what this refers to is this idea that

participants generally want to be good participants.

So if they have a sense of the kind of data that you're hoping to get, they may

actually kind of try to lean towards providing you with that kind of data.

And they're not doing this to mess up your science,

but they can see that you put all this effort into building the system, and

they really do hope that it works out for you too.

And this is quite common in all sorts of studies, but

I think in field studies it may be potentially even more prevalent because

you actually end up developing a relationship with your participants.

In this case we were in their homes every week for

two months asking them questions about their communication practices.

So they got to know us, and they wanted the system to succeed for us, potentially.

And so, if I had just asked them how much they liked the system, I don't know if I

would trust that data, because I think it would be very hard from them to say well,

actually, it kind of sucked and we didn't want to use it at all.

And so we relied more on the quantitative data about the amount they used it,

or the kinds of activities they did with it, rather than just saying well

did you like it better than the film.

So those are the main cautions for field studies.

I do hope at some point you do get a chance to do a field study for

any system that you may be building or testing.

I think it's really an interesting experience, and

it's a much richer evaluation than just running a study in the lab.

So for more information, now I kind of just hinted at some of the data that we

got from the share table deployment.

I didn't really talk about the relationship quality,

I didn't talk about the emotional benefits and costs.

So you can find all of this in the paper that we published on this study.

So check it out if you want to know more,

and just know more about the whole process and the team.

And, I also really like a research paper that kind of talks about this idea of

demand characteristics and field trials in a lot more detail, and provides a lot

more data about why this happens, how it happens, and what you can do about it.

So that's another paper I suggest you maybe check out, and

we'll link all of these in the resources.

So thank you for sticking around for a very long video, that's all I have for

you today and I hope to see you next time.

Explore our Catalog

Join for free and get personalized recommendations, updates and offers.

Coursera provides universal access to the world’s best education, partnering with top universities and organizations to offer courses online.

© 2018 Coursera Inc. All rights reserved.

Download on the App Store

Get it on Google Play