0:00
Hello, and welcome back.
So we've been talking a lot about usability studies that occur in the lab or
another controlled setting.
But there are many contexts where you may actually want to run your study in a field
setting.
So I'll go through the process and what's involved in planning a field study.
And I'll actually do this by example.
I'll use one of the systems that I've built that I've talked about in this
course before, the ShareTable system.
And I'll talk about the deployment, the field study of the system as one example
of how you might want to go about this work.
So as I mentioned, I've already spoken about this ShareTable before, but if
you're looking at videos out of order or you're just joining us now, the ShareTable
is a system that helps parents and kids that little parts stay connected.
So commonly parents and kids may be living apart because of divorce,
separation, military, travel.
And the idea of it is it's actually kind of a piece of furniture
because the hope is to make it as easy to use as possible.
There's no buttons or keyboard in this system.
To start the connection you just open a set of cabinet doors.
That makes the table on the other side ring and if somebody there opens it,
it initiates the system.
Now what it does is it combines something like video chat, so on the monitor you'd
see the other person's face, with a ShareTable top space.
The way the ShareTable top space is created is that there's a camera and
a projector above each table.
It captures what's happening on one table and projects it on top of the other.
So in the image here, you see a paper board game being placed on one side
is being projected on the other side, and
as long as both people have something to use as tokens on their sides,
the tokens will be projected back on to the paper board game.
So, this is meant to support a lot of activities between parents and
kids, because it's kind of boring to just talk to each other, and
it's a lot more fun to do something together.
So whether it's playing a board game like the example here or
if a child puts a book on the table, the parent can see it.
Or a worksheet from school, they can help with homework.
Or if they want to have a tea party,
they can grab some cups from the kitchen and now they're having a tea party.
So we built this system, and we really wanted to see how it would be
used by families that are actually living apart, that are actually facing divorce.
So we wanted to run a field study and we had quite a few goals,
some of these were summative and some of these are formative.
On the summative side, we wanted to see whether parents and children would
spend more more time communicating when they had the ShareTable versus a baseline
comparison, versus what they did before we gave them the ShareTable system.
We also wanted to see whether children initiated more connections with
the ShareTable vs baseline because one of our goals was to make the system so easy
to use that it didn't require this sort of sophisticated scheduling or setting
settings or dealing buddy lists that young children may not be comfortable with.
So we really were hoping to see that they would initiate more of these connections.
5:06
And lastly,
was a requirement that we added after we had a few families reach out to us
who were interested but we couldn't actually set up the system in their home.
The last requirement was that high-speed internet had to be available in that area.
Now the home didn't have to high-speed internet,
we actually provided it for them.
But if there was no availability for it at all our system just wouldn't work.
And there were in fact a few rural areas and the study was done in Georgia.
A few rural areas in Georgia where a high-speed internet was
just not available at all.
Now in addition to these criteria that we actually stated up front,
there were a few kind of self-selection criteria.
So basically,
the families actually had to be willing to participate in a fairly significant study.
Because it was an 8-week study that involves significant data collection,
so it can be kind of a bit of a privacy violation.
And it included weekly interviews, so
it was quite a bit of a time commitment on the participant's part.
Even though we compensated, really for lots of busy families it's really hard to
find the time to dedicate to this kind of a study.
And the other self-selection criteria was that the families had to be low-conflict
enough that both the parents actually could agree to participate in the study.
So we weren't really looking at families where like every form of contact had
to be litigated by a judge.
I think it would be too difficult to try to deploy this kind of a system in
that setting.
So in lots of cases the families just kind of self-selected, nobody would volunteer
for the study if they didn't meet these two self-selection criteria.
But, these were quite restrictive in terms of users.
There was quite a challenge to actually recruit users for this study.
And in the end we actually ended up having to go through a professional recruitment
firm to do the recruiting for us.
Because it was just, so many constrains, so
many restraints on the kinds of families that could be in the study.
But through this process we were able to recruit two families, so four
different households to use this system for the amount of time that we wanted.
So the setting as I mentioned this was done in the the field.
So the ShareTables were actually deployed in families homes, so you see them here.
In some cases, the parents choose to put them in kids room,
in some cases they put them in kind of in a more communal family space.
So the one labeled B is in the living room and
as you see the cat has already appropriated the top shelf of the system.
And the bottom one D of is kind of in the den, the hang out spot in the house.
So it was really up to the families where they wanted to put the system.
And we made sure that we brought enough extension cables and
set up the internet in such a way that they could put it wherever they wanted.
9:49
In terms of the metrics we collected, so during the pre-deployment two week
baseline, we collected baseline measures of relationship quality.
So we used the NRI inventory for that, it's a validated psychological inventory.
We collected these communication diaries that I just mentioned from both
the parents and the children and
we did weekly interviews just to kind of see what their experience was like.
Then during the four week ShareTable deployment, we continued collecting
communication diaries, but now that we've actually put our own system in place,
we were also able to collect text logs any time the system was in use, so
person A tried to call person B but person B didn't pick up,
that would be something we could record in the logs.
And anytime the system was actually actively in use, so both the people were
actually trying to chat, we would record video of the system use.
Now, to make sure to protect participant privacy, all of that video was actually
stored locally on the machine, and when we visited the families on the communication
areas they could note, I don't want you to watch this video,
can you delete this without watching it, and in front of them, we would delete it.
Now that only happened in a couple of cases, so generally we have video records
of everything the families did with the systems.
And again,
we did weekly interviews to understand what the experience was like for them.
And then after the deployment, we continued to collect
the communication diaries, we did weekly interviews, now we did the post measure of
relationship quality to see if the system had actually changed their relationships.
And we also asked them to fill out a validated questionnaire called effective
benefits and costs of communication technologies,
which focused on comparing different systems based on the emotional costs.
The emotional costs might be something like a feeling of obligation or
loss of privacy.
And emotional benefits, which may be things like feeling closer or having that
sense of the other person being there for you, even if you're not there in person.
So we collect all of these things, and
what I'm showing on the image here on the right is kind of the folder that I
have with all of the different scripts and protocols and
questionnaires that we had them fill out and diaries and all of the stuff.
So as you see, a lot actually goes into a field study,
you have to have a lot of these things prepared ahead of time.
You have to be pretty organized,
you want to make sure that you know what's going to happen every single week, so
every time you would go there to do a weekly interview you would have
a different script, and you would want to make sure to bring the right one.
You want to make sure that you have them fill out the right
questionnaires at the right time because if too much time passes, for
example, they might forget what it's like to use the system.
So, overall, quite a lot of work went into actually organizing this kind of study.
And lastly, I just wanted to show you an example of the diary,
the communication diaries that we used.
So we actually had two different versions, so
one was a version that we used with the parents, and
one was the version we used with the kids, I think you can guess which one is which.
So basically the kids had to circle when they talked, how they talked, so
in this case, the child has circled the phone.
They could also draw something that was different from that,
they have to say how they felt after the talking.
So in this case, the child drew, so I know it's a little bit hard to read, but
that's why we also did interviews so we could interpret these.
That's excited, that's what it says, and
the child also circled the topics that they talked about.
So in this case, they talked about how they felt, and
they also talked about earrings, so
I think the girl had just gotten her ears pierced, so she drew the topic.
And then the parents had something similar,
though they probably did less drawing, so
they said the time that they communicated the approximate length of the session,
the date, how they communicated, and what it was about, and how they felt about it.
And also, as you see here, there's this check mark box for if they didn't want us
to actually view the recording videos of that session, they could mark that, so
we got through quite a few of these diaries throughout the study.
Now, the point of this is to really just give you an example of what goes into
the field study, so I don't want to go too much into the results,
but I feel like if I didn't talk a little bit about the results,
it would just be kind of too much of a cliffhanger.
So let me give you just a little bit about what happened once we deployed the system.
So this is the results of the pre-deployment, so what you're seeing is
that for both of the families, there's generally fairly little talking.
So in a week, they were averaging, one family was at five and
the other one was about at 11 or 12 minutes of communication per week.
So this was mostly very, very short phone calls, and
none of them really used video chat regularly, it was just too hard to set up,
and so, telephone was really the technology they used the most.
And, with the pie charts, what you see is what proportion
of the sessions were actually initiated by the child.
So in the first family, it was actually all initiated by the parent,
the child didn't initiate any of the sessions.
In the second family, I think that pie slice represents one of the sessions
was initiated by the child, the rest were initiated by the parents.
And so, then we compared that to what happened after we deployed the ShareTable,
what happened with the ShareTable, and what we see is that the amount of time
spent communicating each week actually doubled, more than doubled, actually, for
both the families, and the children were initiating a lot more of the communication
sessions, though just because of kind of the social practices in the first family,
it was still very parent driven.
Their rule was that they kind of had to coordinate by phone before they could use
the ShareTable,
and so, it was still mostly the parents doing the communication,
but in the second family, it actually shifted so the kids were actually doing
more initiating than the parents were and that was interesting to see.
Now the other thing I want you to note on here is if you look at week three for
family one and week four for family two, these were the weeks that we actually
deployed the ShareTable system, and you see that there's kind of, it looks like
a pretty tall spike for those weeks, and this is what we call the novelty effect.
So we just gave them this cool new toy and they wanted to use it every day, and
they wanted to figure out how it works and
they were doing all these exciting things with it.
But then you see kind of a leveling off, so, in fact, this dip that we saw in week
four and five, and for one family, week five, for the other family,
is them trying to figure out, okay, well, we tried all the kind of cool,
quirky things that we can do with the system, how do we actually now make it
work for our family, make it work for our relationships?
And that took a little bit of figuring out for both families.
We actually see a dip in use, and then you see a plateauing as they figure it out,
they now have their practices around it,
they decide that they're going to use it once a week for a certain amount of time,
or whatever practice they come up with, and you see that leveling off again.
And this is really cool because you can't see this in a lab study.
You could only see this if you deploy a system in the field for
a longer period of time.
And so, the last element that I want to talk about a little bit is the researcher
rules, because it's also part of a user test plan.
And in this case it took a fairly large team to actually run this deployment.
So there was a research lead, in this case, this was me.
The research lead was in charge of doing things like recruitment, and
getting consent from the people that were recruited, the weekly interviews with
children, and just served as a point of contact for all the issues.
If something didn't work, if there were some questions that the participants had,
those went to the research lead.
There was also a research apprentice.
In this case, my wonderful Masters student at the time, Sanica.
And you can see more of her work in the paper that I'll reference later.
So, the research apprentice handled things like the weekly interviews with parents,
so that we could actually conduct all the interviews in parallel, so
we didn't have to have a really long visit.
And also collecting and
cleaning all the log data that was coming through the system.
Tech support lead, very important, so
doing things like doing setting up the share table,
making sure that if there was some sort of a problem that it was addressed.
Also, just kind of keeping an eye on the logs, and if they see something like there
were a lot of sessions where somebody tried calling and
nobody picked up on the other side.
Was that in fact,
because nobody was picking up on the other side, or was there some sort of
a technical problem that actually needs to be addressed with the system?
And this is a research prototype, so it was actually not that robust, and
we had to do quite a bit of babysitting to make sure that
the system would work when participants needs it.
And there were still quite a few situations where there were tech troubles
that we had to address kind of on the fly.