In this course you will learn how to evaluate recommender systems. You will gain familiarity with several families of metrics, including ones to measure prediction accuracy, rank accuracy, decision-support, and other factors such as diversity, product coverage, and serendipity. You will learn how different metrics relate to different user goals and business goals. You will also learn how to rigorously conduct offline evaluations (i.e., how to prepare and sample data, and how to aggregate results). And you will learn about online (experimental) evaluation. At the completion of this course you will have the tools you need to compare different recommender system alternatives for a wide variety of uses.
Matching Evaluation to the Problem/Challenge
Loading...
From the course by University of Minnesota
Recommender Systems: Evaluation and Metrics
109 ratings
From the lesson
Evaluation Design
Meet the Instructors
Michael D. EkstrandAssistant Professor
Dept. of Computer Science, Boise State University
Joseph A KonstanDistinguished McKnight Professor and Distinguished University Teaching Professor
Computer Science and Engineering
In this video, we want to talk about how you design an evaluation to, evaluate
a recommender in a particular context or particular needs and applications.
A question we get asked sometimes is what's the right metric for
evaluating my recommender?
There's a bunch of related questions such as how do I sample the data, how do I know
the right algorithm to use And there's a right answer to this question.
You use the metric,
the evaluation strategy that answers the question that you have.
Now that means that you need to understand what is going
on in the particular application, the domain, the use case And
then understand what each of the metric and evaluation strategy can actually
measure, and find one that matches to the thing that you really care about.
>> So, we've seen this when we went to our early tours.
We looked at Amazon, and say how does Amazon do evaluate.
When it puts a product on the screen, whether that's a good recommendation.
Now, we're not inside Amazon but we have a few pretty good ideas.
They look at, do you click on it.
They look at, do you buy it.
They care a whole lot about, do you eventually buy it.
Not necessarily do you buy it this time
They might care a whole lot less about what's the mean squared error.
They don't care if you rate it all.
Maybe they care a little bit because if it makes data for other people but
if they put recommendations with algorithm A and
you buy those things and they put recommendations with algorithms.
Be and you buy 20% less.
For that product and
maybe overall algorithm a is going to be out there running.
We want to take you through some problems where we can just talk though,
knowing there's not a unique correct answer.
Where we can talk through the evaluation, so
let's start with an example of news recommendation.
Many of you may be familiar with Google News,
Google News combines news from all sorts of sources around the web and
it does so to the extent that you're identified and logged in.
With a recommender system that is personalized but
even if you're not logged in with a non-personalized recommender.
We just started by saying what kinds of questions might we have about
the design and performance of Google News.
We might start with the question like are we putting useful articles
On the screen for people.
That seems like a reasonable question.
What would you ask?
>> I'd look at two things.
One, I'd look at do users click on the articles.
And I would look at a more advanced version of that of when
they click on the article do they stay on it long enough.
That they might actually have read it.
Or did they click on it but they bounce right back to go look for something else?
And then when we have those click traces that's also the kind of data that's
amenable to an offline evaluation.
You can look at article clicks and you can see potential new recommender algorithms.
You can see if they recommend things that the user clicks.
Depending on the exact layout you can actually even synthesize some
negative data.
Because if you recommend a list of five articles and
the user clicks number three, it's not a strong signal.
But there's an okay probability that the user decided one and
two weren't relevant and three was.
And so you can use that as a weak Preference signal that they Better than
one or two, because that you are offline evaluation to with like factor method of
this matrix to be able test the bunch of algorithms without users with them.
>> Absolutely.
But there a lot of other ways you look at useful of this articles.
Let's imagine for the moment you couldn't run a live experiment.
That you're interested in what Google does but you're outside Google.
You could imagine looking at correlated data about the articles
being recommended and saying I really want to see a statistic on
how popular Are the articles that are being recommended.
How many news sources
are showing that information, top newspapers for instance on their websites?
How wide interest versus narrow interest are the articles being recommended?
The point here is, once we've decided what we want is some measure of usefulness
We're not done, we're just starting.
We look at all of the possible ways we could analyze the data,
we look at the data that we have available and then we dig into it.
Let's do the same thing with question about finding potential for
improving a recommender.
One of the great things that you began to have available to you is
a diverse set of metrics that might be relevant for news recommendation.
We also have some metrics we haven't talked about yet
that you might deal with as you go forward.
So, we can talk about diversity.
How diverse is the set of items that's recommended on a page or
in a section of the page.
We can talk about serendipity.
How much are the things that are showing up, things that a user wouldn't have
expected to see and to what extent is that delightful, wow,
I didn't know this, I'm really glad to know about it.
We can look at notions of familiarity,
something that's being done in a lot of current work of you
know when the person sees this today say I already knew about that.
Is that a good recommendation?
Maybe, maybe not.
You know, do you want to see who won the noble price in the articles about them.
I probably do, even if I heard on the radio this was the person who won.
I might want more depth from my news browser.
We can look at things like level of perceived personalization
which is often measured with surveys.
You ask the person how much do you think what's up here is the same for
everybody versus identifying it with you.
And what we can look at in a very concrete way is a concern
that people have had about news recommenders in particular Balkanization.
Is it possible that we are dividing people into people who read
one set of articles and people who read another set And
they don't have any common ground because they just don't agree on anything.
In politics, you might think of this as warring parties or warring philosophies.
But this could also be true by country, could be true in lots of different ways.
So those are just a couple of examples.
Maybe you have some additions that you'd like to add.
>> Well, there's these are.
Often many of these are difficult to measure.
Because we talk about diversity metrics.
One thing that is an open area of research right now is what, how do we
measure diversity in way that correlates with what people actually see is diverse?
And what diversity means is going to be very domain specific Diversity in
movie recommendation and diversity in news recommendation are not the same thing.
And so you need to have an understanding of what are the factors that drive
the user response to recommendation in a particular domain and
how they perceive the items there.
In order to figure out the best way to go about measuring these.
And in many cases the research is not at all settled on
how to do that in a robust way.
There's also some potential tradeoffs.
For example, looking at the perceived personalization and
the balkanization, if you have a recommender that's very,
very personalized that might be precisely the setup characteristics that
are going to lead to a balkanization thing or a balkanization scenario.
So we need to look at the recommender from a variety of different angles,
figuring out how to balance these different concerns once we even figured
out how to measure them is something.
Think that again is going to be very very domain specific and
here's where there are not obvious right answers,
we can't tell you this is how you need to balance perceived personalization and
balkanization and here's how you measure diversity.
You need to figure out what that means for the goals of your users, for
the goals of the system operator.
And how you build the kind of compelling experience that's going to deliver value.
>> That's right. They used that diversity example.
Nobody stop the video or maybe you did and said wait, diverse about what?
Do we mean diversity in terms of the topics covered?
I want some sports and some.
War and some politics.
Do we mean diversity in terms of the sources consulted?
I want to have some big national and international newspapers, and
I want some blog sources.
Do we mean geographic diversity that I want articles about a bunch of
different places?
If you don't start to think about those questions it's hard to do this
kind of a design.
We're giving you this as a heads up because
part of the challenge we're going to give you if you stick around for our capstone,
the final part of this specialization, is to go out and
think hard about the evaluation And the design of a particular recommender.
And thinking about this examples may help you get to that point.
But before we do that, in our next video we're going to take you through two more
case study examples that are rather different from news recommendation.
There is one take away message we want you to get from this just with this example.
It's that you can't really build design evaluate or recommender for
an application, without understanding the application and it's usage.
There's no simple answer that says, this is a good recommender,
just plug it in everyone will be happy.
And.
If you want an example of that,
you'll notice one thing, none of us said yet that we sure should have.
Are the articles that we're recommending right?
Now there's lot of fake news out there and
if Google News regularly gave fake news reports as what it delivered
That might not meet its goals as a system or its goals, for the goals of its users.
If you're not thinking to answer that kind of question,
you've got to immerse yourself inside the topic.
So the lesson is pretty clear, it depends.
Every evaluation requires its own design.
And we'll be back in the next video with a couple more examples.
See you then.
Explore our Catalog
Join for free and get personalized recommendations, updates and offers.
Coursera provides universal access to the world’s best education,
partnering with top universities and organizations to offer courses online.
© 2018 Coursera Inc. All rights reserved.
