0:01

So now we actually use the, similarity values, before we do that, we have to

calculate the similarity values. And, when we actually do the similarities

we do them on the errors,um, that we take obtained in the baseline prediction.

We basically subtract out to figure out how off baseline prediction was, just for

the training set again, 'cause we can only use the training set.

And we augment baseline predictor with the similarity values that we obtained on

the errors. and so there's a couple reasons why do

that. First is that we really need to center

these values. At zero.

and the reason I've set it at zero is that we're kind of subtracting any, any

bias out already, by the fact that we're going to an error.

So the errors are going to include zero. And in a sense, they'll be centered

around zero. And, in order to do correlation, you need

things to be centered about zero. So then you can either have positive or

negative values, so if we did the money ratings themselves, which are from 1 to

5. You know we can never have negative

values, we'd only have positives, and that doesn't give us that sign

differential that we need. So we,we subtract down and we do them

from kind of around 0 to some Positives and negatives, and second reasons that

we're really, we're trying to correct for the errors here, right?

So this is an augmenting the baseline predictor with this, similarity.

the neighborhood method, really it's called.

And, so we, we want to correct for those errors, so we don't really want to do it

on the errors themselves because that's what we want to, sort of to go away.

And I mean we're not just going to add you know the errors back and give us zero

error on everything. We're going to do it in a way that makes

sense and is not reverse engineering. We'll see how we do that in a minute, but

first let's try to calculate some of the values.

So, here I'm showing the table the table that is.

So here I'm showing the table of error values.

And I've got this again, I got this by subtracting the predictions from the

actual values. So I took the actual values and I

subtracted the predictions. So if the predictions were higher than

the actual values, then this is going to be negative which means, well, we should

have made the prediction lower than we had it.

And if the predictions are less than the actual values, then this will be

positive, which means that the prediction should have been higher.

And so we can use the positive as a negative values accordingly.

but now for movies one and two let's just just apply that cosine similarity.

to do that we have to just figure out how the users that have rated both movies

because we can't use in the equation we can't have users that have only rated

one. And clearly we can't use those that have

rated none, none of the movies. but for instance we can't use A here,

because A has not rated movie two. we can use B, we're good because B has

rated both movies. we can't use C because, well this is part

of the, test set again. So we can't use the test set.

we can't use D, because this is part of the test set.

We can't E because we don't know it's value.

but we can use F because we know both of these.

So we have to use Only b and f for this. So, for movies one and two then, we're

just going to apply that equation that we had before.

We multiply b1 times f1. So we do -0.30 times -0.05, and then

remember we add. So we add the product of the terms, so we

do this times this plus this times this So then we add .17 times -0.58 and then,

remember we have to divide by the length, and then we divide by b, square root of

b1 squared plus b2 squared. So, this is negative 0.3 squared plus

0.17 squared. And then we multiply it by negative 0.05

squared, plus negative 0.58 squared. Then, if we do this out.

We get negative 0.0220 over. 0.3041 times 0.6044.

All right, so this whole top right here comes to this.

This comes to this. And this comes to this.

And then that Is equal to negative 0.11, okay.

So the cosine, or the similarity between movies one and two, is -0.11.

Remember now, we said that we have to see whether it's closer to -1, 0, or +1.

And if it's closer To negative 1 or plus 1, then it's useful, and otherwise it's

not. And so you see, this is really, kind of,

close to 0, it's somewhere around here. so that's, that's not very useful at all.

And so these movies, we would say are reallly not, very correlated.

Now lets try movies III and V if is another example.

So we will go through again and we will see III and V now a works so we'll just

do next it will say check mark we can use a because a has rated 3,, movie III and

this greater movie V. We can use B, again because B is rated

movie three, movie five. We can't use, three because this is part

of the test set. Sorry, it's not part of the test set,

this is, value that we don't know, we can't We can use, user d right here

because these we know both these values. We cannot use e because we don't know

this as part of the test set. And, we cannot use f because this is part

of the test set. But we do have three values now, 1, 2, 3

on each of these sides, so. we have to do a little more.

There's a little more terms here. So, now, we do this the same as last

time. Which is, we have 3 terms, that we're

summing instead of just two. So we do this times this, plus this times

this, plus this times this. So we have negative 1 times negative

0.43, plus. Negative, or plus, sorry, 0.25 times

negative 0.10 plus 0.25, again, times negative 0.10.

They actually turn out to be the same, for both of those.

then we divide by the square roots of this squared plus this squared plus this

squared times the square root of this squared plus this squared plus this

squared. So we do the square root of 1 squared

plus 0.25 squared plus 0.25 squared. Times the square root of .43 squared.

Notice that I'm omitting the square root, and that's because we, when you square

something it becomes positive again. So I don't need to write this garder

rits. + .1squared + 11 squared and If we do

this, if you do this whole thing out this entire multiplication I'll leave it to

you to actually run through the calculation but you get .79 right here.

And so that's closer to plus 1 right. So it's a positive correlation and it's

kind of closer to plus 1 so we would see if these movies are positively

correlated, right. So now we can use this then we can come

up with a full table of similarity values and here I'm tabulating this.

this is a similarity between one and two you can see.

the similarities between also 3 and 5, 0.79 we just found, and so on.

And now a couple things to note. First is that this table is symmetric,

right. So we've said before that things

sometimes aren't symmetric but here they are so the similarity from one to two is

the same as the similarity from two to one.

So that's why 1 and 2 and 2 and 1 are the same, just like you take 2 and 4, for

instance, and 4 and 2, same. So you can slap, flop it over this and,

if you mirror image it over it, it will be the same.

now in, in the next in the next segment, we're going to choose one neighbor or a

movie, right. So we're having the neighbor's movies

here, for each movie we're going to choose one neighbor.

and we could choose more, we could choose two, three or four.

But we've already made the math complicated enough.

But and that would get even more complicated.

So we'll just stick to choosing one neighbor for each movie.

Which will simplifly things a lot when actually go to do it out.

And so, basically down the columns right now we'll say movie one we want to try to

find the neighbor with the highest similarity.

Right, and so that's that would be three here.

Right, and these, a lot of the backgrounds now, of green, which are the

similarities, or the ones with the highest similarities.

So one would choose three as his neighbor.

And for that reason, and again where we were finding the magnitude so we want the

magnitude to be the highest. So even though this is negative, very

negative its still really negatively correlated which is a, a useful thing and

now for 2 we are going to choose 1, we choose between point 11, point 741 and

point 88 And therefore we're going to choose this.

Even though again it's negative it's still the highest in magnitude.

For three, three would choose one because it's got negative 0.82, which is higher

than any other values in magnitude. Four would choose two just like two chose

four. Those are actually perfectly.

negative correlation [UNKNOWN] one other. So, four and two.

And the reason that they're actually perfectly negative correlation is because

there's only one value that we're multiplying here.

And so that's not really that great. I mean, normally you need more data

before you say that two things are perfectly negative correlation.

But here that's just the way it turns out.

Now five will actually choose two to be his neighbor right.

This .88 is higher than the other values. And notice twodid not choose five it

chose four. but five choose two.

So they don't have to choose each other. even though the table is symmetric, they

don't necessarily choose each other. And we could define other metrics, for

instance, we could say, all right, well I'm not going to use similarity at all

unless the similarity value is higher, than like, 0.9.

For instance. And then in that case, we would only use

one neighbor for one pair here, we wouldn't use similarity at all.

Some people do taht because sometimes it makes sense to say well, unless

similarity is high enough, I'm not going to use it.

but here we're just going to choose the most similar and just use that to do our

calculation.