0:02

We talked about how you want your learning algorithm to do well on the training set but

Â sometimes you don't actually want to do too

Â well and knowing what human level performance is,

Â can tell you exactly how well

Â but not too well you want your algorithm to do on the training set.

Â Let me show you what I mean.

Â We have used Cat classification a lot and given a picture,

Â let's say humans have near-perfect accuracy so the human level error is one percent.

Â In that case, if your learning algorithm achieves

Â 8 percent training error and 10 percent dev error,

Â then maybe you wanted to do better on the training set.

Â So the fact that there's a huge gap between how well your algorithm does on

Â your training set versus how humans do

Â shows that your algorithm isn't even fitting the training set well.

Â So in terms of tools to reduce bias or variance,

Â in this case I would say focus on reducing bias.

Â So you want to do things like train a bigger neural network or run training set longer,

Â just try to do better on the training set.

Â But now let's look at the same training error and dev

Â error and imagine that human level performance was not 1%.

Â So this copy is over but you know in

Â a different application or maybe on a different data set,

Â let's say that human level error is actually 7.5%.

Â Maybe the images in your data set are so blurry that even humans

Â can't tell whether there's a cat in this picture.

Â This example is maybe slightly contrived because humans

Â are actually very good at looking at pictures and telling if there's a cat in it or not.

Â But for the sake of this example,

Â let's say your data sets images are

Â so blurry or so low resolution that even humans get 7.5% error.

Â In this case, even though

Â your training error and dev error are the same as the other example,

Â you see that maybe you're actually doing just fine on the training set.

Â It's doing only a little bit worse than human level performance.

Â And in this second example,

Â you would maybe want to focus on reducing this component,

Â reducing the variance in your learning algorithm.

Â So you might try regularization to try to bring

Â your dev error closer to your training error for example.

Â So in the earlier courses discussion on bias and variance,

Â we were mainly assuming that there were tasks where Bayes error is nearly zero.

Â So to explain what just happened here,

Â for our Cat classification example,

Â think of human level error as

Â a proxy or as a estimate for Bayes error or for Bayes optimal error.

Â And for computer vision tasks,

Â this is a pretty reasonable proxy because humans are actually very good at

Â computer vision and so whatever a human can do is maybe not too far from Bayes error.

Â By definition, human level error is worse than

Â Bayes error because nothing could be better than

Â Bayes error but human level error might not be too far from Bayes error.

Â So the surprising thing we saw here is that depending on what human level error is

Â or really this is really approximately Bayes error or so we assume it to be,

Â but depending on what we think is achievable,

Â with the same training error and dev error in these two cases,

Â we decided to focus on bias reduction tactics or on variance reduction tactics.

Â And what happened is in the example on the left,

Â 8% training error is really high when you think you could get it down

Â to 1% and so bias reduction tactics could help you do that.

Â Whereas in the example on the right,

Â if you think that Bayes error is 7.5%

Â and here we're using human level error as an estimate or as a proxy for Bayes error,

Â but you think that Bayes error is

Â close to seven point five percent then you know there's not

Â that much headroom for reducing your training error further down.

Â You don't really want it to be that much better than 7.5% because you could achieve

Â that only by maybe starting to offer further training so,

Â and instead, there's much more room for improvement

Â in terms of taking this 2% gap and trying to

Â reduce that by using

Â variance reduction techniques such as regularization or maybe getting more training data.

Â So to give these things a couple of names,

Â this is not widely used terminology but I

Â found this useful terminology and a useful way of thinking about it,

Â which is I'm going to call the difference between Bayes error or

Â approximation of Bayes error and the training error to be the avoidable bias.

Â So what you want is maybe keep improving your training performance

Â until you get down to Bayes error but you don't

Â actually want to do better than Bayes error.

Â You can't actually do better than Bayes error unless you're overfitting.

Â And this, the difference between your training area and the dev error,

Â there's a measure still of the variance problem of your algorithm.

Â And the term avoidable bias acknowledges that there's some bias or

Â some minimum level of error that you just

Â cannot get below which is that if Bayes error is 7.5%,

Â you don't actually want to get below that level of error.

Â So rather than saying that if you're training error is 8%,

Â then the 8% is a measure of bias in this example,

Â you're saying that the avoidable bias is maybe 0.5% or 0.5% is a measure of

Â the avoidable bias whereas 2% is a measure of the variance and

Â so there's much more room in reducing this 2% than in reducing this 0.5%.

Â Whereas in contrast in the example on the left,

Â this 7% is a measure of the avoidable bias,

Â whereas 2% is a measure of how much variance you have.

Â And so in this example on the left,

Â there's much more potential in focusing on reducing that avoidable bias.

Â So in this example,

Â understanding human level error,

Â understanding your estimate of Bayes error really

Â causes you in different scenarios to focus on different tactics,

Â whether bias avoidance tactics or variance avoidance tactics.

Â There's quite a lot more nuance in how you factor in

Â human level performance into how you make decisions in choosing what to focus on.

Â Thus in the next video, go deeper into

Â understanding of what human level performance really mean.

Â