Explore
For Enterprise
Join for Free
Log In

Browse
Search
For Enterprise
Log In
Join for Free

Lasso demo

Loading...

Machine Learning: Regression

University of Washington

4.8 (4,471 ratings) | 97K Students Enrolled

Course 2 of 4 in the Machine Learning Specialization

Case Study - Predicting Housing Prices In our first case study, predicting house prices, you will create models that predict a continuous value (price) from input features (square footage, number of bedrooms and bathrooms,...). This is just one of the many places where regression can be applied. Other applications range from predicting health outcomes in medicine, stock prices in finance, and power usage in high-performance computing, to analyzing which regulators are important for gene expression. In this course, you will explore regularized linear regression models for the task of prediction and feature selection. You will be able to handle very large sets of features and select between models of various complexity. You will also analyze the impact of aspects of your data -- such as outliers -- on your selected models and predictions. To fit these models, you will implement optimization algorithms that scale to large datasets. Learning Outcomes: By the end of this course, you will be able to: -Describe the input and output of a regression model. -Compare and contrast bias and variance when modeling data. -Estimate model parameters using optimization algorithms. -Tune parameters with cross validation. -Analyze the performance of the model. -Describe the notion of sparsity and how LASSO leads to sparse solutions. -Deploy methods to select between models. -Exploit the model to form predictions. -Build a regression model to predict prices using a housing dataset. -Implement these techniques in Python.

Skills You'll Learn

Linear Regression, Ridge Regression, Lasso (Statistics), Regression Analysis

Reviews

4.8 (4,471 ratings)

5 stars
3,627 ratings
4 stars
715 ratings
3 stars
77 ratings
2 stars
20 ratings
1 star
32 ratings

PH

Apr 07, 2016

This is an excellent course. The presentation is clear, the graphs are very informative, the homework is well-structured and it does not beat around the bush with unnecessary theoretical tangents.

FR

Jan 02, 2017

This course is great. Things are very clearly explained. I am particularly happy because it helped me to understand many mathematical concepts. I will try not to be scared about formulas anymore.

From the lesson

Feature Selection & Lasso

A fundamental machine learning task is to select amongst a set of features to include in a model. In this module, you will explore this idea in the context of multiple regression, and describe how such feature selection is important for both interpretability and efficiency of forming predictions. <p> To start, you will examine methods that search over an enumeration of models including different subsets of features. You will analyze both exhaustive search and greedy algorithms. Then, instead of an explicit enumeration, we turn to Lasso regression, which implicitly performs feature selection in a manner akin to ridge regression: A complex model is fit based on a measure of fit to the training data plus a measure of overfitting different than that used in ridge. This lasso method has had impact in numerous applied domains, and the ideas behind the method have fundamentally changed machine learning and statistics. You will also implement a coordinate descent algorithm for fitting a Lasso model. <p>Coordinate descent is another, general, optimization technique, which is useful in many areas of machine learning.

Visualizing the ridge cost7:59

Visualizing the ridge solution6:11

Visualizing the lasso cost and solution7:50

Taught By

Emily Fox
Amazon Professor of Machine Learning
Carlos Guestrin
Amazon Professor of Machine Learning

Transcript

[MUSIC] Well, let's see this sparsity in action. And for these, we're gonna go back to our polynomial regression example. But instead of just doing least squared for increasing polynomial orders or looking at ridge regression, we're now gonna look at our lasso solution. Again, for different values of our tuning parameter lambda. Here we are back at our polynomial regression demo and for this lasso regression example, all we have to do is take our polynomial regression function and modify it, because we're using graph I've created in just this dot linear regression. Remember, there was this l2 penalty and l1 penalty. Well, now we fully understand what these two penalties are and we know that for ridge, we were looking at the l2 penalty. But for lasso, the l2 penalty is gonna be set to zero and instead, we're gonna focus on this l1 penalty. And in this demo, we're gonna explore different values of that l1 penalty. And I also wanna mention that here, we're specifying a solver. So Vista is one solver. We're gonna look at other ways of optimizing the lasso objective in this module, but we can of this as just a fancy version of gradient descent. So this is our polynomial lasso regression function and now we're just gonna explore a set of l1 penalty values. So again, these are lambda values going from 1 e to the minus 4, all the way up to a value of 10. And like in our ridge regression demo, we're starting with our 16th order polynomial, which was that really crazy fit in the least squares unregularized case. But now, let's think about what is going to happen in this lasso scenario. And so what we see is that when the penalty strength is really small, we don't get any sparsity at all. So we have 16th order polynomial and what we're seeing is that all 17 coefficients, cuz there's 16 orders of x plus this intercept term. So in total, 17 coefficients and all 17 of them are non-zero and that makes sense, because as lambda becomes really, really small. Remember, we default back to our least square solution where we don't have any coefficients set exactly to zero. But as we increase this lambda value, we get more and more sparsity in our solution. So the number of nonzeroes here is 14, then we get five coefficients being nonzero. And by the time we have a penalty strength of ten, we only have two of our coefficients being nonzero. So you can see very explicitly from this how lasso is leading to sparse solutions, especially as you're increasing the strength of that l1 penalty term. But now, let's just look at the fits associated with these different estimated models. So this is for our very small penalty value. This function doesn't look as crazy as the least square solution, still fairly wiggly. But remember that in lasso just like in ridge, the coefficients are shrunk relative to the ridge regression solution. So even in this case where we don't have any sparsities and none of the features have been knocked out of our model, we still have that the coefficients are a little bit shrunk relative to those of the least square solution and that's providing enough regularization to lead to the smoother fit in this case. But as we increase this lambda value, we see that we actually do get smoother and smoother fits. This starts to look like the fit that we had for our optimal setting of our ridge regression objective or the one that minimized or leave one out across validation error. But again, when we get to really large lambdas just like in ridge regression, we start to have fits that are over smoothing things. So this was a case where we only had two nonzero coefficients and we see that it's really just insufficient for describing what's going on in this case. So again, to choose our lambda value here, we could do the same. Leave one out cross validation, we did in our ridged regression demo. But the point that I wanted to show here is how we get these sparse solutions where we're knocking out, in this case, different powers of x in our polynomial regression fit. And this is in contrast to ridge regression, which simply shrink the coefficient of each one of these powers of x in our degree 16 polynomial fit. [MUSIC]

Explore our Catalog

Join for free and get personalized recommendations, updates and offers.

Coursera

About
Leadership
Careers
Catalog
Certificates
MasterTrack™ Certificates
Degrees
For Enterprise
For Government
For Campus

Community

Learners
Partners
Developers
Beta Testers
Translators
Blog
Tech Blog

More

Terms
Privacy
Help
Accessibility
Press
Contact
Directory
Affiliates

Learn Anywhere

Follow Us

© 2019 Coursera Inc. All rights reserved.