This course has been a very brief introduction to supervised machine learning and as it relates to Sports Analytics. We've talked about classification and regression techniques. We looked at both the SVM and tree-based models. We've also touched on a number of important considerations along the way. The train test validate method and how you create those sets. Evaluation metrics like accuracy and techniques like cross-validation, how to interpret the confusion matrix, which is so incredibly important when it comes to actually putting together a model and deciding to go with it to some deployment, and then how you can bring multiple different kinds of models together and create these ensembles. There's so much more though. This is really just the tip of the iceberg. We talked about class imbalance, for instance, but we haven't talked much about how to deal with it. Instead, all we did was stratify our data sampling so that our test and our validation data sets looked similar as far as imbalance, but we can actually boost the number of samples that are in our classes by generating new synthetic ones, or we can minify some of this and bring down the size of our dataset, which allows us to use some other methods our linear SVMs that take a long time with balanced classes. There's also additional evaluation measures, and these measures are commonly used: area under the curve, precision and recall, F1 score and so forth. We talked really just about accuracy and then a weighted accuracy in the confusion matrix, but these other measures are important to know as well. Scaling and manipulating features to improve model accuracy is one thing we didn't talk about really at all, and that's probably the biggest flaw in this course from a machine learning perspective. Understanding how to scale and manipulate your features to create new features and to make sure you're features are balanced with respect to what you'll see in the future is really important, and that goes into this feature of collection and engineering techniques. Where do we get our features from? How do we bring those features together and how do they become a meaningful? Of course, there's a plethora of other modeling approaches: Bayesian methods, neural networks, deep learning is very hot right now, for instance, and we didn't even get a chance to talk about those other kinds of machine learning: unsupervised machine learning or clustering, or reinforcement learning, which I think is incredibly powerful when you're trying to do moment by moment or play-by-play predictions. I don't want to leave you empty handed, I want to give you a few resources that you can take to continue learning on your own. First off the book, Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow. This is a wonderful book. I think it's great, it's affordable, and the newest version of it is actually got really high production value as well. I can't recommend this book enough. On Coursera, if you want to dig into Data Science a bit more, you can do that with myself and some of my colleagues here. In this Applied Data Science with Python specialization, I tackle some data cleaning and data visualization. My colleague Kevin tackles in more detail the classification tasks that we've talked about, and he talks about dimensionality reduction, which is incredibly important as well. My colleague Vinod talks about natural language and how you can start to understand information from natural language. You can imagine how this is powerful with respect to, for instance, tweets about players and how that public sentiment might be important. Then my colleague Daniel, he goes into networks, and networks are really interesting aspect of Data Science that I think are being used more and more. Certainly we use them all the time or see them being used when it comes to search, and Google PageRank, for instance, is a network problem, was when it was first conceived. We can start to think about teams and model teams as networks of players. But there's also other places that you can go, and one of those is with us at the University of Michigan, and our Masters of Applied Data Science or MADS program has a Sports Analytics class. If you're interested at that might be something for you there as well. Speaking of this program, one of the students of the program, Anthony Jove, helped me write some of the content for this course and brought together the data sets, and I've got an interview coming up with him where he's going to share some of the work he's done in Sports Analytics because of his interests and because of the skills he's getting in the program. If you want to learn a little bit more about baseball, the Hall of Fame of baseball, data acquisition and cleaning and so forth, catch us in the next lecture. Thank you for taking this course. It's been a pleasure on my side of the camera and I'm looking forward to interacting with you in the forums or wherever we might run into one another.
