All right, so now let's transition into thinking about machine learning in the context of corporate credit risk and in particular in the context of predicting credit ratings. So let's be precise, as we should be according to the scientific method, as to what we're trying to do here. What I want to be able to do is I want to be able to develop a model that can distinguish between investment grade rated firms and speculative grade rated firms. And just to remind you, let me go back a couple of slides and clarify exactly what we mean by investment grade and speculative grade. Remember this line right here that I'm sort of hovering over, the line between BBB- and BB+ on the Standard and Poor's or Fitch rating scale or between Baa3 and Ba1. That line demarcates investment grade which corresponds to all companies rated above that line, and speculative grade which corresponds to all companies rated below that line. All right, so that's what we're trying to do. Now we could be much more precise and develop a model that classifies ratings at the individual notch level, so we could distinguish in theory between AAA and AA+, we could write down a model that distinguishes between every single one of these individual ratings. But let's keep things relatively simple for the time being and just try to allocate to these two broad buckets of investment grade and speculative grade. Okay, now the question is how are we going to measure success, how do we know if our model is doing a good job? And it's not as easy as you might think, so put differently what's our null hypothesis? Let's hypothesize here, one hypothesis is that we have a model that does better than just flipping a coin. In other words every time we see an observation that can be either investment grade or speculative grade we can always just flip a coin, hopefully our model does better than that. But imagine the following model, imagine a model that predicts for every single firm and we're going to be talking about firms at different points in time so let's think about firm year observations, right. For example Intel in 2014, or AMD in 2017 or 2018, etc., right, let's imagine a model that just predicted investment grade all the time. So if we predict that every firm, firm year observation since we'll be following firms over time, but if every single one of our predictions is that the firm we're observing is investment grade the accuracy rate for classifying investment grade firms will be 100%. Of course the accuracy rate for classifying speculative grade firms will be zero, we'll get every single one wrong because we will classify those speculative grade firms as investment grade. Now on the one hand you might look and say well we got we got 100% right on the investment grade firms, on the other hand we got none right of the speculative grade firms. And so this really comes down to clearly defining success and in this context, and most classification contexts, that breaks down to what are the costs of the different types of errors that you can make. So let's bring this together into a visual so we can talk about it a little bit more clearly. So take a look at the slide and each box on the slide, each row corresponds to an actual observation in the data. So every single firm year observation that we'll have in the data such as Intel in 2014 or AMD in 2018, whatever it is, will either be speculative grade or investment grade, that's the actual classification. Our model is going to predict either speculative grade or investment grade, and so there's four possible actual predicted combinations that we can see, right. We can correctly predict a speculative grade firm, that is get what's called a true negative, or we can correctly predict an investment grade rated firm, a true positive. Now as I said there's two types of errors, we can either incorrectly predict speculative grade for an investment grade firm or we can incur incorrectly predict investment grade for a speculative grade firm. Now I just said with that naive model of always predicting investment grade we can make our true positive rate 100%, right, but we're never going to get any of the speculative grade classifications correct. So what we need to think hard about is what is the cost of making these two errors? If they're both the same, if the costs of incorrectly classifying a speculative grade firm as investment grade is the same as the cost of incorrectly classifying an investment grade firm as speculative grade, then we just want to balance the two errors, right, or equivalently the two accuracy rates, the accuracy rates for both classifications. If on the other hand it's particularly costly to make one error relative to the other, then we're going to want to put weight on models, more weight on models that avoid those costly errors, all right. Let's make this concrete with an example, imagine you're a bank and you're lending money, okay. Now if you incorrectly classify a potential borrower as investment grade when they are in fact speculative grade, that can be very costly because you're not going to make enough money, you're going to give that firm relatively loose terms on the loan, a low interest rate, maybe loose covenants, so that you are more likely than not to lose money on that loan. Whereas if you classify an investment grade firm as speculative grade, well the loss there assuming they take the loan isn't so bad, right, because you'll be charging a relatively high interest rate to a safe firm, you're going to make more money. Now of course there's the argument that if you try and do that you won't get the business in the first place and there's a cost to that as well. But one might think that mischaracterizing risky firms as safe could be particularly costly for a lender over any significant amount of time or significant number of deals. In which case the box we're particularly concerned about are these false negatives, we do not want to inaccurately characterize or classify speculative grade firms as investment grade firms so we really want to tamp down on those errors, for example. But of course, right, it's not that there are no costs here, if we just start charging every safe borrower that comes through the door really high interest rates no one's going to want to borrow from us, okay. All right, so it's just an example to illustrate that there are relative potentially different costs associated with different types of errors. So which model is a success really depends upon what your objective function is, which needs to be clearly articulated before you start the modeling process. In other words we need to answer this question what is success very clearly and precisely up front before we dive into the modeling so we know what we're trying to accomplish. All right, so let me summarize this. Our goal is to try and develop a model that distinguishes between investment grade and speculative grade, success in our example is going to be one that minimizes both errors. We're not going to assign more or less weight on any individual error, we're going to see how well we can accurately classify both speculative and investment grade, that's going to be the goal, that's where we're headed.
