So we've explored a bunch of different models, we've have explored different data to push through those models. Another important element of the process is what I call error analysis. In other words, where do we go wrong, how are we misclassifying firms? Let's try and understand that. So what I've done here is I've put up a little table. Each row represents a different rating category from AAA, all the way down to or up in the table to BBB- which is the last rating that is still investment grade just before it goes below investment grade, okay? You've got the total number of observations in this column. I've got the number of mistakes or errors, the error rate, which is the ratio of these two numbers, and then I've got the average value for each of the input, the data input, the input variables, the predictors that go into the model. So in this example I'm using the moody specification that consists of interest coverage, leverage profitability, leverage volatility, revenue stability and firm size. So for BBB- rated firms, there were 862 in the training data. We mistakenly classified 234 of them as speculative grade for an error rate at 27%. Those firms have an average interest coverage ratio of 6.44, leverage ratio of 0.5, profitability of 0.04, volatility of 4.96 and so on, okay? So this is just giving us a perspective of where we're making mistakes in classifying and some characteristics of those firms in that bucket, not of the mistakes, but of those firms more broadly. Okay, now, notice we didn't get any of the AAA's incorrect. That's good to know because if we're missing AAA, if we're getting those wrong, something must really be wrong with the inputs. Because they're so far from that investment grade speculative grade boundary, they should be easy to identify. But we did in accurately classify for AA minuses and so I'm going to take a look at those, all right? I've got three of them on the screen. These are three of the mistakes and they all correspond to Alltel Pennsylvania in the mid 1990s. So when I look at the interest coverage ratio of 11:10 and I compare it to a AA-, it's quite a bit lower, but still comfortably investment grade 11:10 is still AA rated. So I don't think that's the issue. Leverage ratio of around 30%, the leverage is actually quite conservative, as we can see. Profitability is 0.9, again, comfortably in the investment grade range, right? Investment grade firms tend to have higher profitability. So this is a bit puzzling so far. Perhaps it's volatility it's leverage is very volatile, 2.6, no, hardly, hardly. Although leverage volatility almost seems to be quite a bit higher for the more highly rated firms. I'd want to compare that against the leverage volatility of speculative grade firms, but that could be one reason right there, it seems to be little bit low. Revenue stability 5.5, well, it's revenue stability is quite low, so that's a little bit disconcerting. However, what really stands out, look at the size of the firm, okay? Alltel, it's firm size and I think this is total assets in millions of dollars. We're looking at about $234 million, 34 to 275, let's say. But when I look at the size of investment grade firms, they're big. The assets are in the billions of dollars, right? You can see that there's almost a monotonic relationship in terms of size, it's certainly loosely speaking increasing as we move from riskier or lower investment grade rated firms to hire investment grade rated firms. But when you compare the $250 million dollars of assets, let's say, to 7 billion, I mean, this just looks like a tiny little firm. And I'm pretty sure that, and to a lesser extent, the revenue stability and maybe leverage stability is what's throwing things off here, that's what's throwing things off. Now, the goal, just to be clear here is not to get every single firm your observation are correctly classified. We can do it with the most highly parameter rise and complicated model that will get every single observation correct, but as soon as we take it out of sample, it's just going to choke, it's going to do terribly. So we're naturally going to miss some. But what we do want to do is we want to understand why our model is making mistakes because if it's making mistakes in a systematic manner, we want to be able to incorporate some measure into the model that will capture that variation, okay? And so if I look at these other errors, if I print out the data, which I should and look at why I'm missing misclassifying some of these firms of speculative grade. If I see that it what keeps on coming up is firm size and revenue stability. Maybe I want to move to a more flexible functional form for firm size for example, or maybe firm size isn't important. I happen to know it is, but I want to try and tease out some common theme among my errors, learn from it and improve the model. In fact, that's exactly what the process of boosting a model does is it learns from its mistakes, okay? All right, so another important element of the modeling process is error analysis. So we can understand why the models making mistakes, learn from those mistakes and improve the model to improve the classification accuracy.
