So let's start with Bob. We can see Bob down here in the lower right of the table. Now remember, I asked you to make a prediction for Bob or the Bobs. I asked you how many donations out of the next five do you expect him to make? You can see that according to the model, this isn't necessarily the true number but it's close to it. The prediction is 3.75. I bet some of you are looking at that number say, woah, that's pretty low. I bet that some of you predicted five. I bet that some of you said look, Bob went six out of six. He's going to get every time you give an opportunity. So out of the next five, he's given five. And that might be a little over optimistic but maybe even a little naive to say, is he always going to give, is he always going to be around to give. When we think about our story, when we explicitly recognize the randomness in people's behavior, we realize, you know what, there's a pretty good chance that over the next five years, things can happen even to the Bobs, that would cause them to drop out. And even while they're alive and contemplating a donation, it's not a 100% sure that they will give. And so you might be shocked at this low number 3.75. You might say, wow there something weird about that data set or that model. But let me tell you that it always works this way. I've looked at lots and lots and lots of data sets where we track those really high end customers. Those ones who are really, loyal. Those ones we think would go through the gates of hell to stay with us. And we look at what they do in a future period and it's often much, much less than you think. This by itself has very important implications. Cause I see lots of companies who say, woah, let's build the president's gold medal, blue ribbon, red carpet club for the bobs. Let's show them how much we love them. They try to enhance all that value. And so we set up this special club or whatever and then we watch what the Bobs do over the next five years, and we say whoa, what happened to the Bobs? They used to go six out of six, and now they're only going 3.75 out of 5. What did you do to the Bobs? You're fired. While I'm joking around, I'm also pretty serious, that if we don't have the right long run expectations for our customers, we're often going to over or perhaps under invest. And the ways that we judged the quality of those investments might be way way off. So there's a cautionary tale just about the Bobs, but he's not the only one we want to look at. We can shift our eyes to the upper left and we can say, now I'm sure that all of you looked at Sarah and said, I bet that she's not going to be worth a whole lot in the future. Out of the next five, she'll make maybe a fraction of one donation. And you're absolutely right. According to this model, our expected number of donations out of the next five for the Sarahs would be 0.07. So once again any individual Sarah is just a gentle whisper. But collectively the Sarah's are a loud roar. And that's why we don't necessarily want to worry about any one of them. There are 3,464 Sarah's in this data set. And so collectively they're actually worth quite a bit. They're collectively worth more than most of these other RF cells that we see in the table here. And then there's Mary and Sharmila. So you can see where Mary and Sharmila are located in the table. You can see that Mary donated one time less than Sharmila but was with us most recently. So if you zoom your eyes in on that spot of the table, you'll see that our best guess for the Mary's is 2.71. In other words, for those Mary's four of those people, regardless of the actual pattern of zeros and ones, before those people who made four purchases, four donations, the last one being at the most recent opportunity. That our best guess for them would be 2.71 out of the next five. For the Sharmilla's, this one's the interesting one, even though Sharmilla went five out of six, Sharmilla's really good. She donated over 80% of the opportunities that we gave her. But because she missed the most recent one, there's a much greater chance that that misdonation might be a signal of, or actually going away completely, and not just skipping one, that the Sharmilas are only worth 1.81. Notice out of the next five, we only expect the Sharmilas to make 1.8 or about 36% of the donation opportunities. And the Marys are worth 50% more than the Shermellas. Now, for many of you who said it's no brainer, of course Mary is worth much more. You might not be surprised at that. But those of you who voted for Sharmilla, and I see you out there, you'd be shocked to know that not only is the Sharmilla, despite her five donations worth less than the Mary's, who only made four. But 50% difference, and that's really huge. And once again, we see this kind of pattern in every single data set of this sort. I'm not saying that the number is always 50%. Sometimes its smaller, sometimes its larger, as you'll see in another example that I'm going to show you soon. But it's substantial and it can't be ignored. And the idea that recency trumps frequency is just a very, very important lesson to keep in mind. And of course, finally, last and least, there's Mary and Chris. And in this case, because the model only looks at recency and frequency and because Mary and Chris have the same recency and frequency we're going to say. They're the same. Yeah, we can look at their specific patterns and maybe try to differentiate them but why bother. When we're looking at Mary and Chris in the context of everybody else, they're pretty much the same anyway so let's not slice it any more finely than we really need to. So I hope you appreciate the predictions from the model. Not only because we're going to assess how well this specific model does but some of the lessons that we learn about how, say the Bobs become less valuable and the Mary [INAUDIBLE], those are general ideas that when we watch a customer base over time regardless of what model we are using. Maybe no model at all. When the time our predicted analytics is not necessary. But hey here a statistical model that works but its having insight about things that will happen in the future. That we have another chance to observe yet. So this patterns are very generalizable. But to really convince you of that we need to see how well the model works. So what I want to do is to take this table that we've been looking at, and to kind of summarize it where we can look not only at the model predictions, but the equivalent actual numbers. And instead of showing you the actual number for every one of these specific recency frequency cells, I want to collapse them together and just look at it on the basis of frequency and then on the basis of recency. So what I'm going to do is, I'm going to take our table and I'm going to look at the rows and I'm going to average across the columns. We're going to take a weighted average across the columns to say what's the expected number of purchases that people will make based on how many purchases they have made and how many actual purchases do they end up making out of the next five opportunities. You can look at these two graphs right over here and you can see how well the model predicts the actual both for our frequency and for recency. If you look at the frequency graph on the left at the top of that graph would be the Bobs. Those would be the people who made six donations in the past, which can only be the Bobs. That number at the tippy top of that graph would be 3.75. If you look carefully, you can see it. That's the prediction according to the model. If you look just below that, you'll see the actual number associated with the Bobs. And that number is 3.53. In other words, the model over-forecasts the number of purchases that the Bobs would make in the future. Now it's close, it's actually quite close, it's within, say, 5%, something like that, and in fact it's an even lower number, so if you thought 3.75 was low, like whoa, what happened to the Bobs, In actuality, it's even lower than that. So you got the Bobs up there making six purchases. You have the Sharmilas making five, the Marys making four, and all the way down at the bottom of that graph, you have the Sarahs making zero. And in all those cases, the model does a pretty good job of predicting the actual number of purchases by frequency group the graph on the right would be the equivalent graph for recently. So it averages everybody on the basis on when you made your last purchase. So in this one we're going to take the Bobs and the Marys and kind of throw them together, because they made their purchases in the most recent cell, or period. We're going to say, how many purchases do we expect people to make on the basis of when they made them as purchased? And how many purchases did they make? And once again, the actuals, the expecteds map each other out pretty well. Admittedly, this graph isn't quite as pretty and perfect as the other graph. Especially towards the let side of it. Those people who haven't made purchases for awhile, the model thanks to be killing them off and underestimating how many purchases they will make but it's not bad. It still does a pretty good job. Considering this is a very simple model that we are running and projecting over a five year period, I think we can live with those errors. Yes, I have all sorts of other academic papers that close that gap further to change the story with the coin flips and so on. If you are interested, I could point to some papers for you. But I think you get the basic idea. The model did a very good job at predicting the number of purchases on the basis of recency or frequency. But let's roll it all together and make overall statements about purchasing for the customer base as a whole. And that's what you see in these two pictures over here. The graph to the left shows you the cumulative number of purchases. So of these 11,000 people, let's just watch them buy, buy buy. As time goes on, you can see where we break out the six year model calibration period from the five year forecast period. And in that five year forecast period, it's very hard to see any difference between the lines at all. The models nearly perfect. It becomes a little bit less perfect when we look at it on a year by year basis. That's what the graph on the right is showing and this is actually a very diagnostic graph. In any one year the model's going to be a little too high or a little too low. Again, it's not necessarily our goal to hit the mark in every single year, it's our goal to map out a pretty good trajectory over the next five years or maybe beyond. If we were standing at the end of 2001, at the end of our model calibration period. Remember 2001 was a horrible year. 9/11. The recession starting. What are we doing with those factors in this model, we're ignoring them, completely ignoring them. We're just saying, people just flipping their coin, same as ever. This shows you the power of this as if random story, that we can ignore very impactful factors. And if someone stood over here in 2001 and said this is my prediction for what the trajectory will look like over the next five years, and we watch the way that trajectory emerge from 2002 to 2006, it's pretty amazing. So again, sometimes we're too low, sometimes we're too high, but to get an overall feel for what the purchasing would look like for that group of customers. It gives you a good feeling in the belly that if we wanted to project beyond 2006 into 2007, 8, 9, if we wanted to make statements about overall customer lifetime value, we could do a pretty good job of that. That's the analysis I am going to share with you for this one data set and I think you'll get a pretty good idea about it. In the actual paper that we wrote, we show all kinds of other pictures, all kinds of other diagnostics and all kinds of other details and stories. I think you get the highlights. I just want to spend time showing you one other data set. So I want to show you the same kinds of things. I'm not going to give you all the parameters. I'm not going to show you the estimating spreadsheet. I just want to talk about the same kind of recency, frequency trade off table. This is going to be with another non-profit, but one, we we're watching the watching the donors, the customers for a much longer period of time. You can see that in this chart, right over here. The same basic idea that we have recency on the top. We have frequency on the rows. Just many, many more opportunities. But you can read it the very same way as the table that I showed you with Bob, Mary, and Sarah and so on. In fact, we still have the equivalent of the Bobs down at the bottom right. So these are the folks who have donated every single opportunity since they were acquired. You can still see the trade offs between the Marys and the Sharmilas. In this case, the Sharmilla one is very interesting. Here are folks who made, out of a total 19 opportunities, they donated, out of the first 18, they went 18 for 18 and then they skipped. So if you went 18 out of 18 and then you skipped, it's not a matter of, I forgot to send you the check. Something more fundamental happened. Of course in our model we would say you died but there's a very, very large chance that you dropped out. We don't know for sure, it's impossible for us to know for sure. But there's a much higher likelihood that if you went 18 out of 18 is skipped, there's a much better chance that you dropped out compared to our original Sharmilla, who went five out of five and then skipped. Again, I hope you have this intuition. It's not about the model. It's about understanding these patterns of future looking behavior. As you can stare at this thing you can see the way that we framed as a heat map to really understand who the best customers would be and you could really start to see those trade offs between recency and frequency. Once again how recency Trump's frequency. The last thing that I want to show you, one of the other nice diagnostics that emerges by this buy diadem model is this idea of how alive are the customers. Let's not only look at how many purchases we think they'll make, but let's look at the likelihood that a customer like this with this particular RF pattern would indeed be alive. And so that's what we see in this graph over here. Now if you look at it, on the right side every body alive, and that makes sense. If you made a donation in the last period, regardless whether it was your first donation since being acquired, or if you're a Bob and you've done it at every single time we know that you had to be alive. There was 100% chance that you were alive last time, and so there's only a small chance that you might drop out as we move to our next period. So, you see a slightly different pattern here. Once again, if we look down at the bottom of the table, you can look at the Shamillas and see, you know what, if you went 18 out of 18, but you didn't donate in the most recent time. You're chance of still being alive is fairly low. And then you can look at the other end of the table. You can look at the Sarahs and say, the Sarahs, they're probably not around with us. If you haven't donated for the last 19 years, you're probably gone but on the other hand, it might surprise you to note that while the Sarahs, there's not a lot left in them, but those Sarahs are actually more alive, or at least have a greater chance to be alive, than someone who used to donate a few times but hasn't been around for a while. If you used to donate, but now you've dropped out, kind of like the Sharmilla story, then you are deader put it in quotes, than someone who's just never donated at all. because with a Sarah, we don't know if it's that she's gone or if maybe she's actually alive, but her donation propensity is just so, so light, so rare, that it's going to be a while before we see her do anything. So our guess is that the salaries are actually more allied than someone who used to donate but isn't around anymore. They've given us good reason to believe that they've dropped out. And once again, it's not just to show off one particular model, but these are forward-looking insights that emerge from customer-level datasets all the time. And again, happy to share much more information about it but I think this is just a really nice example of predictive analytics in action and I hope that you appreciate it.
