[MUSIC] As I mentioned in the last module, a working knowledge of data science will allow you to judge what good looks like, when you are presented with analytics. In this module you will earn how to tell apart good analytics from bad analytics. This is a crucial skill when teams report to you, you participate in leadership meetings, or you evaluate vendors. To get us started, I'd like you to read the Pentathlon case. This case describes a situation at a European sporting goods retailer. I want you to take the perspective of the chief marketing officer and figure out how to resolve two conflicting pieces of analytics. Stop the video, read the case, try to answer the case questions, and then come back for a debrief. As you know from reading the case, Pentathlon is a European sporting goods retailer. They have a new director of digital marketing, Ana Quintero. She wants to impose an email limit, and has pushed back from the directives of the seven departments. And so, we want to sort out why does Ana think that there are too many emails? Why is there push back from the product department directors? And then finally, what should the CMO do about it? Let's start with Anna. She thinks that Pentathalon is sending too many emails to consumers. Why is that? What motivates her to tackle this issue, is that the 4.3 emails that Pentathalon sends each week is higher than what she has seen in previous companies. But her core evidence is a survey she sent to consumers. And, in that survey, she finds that, overall, 72% of consumers think that they receive too many emails. And among the core audience of 18 to 34-year-olds, the percentage is even higher, namely 88%. Another way to cut the data, is to look at the top 25% of customers by total euro sales. These customers overwhelmingly think that they're getting too many emails. The only customers who seem more or less okay with how many emails they receive from Pentathlon, are the bottom 25% of customers by euro sales. Now Francois Gabret who is one of the product department directors has the opposite opinion. He doesn't think that Pentathlon is sending out too many emails. His conclusion comes from a piece of analytics that he had his team conduct. The team showed that consumers who received more promotional emails from pentathlon, tend to perform better. They order more, they spent more, and their last purchase was more recently. Now we have a bit of a puzzle here. One set of data suggests that consumers in general don't like getting as many email as they currently do. The other set of data suggests that more emails are associated with better business outcomes. So the case asks you to put yourself in the shoes of Colin Stark, the chief marketing officer of Pentathlon. From his perspective, how do you reconcile these two pieces of seemingly contradictory evidence. Let's get the most obvious explanations out of the way. We are going to assume that the data quality is good, and of course more data would have been nice, different cuts of the data would have been nice, but this is the data that we have. So, there are basically two ways to reconcile this data. The first line of argument goes like this, Anna Quintero's survey evidence is opinion, while François Cabret's evidence is fact. Yes, it may be true that consumers think that they're getting too much email, but as François Cabret's evidence shows, more emails drive more revenue, and this means that the survey reflects opinion, not action. After all, there are many things that firms do, that consumers don't like. But doing those things, might make good business sense, even if consumers don't like them. The second line of argument goes like this. The evidence that Francois Cabret provides, does not show that more emails drive higher revenues. If anything, it suggest the reverse, so what do I mean? Suppose a customer buys a pair of ski's, because email is handled by each department separately, this customer now starts getting promotional email from the winter sports department, but not from the other departments. Now suppose that the same customer buys cycling gear from the endurance sports department. That customer now gets promotional email from two departments, winter sports and endurance sports. Now suppose that the same customer buys from a third department, say racquet sports. Now this department sends email too, and so on. Okay now look across the customer base. Some customers buy from lots of departments, and therefore get Email from many departments. Other customers buy from only one department and therefore get email from only one department. Who do you think spends more on average? The customers who buy from one department or many departments? Probably the customers who buy from many departments. So the consequence is this. Customers receive many emails because they buy from many departments, and therefore spend lots of money. This means that revenues drive email frequency, instead of email frequency driving revenues. The key insight is that we have no evidence in this case, that more emails cause greater revenues. On the other hand, it is also true from the first line of argument, that the survey does not prove that high email frequency harms the company. So when I ask you to answer, given the evidence, not your personal opinion, would you impose a limit on promotional email activity? You should have answered, we don't have enough evidence to call it either way. Because neither piece of data tells us what effect email frequency has on business outcomes. Anna Quintero's survey evidence measures how consumers feel, but not how they act. Francois Cabret's evidence measures business outcomes, but fails to show that more emails cause greater consumer spending. In practice it is common for decision makers to draw the wrong conclusions from the kind of evidence I'd showed you in this case. And so the question is. Is there systematic way to determined, With the one should or should not, believe in analytics of this type. And the answer is yes, and that's where we will go next. [MUSIC]
