Welcome back, we've spent quite a bit of time talking about how our analysis are meant to drive towards some eventual action in the market. But sometimes, our analysis actually reveals that there are more questions we need to answer before we can move forward. In this short video, we're going to talk about one way in which we launch a new wave of analysis using experimentation. Let's say that we're a wireless company and we've done a predictive analysis to understand who is most likely to cancel their service. Those of you who have taken earlier courses should recognize this example. We actually want to do something with that analysis but we're not exactly sure what. We have a lot of ideas about how we might reach out to those customers and get them to stick around, but we don't know which ideas will work best. We also know that if we get it wrong, we can cause a lot of damage. In this case, it makes sense to do some sort of small scale test before we roll out these ideas to our larger customer base. Now there's an awful lot of material available on how to run and measure controlled experiments and on the statistics that underpin both the design and execution of those experiments. I fully encourage you to take a full blown statistics and experimental design course at some point if you can. But we won't get into those details here. Rather, we're just going to focus on a couple of general principles and a few examples of how things happen in the real world. As you might imagine, some of the things aren't necessarily ideal. the basic idea behind a controlled market experiment is that they do something to a set of customers. We call this applying a treatment to those customers, and I compare what happens to those customers with another set of similar customers that didn't get the treatment, we call this a control. If I do this in the right way, I can attribute differences in the behavior between the treatment group and the control group to the treatment itself. In an ideal world with a stable population and predictable behavior, we can assemble a treatment group and a control group by randomly selecting people from the population and randomly assigning them to each group. If we do this correctly and our sample are large enough then we can pretty easily measure what happens after we apply the treatment to the treatment group. Here's an illustration of that scenario. Let's say we were measuring some outcome with the value shown on the y axis. If our treatment and control groups are truly random and our treatment has an effect, we should see identical behavior in the groups before a treatment is applied and different behavior after the treatment is applied. This is what you see here. It's almost always best that this kind of controlled experiment can be achieved. However in the real world, it's sometimes the case we can't actually execute experiments with such tightly defined control groups. For example, let's say that the test required some sort of mass advertising or promotion. It would be difficult if not impossible to ensure that only a randomly selected set of people in a population saw a TV commercial or heard a radio spot. In this case, the best that we can do is try to find two different geographic markets that are as similar as possible. Apply the treatment to one and compare it to the other. Here's an illustration of what this might look like. In this case, the groups are not quite the same prior to the treatment, but they appear to be similar in performance. We can look at the relative differences between the groups before and after the treatment, to measure the effects of the treatment. This approach can work, but it presents some risk. It's entirely possible that something else may have happened during the experiment that impacted the two markets differently. Say for example, there was some severe weather in one market, and fair weather in the other. In this case, our measurement might have some unknown error which skews the result. Nonetheless, it's quite common to see this approach applied in business experiments. In the worst case scenario there's no way to establish a separate control group to compare against our treatment group. And the only option we have is to try and compare the treatment group against itself before and after the treatment was applied. This obviously requires us to make some assumptions about what that single group would have done had the treatment not been applied. Unlike the ideal in market to market cases, we don't have a proxy that helps us account for what else might have been going on in the market as a whole at that time. It goes without saying that this approach is the riskiest of the three. If anything other than the treatment impacted the group during the experiment, we'd have no way of accounting for it explicitly. Now the way we've introduced experimentation here implies a purposeful planning and execution process for the experiment itself. It turns out that's not the only kind of experiment we might try to analyze as our role of data analysts. It can also be the case that we're trying to gain some insights, by looking at things that have naturally occurred in the past that have some aspect of a test. We call these natural experiments. One example might be looking at how a specific store was impacted when a competitor opened up across the street. Another might be assessing whether a local customer care activity was influenced by a severe weather event in one market. Natural experiments almost never have well defined control groups. After all, we're looking at things that just sort of happened versus those that were planned. As a consequence, we're usually forced to use the less ideal group to group, or self comparison methods we outlined earlier. Whether we're looking at planned or natural experiments, as data analysts, the task of measurement usually falls to us. Generally we're looking for differences and we apply classical statistical methods to characterize whether those differences are significant or not, based on the magnitude of the difference, the variance in the measures, and the number of data points we have in the experiment. Much of analysis is pretty straight forward. However there are some common traps we can fall into if we're not careful. Let's take just a minute and talk about a couple of them. Usually our experiment involves trying to get people to do something. For example, we may make an offer to a customer with the hopes that they will take that offer. It's tempting to want to look at only the results of the people who actually took the offer, a common mistake is to assess results by comparing the takers to the control group. This is a big mistake. What about the people who didn't take the offer? How do we know we didn't have an impact, especially an adverse impact, on those customers? The rule here is to always compare the whole treatment group to the control group. It's okay to break the measures into those for takers and non-takers, but the non-takers always need to be included. The other mistake we tend to make is looking too narrowly at the impacts of the experiment. Let's circle back to our customer cancellation example. Suppose wanted to keep customers from canceling their service. It would seem to make sense to simply look at the difference in cancellation rates between our treatment and control groups in assessing the financial impact of the effort. However, there are a lot of other things the customer might do in response to our communication. Did they change their calling plans perhaps changing monthly revenue? Do we stimulate usage or additional calls to customer care, perhaps changing the cost to serve. These other impacts can have a dramatic impact on the overall success of the campaign and should be included. The way to avoid this problem is to really think about all the things that a customer might do in response to an experiment, and plan in advance to measure them as part of the analysis. So that's a quick spin through the high points of experimentation. Again this isn't meant to be a complete course on the topic, but in line with our theme of real world analytics, we thought it would be valuable to point out a few ways in which compromises to theory emerge in the work that we do and to highlight a couple of common pitfalls that we seek to avoid. Keep this in mind is you consider how you can use experiments as actions in your analytic process. See you next time.
