[MUSIC] Welcome back, let's talk about what it looks like when a user is put into an experiment. Some software features are aimed at a particular situation. So in order to see the new feature, some eligibility criteria must be met. In addition to the general eligibility criteria, sometimes the user must perform an action. Maybe something simple, like just visiting the site, or perhaps something more complicated. In this lesson, you'll see some potential problems, and be able to practice the data quality checks that I introduced in the first module to figure out what could go wrong. But this time, it'll be in the context of AB testing. So let me show you what I mean. Let's talk about what it looks like when an user is put into an experiment. Some software features are aimed at a very particular situation. So in order to see a new feature, some eligibility criteria must first be met. In addition to the general eligibility criteria, sometimes the user must perform an action. Maybe something simple like just visiting the site, or perhaps something more complicated, in order to see the feature. When an eligible user triggers, or would trigger, the new feature, we call that a diversion point. At that point, we will include them in the experiment, and record which test assignment they were given. Sometimes this step is called treatment, which can be a little bit confusing, because the test variant group can also be called treatment. So I could say that a user was treated with control, or that they were put into the treatment group, meaning that they were shown the new variant. Sorry, these are just the words. Okay, what information do we need to record about this? A treatment event should have the user ID, the time of the treatment, so we can look just at the actions that happened after. As well as the test ID, assuming we're running multiple experiments, and the assignment, whether it was treatment or control. So let's first go through a kind of simple case. Imagine, we already send a welcome email when a user creates their account. We'd like to update the designs to include images and links to our top selling items. So for eligibility, we're going to look at users who've created an account recently with a valid email address. For the divergence point, we're talking about whenever the automated welcome email is scheduled to be sent. The control is to get the existing email. The treatment is to get the new welcome email, with these images of top selling items. For engagement metrics, we're going to look at email opens, email clicks and whether or not an order was placed, the total number of orders placed, and revenue. So what does the treatment look like? There's three kinds of users at this point. There's users who are not assigned into the experiment, so that could be existing users who don't need another welcome email. We could give them the top selling variant, so that's new users who create an account after the beginning of the experiment, who will receive the top selling variant email. And then there's also the control group. These are new users who created accounts after the beginning of the experiment. Okay, that seems simple enough. Let's go through a slightly more complicated case. We'd like to send a push notification to mobile users if they have items in their cart that they have not yet completed in an order after 24 hours. And only if they haven't received a push notification from us in the last two weeks. So here, we've got some complicated eligibility. Users with a view item event on a mobile device, who have added items to their cart in the last 24 hours, but haven't received any other push notifications. The control is no push notification. The treatment is personalized push notification about the items that are still in your cart. Engagement metrics are push notification opens, mobile app visits, and item views. The business metrics, kind of the same every time, orders completed, total orders placed, revenue. Here, what does treatment look like? We need to log the eligibility for treatment and for controls, so we can compare the right populations. We have all of these people who are not assigned. Maybe they're existing users who haven't put anything in their cart, or they haven't used a mobile device. Then the variant, they get the push notification reminder. Or control, they were eligible to get a push notification, but they did not. Okay, now let's talk about one of the most common questions I get asked. Pretty much every engineer I've ever worked with has asked some data person this at some point. The question is, can we run multiple experiments at the same time? Okay, let me explain how this is possible. Imagine we wanted to run both of these experiments at the same time. There's a few things we need to make sure of. One is independence. We assign the treatment groups independently, which means that the welcome email you get doesn't affect whether or not we'd send you push notifications. And then eligibility, we only include users that are eligible. There might not even be that much overlap here, because we're talking about one experiment on new users, and another on frequent users, maybe. Analysis, we are only going to look at the things that happen after the divergence point. So if a new welcome email was super awesome and the user added a ton more stuff to their carts, we'd see that represented equally in both sides of the experiment for the push notification. Of course, the first time you run an experiment, something will probably go wrong. So let's talk about what kinds of things typically go wrong. One thing that is very common is to have imbalanced assignments. Maybe you wanted it to be 50% gets the variant, 50% gets the control, and it doesn't look quite right. Another thing that can happen is improper logging of the diversion point. So maybe the event is missing a user ID, maybe it's missing a test ID, maybe it's missing something. This is probably going to happen the first time you try to run an experiment. And then, another thing that can happen is maybe just a bigger problem in your data system is maybe there's some missing data, maybe it's delayed, etc. So that's something you should be ready for. So now, how do we deal with it? Question is, should I panic? What if the test split isn't exactly 50-50? Well, it's probably not actually going to be exactly 50-50. You can intentionally run experiments of different splits too, just be careful. Make sure that it's within an acceptable range. What if the test split percentages are changing over time? That does seem like a warning sign, you should figure out why that's happening. What if there are days of treatment data missing? You should track those down. And what if there are multiple experiments running at the same time? Nope, no need to panic. Really, just go through the exercise, make sure you're checking independence. Make sure you ask, will this test and the control groups be equally represented in an unrelated experiment? And if the answer is yes, no need to panic.
