Now that we know that experiments are the best way to evaluate the effectiveness of your advertising, let's talk about how they work. In the previous video, we learned how experiments are set up. Experiments are a comparison between two groups. A control group that didn't receive a treatment and a treatment group that did. The treatment involves one variable that you change while otherwise keeping everything in the two groups the same. We used the example of two plants that were treated exactly the same except for one variable: one got fertilizer and the other one didn't. When we test the effect of advertising, being exposed to an ad becomes the treatment. One group of people is exposed to ads. Any other group isn't. But people aren't plants. You can't neatly control them and make sure nothing in their experience is different besides the ad exposure. So that makes setting up an experiment a bit more complex. How do we deal with human variation and still set up experiments that work? That is where randomization comes in. In order to compare the effect off changing one variable between two groups of people? It's important that two groups of people be a similar as possible. In order to create two similar groups, you can use randomization. Randomization is the process of signing people to the different groups, by chance or randomly. By doing that, you distribute people with certain characteristics equally over the groups, and that way you create groups that are as similar as possible. Experiments in advertising use randomization to assign people to a control group or a treatment or test group. Studies that use this type of experiment are referred to as Randomized Controlled Trials or RCTs. Let's take a closer look at what it takes to design a good, random controlled trial to assess the effect of advertising. Let's go back to Calla & Ivy, a flower business in Amsterdam. Imra, the owner plans a campaign to drive online sales of the new fall bookcase she's created. She's interested in knowing whether it's really worth it to spend money on advertising, so she plans to run an experiment. Here's what would need to happen in order for the experiment to really prove to her what the effect of the campaign was. First you'd identify a group of people you'd like to show the campaign to, in other words, the target audience. Out of this target audience, people will be randomly assigned to either the test group or the control group. People who are assigned to the test group may be exposed to a treatment. In other words, they may see Imra's ad. People in the control group don't receive that treatment. They won't see the ad. After the campaign runs. Imra can compare the results between the control and the test group. In this case, she'd look at the difference in conversions or sales of the fall bouquets in the test group-- the group that saw the ads, compared to the control group-- the group that didn't see the ad. If there are significantly more conversions or sales from the test group than the control group, we can confidently say that the advertising campaign worked and had a real impact. Well designed RCT's can show an ad campaign's true impact. Any difference between the groups is likely to have been caused by the ad or campaign. This is also referred to as the campaign's incremental impact. There's one more thing to consider when thinking about randomized controlled trials in advertising. An RCT is set up so people will have the experience intended for whichever group they were randomly assigned to. They see the ad in the test group or they don't see the ad in the control group. However, it's not that easy to achieve this in advertising. This is because it's not always possible for the researcher to make sure that everyone in that test group receives the treatment. I may intend to show you an ad on Facebook, but what if you don't use Facebook while my test is running? This could mean that you're not getting the treatment that I intended for you. Should you then move over to the control group? Well, actually, no. That's not a great idea, because then my groups would no longer be random. If I send over all the people that weren't using Facebook while my ad was running, my control group wouldn't be as similar as possible to my test group any longer. That's why well designed tests use intention to treat or ITT. ITT keeps all experiment participants in the group they were originally assigned to, regardless off what they see or do. This minimizes the effect of potential bias. Let's talk about ITT with an example. Imagine the medical community is testing a new drug and you're placed in the test group you're given the drug to take. But the people conducting the experiment can't always control whether you actually take it. You may forget, or you may decide that you don't want to take it. In a ITT setting however, your outcome will be measured as part of the test group, regardless of whether you actually took the medicine or not. It's the same in advertising. Whether or not you actually receive the treatment, you will remain in the test group, and your result will be part of the test group. When you use intention to treat, it reduces the effect of the treatment, which makes it a bit harder to achieve significant results. But it keeps bias out. So experiments are a powerful way to measure the effect of a campaign. But if you plan to use experiments in your role as an advertiser, make sure that these two conditions are met. The experiment should use a randomized controlled trial design and adhere to ITT or intention to treat for the test group. This way you can be reassured that your test is not biased and that you can be confident in your results.
