Lesson: Acing Execution and Data Analysis Questions. In this lesson, you will learn how to ace execution and data analysis questions. Facebook likes to ask execution questions. These questions test your ability to make decisions based on metrics and are reminiscent of Our courses on KPIs and experiments. Other companies call these data analysis questions. Here's an example of a multipart Facebook execution question. Part 1. You are the PM of the share feature. How would you measure the success of this feature? What metrics would you use and why? Out of the ones that you've mentioned, what is the most important one and why? Part 2. Your team has implemented a change in the share feature and released it for A/B testing. You realized that there is an increase of 20 percent usage of the feature. Would you release it? Part 3, you have recently released the new feature, and you were formed by all local sites, Facebook sites in each country, that they usage has indeed increased by 20 percent, but the overall global data shows only five percent increase. How is this possible? Step 1, the interviewer can't move on to part 2 until I completely answer part 1. It's important to answer each part quickly and efficiently. Also, as part of your pre-interview preparation, you should have already known about Facebook's vision and metrics. Facebook's vision is people use Facebook to stay connected with friends and family, to discover what's going on in the world and to share and express what matters to them. How does Facebook make money? If you don't know it, Google it. You'll learn that advertising revenue accounts for 98.5 percent of all of Facebook's revenues in 2018. This means the users of Facebook, the people, are generally not the buyers, the advertisers. In the interview, you should analyze both users and buyers. Next, what are the metrics that users and advertisers would care about? What do you think matters most to Facebook users? If you don't know, Google that question before the interview. Facebook's financial statements names these KPIs. First, daily active user or the DAU at 1.4 billion. Monthly active user or the MAU at 2.1 billion. Daily engagement, which is the DAU over the MAU, is 66 percent. Their revenue is 10 billion and the average revenue per user or the ARPU is five dollars. What do you think matters most to Facebook advertisers? Again, if you don't know Google that question before the interview. According to Google, Facebook advertisers want to reach the most users with their ads and receive the most minutes viewed, collects reactions, comments, and shares per dollar of advertising. I'm going to ask the first execution question again. Notice how this question becomes so much easier thanks to our three pre-interview Google searches. Again, part 1, you are the PM of the share feature. How would you measure the success of this feature? What metrics would you use and why? Out of the ones that you have mentioned, what is the most important and why? Step 2, cross-examine the interviewer. There's intentional ambiguity in what the share feature means. If someone post texts or messages on Facebook, they are sharing their text or images. Is that what the interviewer means by share feature? Is a share feature something that Facebook allows its third-party developers to use? So we might think about those metrics. Let's assume the interviewer says the share feature refers to the share link on the bottom right of the newsfeed and nothing else. You've proven to the interviewer that you can extract certainty from ambiguity. Step 3, answer the question about three success metrics and selecting the most important. I'd say, since the share feature is embedded within Facebook's ecosystem, we should look to the ecosystems existing KPIs, specifically one, daily engagement, the DAU over the MAU, and two, average revenue per user. We should also look at advertiser's success such as net increase and three, ad reach by unique users reached and four, viewer engagement by the number of ads minutes watched, clicks of call to action, engagement and shares. I think the most important metric would be average revenue per user because it is a single metric that measures the success of our change for both users and advertisers. Notice that I propose for metrics instead of three to demonstrate an overall understanding of Facebook's metrics. That's fine so long as you're answering the question quickly. Part 2, your team has implemented a change in the share feature and released it for A/B testing. You realize that there is an increase of 20 percent usage of the feature. Would you release it? Here the interviewer is testing if you can differentiate bad or unreliable metrics from good ones. Put another way. The interviewer is asking, is further engineering work for a general release justified because the cohort with the changes clicked share more often? The correct answer is I don't know. That's not a useful metric. What were the results of this A/B test on the metrics we do care about, such as daily engagement, revenue per user, ad reach, and ad engagement? If they moved the needle in the positive direction, then good, let's release it. If they didn't move the needle or move the needle in the wrong direction, let's not release it, but find out why instead. Part 3, you have recently released the new feature and you were informed by all local sites, Facebook sites in each country, that the usage has indeed increased by 20 percent. But the overall global data shows only five percent increase. How did this happen and how is it possible? The question is focused on cross-examining to find the ambiguity. I would first confirm that all of the statements are true. Then probe for the ambiguity that makes both of these seemingly contradictory statements simultaneously true. One ambiguity is in the juxtaposition of local sites, meaning websites against global data. How do people use Facebook? Well, people using Facebook on their phone or other smart devices generally don't visit the website in a mobile browser. They open the Facebook app. Did our changes make the share feature more difficult to locate or use in the mobile app? Another ambiguity to cross examine is that the interviewer never mentioned the unit of analysis. What is usage? How did they define it? Is it a single logged in user clicking the share link? How does Facebook aggregate users into local sites versus global data? Does each user belong to only one site? Is there any potential of non-counting or double-counting? To catch these ambiguities, feel free to ask the interviewer to repeat the question and don't hesitate to nitpick through their language. For our see one, do one, teach one, you know what to do. Record yourself answering an execution or data analysis question. After you finish answering, re-watch you're recording with this rubric so that you can self evaluate where you excelled.
