Let's talk about how to get great outcomes with your analytics ,and or data science resources. These may be people that are focused on your particular product, on your product team, or there may be a pool of resources they have access to regardless. We're going to look at kind of the jobs of of doing this, and then how we get to good solutions, how we create good interfaces with these folks. The hypothesis testing here, has really the same success metric as continuous design, and as a product manager, one of the most important things is to create continuity between these two. Go out and work like a designer to create good hypotheses, about what is valuable to the user. But then pair those with a scientific testable approach, to seeing if the users really use them the way that you thought they would, doing this in small batches to minimize waste. And so by analytics, I really mean just the kind of basic descriptive analytics that would tell you, did the user use this feature or not? And by data science, there we're using large datasets to make predictions, or even prescriptions, recommendations, about some kind of action for the user, and so how do we do this? Well, here again, this is tightly coupled to how much of our content is successful, by virtue of the fact that. It's really kind of the other side of the coin, of continuous design in this sense, and the way that we want to work with these folks to create good outcomes, is very similar to the other functions. Just like design, if we bolt on analytics, we bolt on data science after the fact, as kind of an arbitrary thing, it's very hard for those people to do good meaningful work. But if we understand enough to bring them good, vivid, actionable inputs, then we can get really great outcomes with this team. And so principally, there's really just one skill that you really have to master to be good at this, and it's being able to frame dependent and independent variables. And so for example, we talked about an experiment design, what we're saying is, well there's certain independent variables that we want to get in our observations. So for example we mean classic kind of trivial example as well, we changed the button, we A/B tested having an orange button. So the button was blue, we introduced a treatment to a select set of users, that made it orange. Did that make people click through to where we wanted more, or did it not really affect it at all? So the change of the button color would be our independent variable, and our dependent variable would be, we want click through rates to improve. And usually these are some kind of ratio or rate, we'll talk more about how to do that in the next week. Here's a simpler example, let's say that we're managers of an ice cream shop, and we bring in a data scientist to help us forecast demand for ice cream, pretty typical thing to do, time series forecasting. And so the data scientists or analysts comes back to us with the forecast says, okay, here's, on this day, we think you're going to sell £18.2 of ice cream these days,12.4. Well, that probably isn't very actionable, the reason is, the dependent variable wasn't really thought through design very carefully. What is the intervention, what is the action ability of that forecast, that's how we get to framing good dependent variables. And it's probably something like, let's say labor is our main cost driver, well then the DEV really is. How many people do we need to bring in for a given shift based on this forecast, or how few people can we get away with, and still provide a good level of customer service? So we need a forecast that doesn't just tell us pounds of ice cream, but helps us think about, how many staff we're going to need a peak periods for these various shifts? And the dependent variable can't be a bunch of fancy graphs, and predictions, and envelopes. It just has to be something that a shift manager can look at, as they're making up the schedule and say, okay, on this shift, we need two people, on this shift we need one person, and so forth. So when we talk about framing the dependent variable, that's a lot of what we mean is, how do we make sure that this is actionable? Because the most important thing, the most important, most observable attribute of the strong analytics and data science program. Is that the results are changing your behavior, and and they're making you, they're helping you do things differently, do things smarter. If they're just kind of an interesting thing that you look at and puzzle over, at the end of each sprint, that' probably just a waste of time. Or maybe, hopefully, get a more optimistic view, I guess would be a stepping stone to get into that capability. So framing the dependent variable, understanding its relationship to independent variables, something we'll look at in week two. It's probably, I would say, the single most important skill, to creating a really strong interface with the talent that you have available.