But that secret source isn't code. It's not just an algorithm. It's actually this organizational know-how that we've acquired over the years of managing probably more value generating ML systems than any other company in the world. If we're going to share this organizational know-how, why start with technical ML skills? Well, we want you to become great ML strategists and to do that, we believe that you need to get your hands dirty. You actually need to go up and you need to build some of these systems and learn about them and the good news is that these technical ML skills that you're looking for here are mostly software and data handling skills. Skills you may already be very comfortable with. Talking about these technical skills, will also give us an opportunity to leverage Google's experience to help you avoid some of the most common pitfalls. What are some of these common pitfalls? I'm glad you asked. Here is our clickbait hyphen, top 10 pitfalls organizations hits when they first try ML. Here's a list very informally, we've aggregated after several years of talking with new ML practitioners. They come to us and they say, we're so excited about this great new thing, it's going to be awesome, and then they might fall into some common pitfalls. We've seen is with Google and we've seen it with our partners as well. The first one, perhaps one of the most common. You thought training your own ML algorithm would be faster than writing the software. Usually, this isn't the case and the reason is that to make a great ML system beyond just the algorithm, you're going to need lots of things around the algorithm, like a whole software stack to serve to make sure that it's robust and it's scalable and has great uptime and all of this you're going to have to do for software anyway. But then if you tried to use an algorithm, you put in additional complexities around data collection, training and all of that just gets a little bit more complicated. Usually we really push people to start with something simpler in software only. Next one, one of our favorites. You want to do ML, but you haven't collected the data yet. Full stop, you need the data. There's really no use talking about doing great ML, if you haven't collected great data, we don't have access to great data. Let's say you do have the data, you've been logging in for years. It's written on some system that someone in another department controls, but you haven't looked at it, willing to bet that if you haven't looked, that data isn't really ready to use. It goes even beyond that. If there's not someone in your organization who's regularly reviewing that data or generating reports or new insights. In fact, data isn't generating value already. Likely the efforts to maintain it isn't being put in and data has this magical way of going stale. Of all the clients we've ever talked to, we never met one who overestimated the amount of effort it would take to collect clean data. No one has ever said that it was easier than they expected. Expect that to be a lot of pain and friction here. What's then next one? You forgot to put and keep humans in the loop. When we get into these ML systems that start to perform core tasks or core business processes in our organizations, they become really important. Appropriately, organizations become risk averse around these systems because they're the breadwinners of the organization, and then it becomes very important to mitigate this risk. One of the myriad of ways we do that is we keep humans inside the loop, so they're reviewing the data, handling cases the ML didn't handle very well and curating its training inputs. We're going to talk about this more later. But this is a feature of every production ML system we know in google, is that it has humans in the loop. What about this one? You launched a product whose initial value prop, was it's ML algorithm instead of some other feature, so this is a problem because your users probably don't care if what you're giving them is the ML. They just care if it's got that new cool feature or if its recommendations are really good. If you launch something whose initial value prop is just a male, it also needs new data to operate on. It needs a lot of uses to generate that data, so it may learn how to interact better. What about you made a great end ML system? It just happens to optimize for the wrong thing. Imagine if Google search was optimizing for, let's say, a user engagement as measured by how often someone clicks on search results. It sounds good. We want our users to like our product. We want our users to stay engaged. But if we optimize for how often they click, maybe then the ML algorithm will learn to serve bad content because it forces uses to come back, keep clicking. We always want to be careful about optimizing for something that's pretty good, need not be perfect. But we would always want to look out for perverse incentives. So what happens if you forget to measure if your ML algorithm is actually improving things in the real world? You put it out there, you turned it on. It serves users, but you can't tell how much better it is. You can't tell if there's any uplifting customer engagement or lifetime value. That's always really worrisome. Because then how are you going to go back to your boss or your boss's boss and say, hey, I want to do this for another product if you cannot show the impact of the success. Then we've seen a couple of customers with these next ones, you can fuse the ease of use and the value out of somebody else's pre-trained ML algorithm with building your own. So Google Cloud has a couple of what we call ML APIs. For instance, with vision, you can send it an image and it will perform image classification on some predefined labels. Well, that's great, it's super-easy to use. You don't have to worry about any Infrastructure or any training data or any data collection, really easy to use. It's a very different ballgame than if you wanted to start to build your own. Especially if you want to do your own ML algorithm. That doesn't come pre-canned. It's a lot more effort. We thought after we researched that production ML algorithms were trained only once. You like, hey, it's on my laptop. It's doing great on that dataset. I'm basically done. No, you probably about 10 percent of the way through. It turns out that if you're going to have an ML algorithm that's going to be part of your core business processes is going to be retrained many times and you're going to want to invest the effort to make that process very easy and seamless. The final one is actually the only one of these we have that addresses a confusion about the challenge involved in optimizing the ML algorithm. That's you want to design your own in-house perception, ie, image or speech, or NLP classification, that's natural language processing. These are a peculiar pitfall in the sense that they seem like they're much easier than they really are. In fact, all the algorithms we have to address these are very highly tuned from decades of academic research and you should almost always take one off the shelf, already made or already defined instead of trying to do your own research is that it's very expensive. There's a lot of pitfalls, a lot of problems. But what's the good news? The good news is most of the value comes along the way. As you march towards ML, you may not get there, and you'll still greatly improve everything you're working on. If you do get there, ML improves almost everything it touches once you're ready. Think about this. If the process to build and use ML is hard for your company, it's likely hard for the other members of your industry. Once you have that ML enabled product or internal process, it's going to provide the users or the consumers of that process great experiences that become very hard to duplicate or catch up to because of his beautiful feedback loop, where it's collecting more data and learning over time. Let's double-click into this idea that value comes along the way. I know it's tempting to try to jump to a fully machine learned, automated, end-to-end auto magic, everything solution. We wanted to make this leap, but it usually doesn't lead to great products or organizational outcomes. We've seen that in google, and we've seen that in our partner organizations as well. Let's review a more realistic path and all the great things that come along the way.
