Hi. I'm here with Patrick Cullen. He is senior director of Zeus Technology at The Washington Post. I'm here to talk with him about AI, how it helps the news media organisations, and what the journey has been in transforming Washington Post. Patrick, thank you for being here today. It's my great pleasure to have you with us. Let's start by talking about the journey of AI in Washington Post. Where did it begin? What was the journey like of using AI in Washington Post? >> Raj, it's a pleasure to be here. And I think that's a really great question that you ask because it's really has been a journey for us. The journey really began a few years ago when we were using a third-party recommendation system for our site, but we weren't getting the results that we wanted from it. And so we thought about looking for other solutions, but decided instead to bring the technology in-house and try to develop the models ourselves. At first, it was a difficult journey. It wasn't all success. We had a lot of failures, but each failure helped us to learn and grow. And eventually, we started getting success and producing results that were better than the third-party solution that we had replaced. >> That's great to hear about the journey and the path for bringing AI into Washington Post. Let's start talking about some of the innovations that were created with this AI technology. Let's start with AI-powered storytelling and how that is being used in the Washington Post. >> That's a great segue in the journey. So one of the challenges that we always have is how to produce business value from some of these systems. And so one time we were talking with some people in the newsroom. And they mentioned that for reporting on elections and sporting events, that there's a very short time window to try to cover those events. And as a result, they are not able to use much creativity. And so we built a system that we called heliograph. That was an automated content publishing system. And it took in data and produced sentences that were fed into a template on the website. And because it was completely automated, heliograph could report on the results very quickly and very accurately. And that freed up people in the newsroom to actually focus on more creative analysis of the event. And it was a really great example of having humans in the loop with these machine learning systems, and also of using the best of both automation computers and with humans are best at. And so one of the great things about heliograph, besides the results that we got, was the trust that it built with our business and stakeholders. They saw a lot of times people think of machine learning as replacing people. You know the robots are coming for our jobs. But in this case, they realized that just like computers, machine learning or a powerful tool that enhanced their abilities. And so because of that, they're very excited about machine learning systems. There's not much resistance to trying new things with machine learning because they saw for themselves that it could be something that was very productive and not something that was taking away from what they wanted to be doing. >> That's a fascinating story about how machine learning and AI is actually enhancing humans and journalists, and letting journalists actually be better at what they do, investigative journalism and writing good stories. So let's move to content moderation systems, right? So we all know about comments and we're all active on the media. And it's interesting to me to hear about how machines are helping us ensure this content and comment sections are moderated properly. Could you talk a little more about how you use AI for this? >> Yeah, that's a great example, I think, of machine learning enhancing our abilities in the newsroom and as humans. Comments are very important to newsrooms because it's one way of hearing back from readers. And often journalists really want to hear back from those that are consuming what they're writing to understand how to make it better. And so we want to have comments on our site. But the challenge, of course, is the sheer volume. We're talking about millions of comments a month on our site, which isn't possible for a person to moderate. And as a result, you get a lot of comments that are really bad that reduce the quality of the experience on your site and the dialogue that happens. So we wanted to get the best of both worlds here. And we didn't want to give up on user feedback through comments. So we built a machine learning system that was trained on tens or hundreds of thousands of comments that had been moderated by people. So it started to understand what the terms of use required on our site, what types of comments would be allowed, what wouldn't. And then it developed a probability model to understand how likely a comment would have been filtered by a person. And then it assigned this probability to each comment as it came in in real time, so very quickly. And then the newsroom use this to set thresholds where they decided that if it was above a certain probability that they would have the kind of moderation system, automatically moderate the comment, so that that could free up people to work on the comments that were more borderline that needed, like more human intuition to understand. And so again, this is another great example of machine learning enhancing the capabilities of the newsroom rather than replacing, but allowing them to do something they couldn't do without this technology. >> That's awesome. So to me, the story of Zeus Technology is fascinating as well, how the learnings you have here are being used by other newspapers and news media organisations as well. So maybe we can talk a little more about Zeus technology. What was the idea there? And how it's performing now, something that you are leading now? >> Zeus Insights is a suite of products that's part of Zeus Technology Suite. And Zeus Insights was one of those products, which is a topic classification system. And what that means is it assigns topics like, for example, sports, weather, politics to different articles that we publish automatically in real time. And so we used this product for a couple of different things. One of them is for recommendations for our readers. By assigning topics articles, we can then associate what topics readers are interested with and then we can recommend content in the future to them that's related to that topics they are interested in. And this increases engagement on our site that helps readers find more interesting information. And the other thing that we've used it for is also for brands that want to associate their ads and their brand with certain types of content or steered away from certain types of content. And this just gives them more control over how their advertising and brand information shows up on our site. That helps them get their performance as well from their ad campaigns as well as providing a way for readers to see more relevant ads. And we think this is really kind of a win-win situation. And we realized that we weren't the only publishers that had this challenge. And so we built out the Zeus Technology Suite, including Zeus Insights and offered it as a SaaS product, a software as a service solution to other publishers. And now we're rolling that out where we offer the same capability to other publishers, allowing them to the same types of things that we can do internally. And so we think this is also one of the real powers that's kind of AI journey is that you can actually develop technology that's unique in the marketplace and offer that to peers that have similar business challenges. And we've gotten really great reception from the publishers that we've approached with this because it really fills a need that they have because they have a similar kind of challenges that we do.
