The other thing that what feels natural to me to talk about now is that when you think about these open domain conversational systems, like the Alexa prize that my students are working on. Yes, how was that? So you worked on this Alexa prize Was that [inaudible] this? We're doing it again. For this next year, they picked seven teams, a smaller number of teams, and they're giving each team more money, and we've also got into the competition for the second year. So two of the same team members are going to stay on, and then we have three new team members, and we're doing it again. How exciting. What's the task? What is the competition? The competition is that you could carry on a conversation about any topic for 20 minutes. With Alexa? With Alexa. What's interesting about that task, as compared to like the Book a Flight task, is that you can't write a call flow for it. That is right. So you don't know the document. You don't have a form behind it or- You don't have a form behind it, and so you have to understand a lot more about how conversation actually works. What you do need, what I believe you need, and I keep emphasizing to my students, is what you need is a system that can take the initiative. So you need a system. So if you think about like Google search as a paradigm for conversation, that's all user initiative. The assumption is that the user has an information need and they keep the dialogue going by. They keep asking questions and the system provides the answers. Imagine you had a friend, so they could answer questions, they could set a timer, or they could do a few little things like that. Well, they're not a very interesting conversationalist. There are people like this. It's just nobody spend 20 minutes on them. You wouldn't want to spend 20 minutes. You wouldn't spend 20 minutes talking to them. So I think the formulation of the Alexa prize is good. It's the original Turing test formulation. The formulation is good, because it moves people away from thinking about these tools focus, or this search focus, and to actually think harder about how conversation works. So to return to the thing about the mixed initiative, the system initiative, I think the system has to have things that it wants to talk about. It has to be able to- It has to be an interesting conversationalist. Yes. It has to come up with topics it wants to talk about. It has to say, "Have you seen The Crown on Netflix?" But it also has to know me a little bit. I think that's a really good point. Right. It has to pick something up, because if it comes with- I think it has to start, yeah. Yes. It has to start knowing things that you're interested in. Because it doesn't see my face. When I meet somebody new and it's like, okay. I can flexibly switch the topics but Alexa- Well, they might start it. It's scary, actually, now the see your face thing. So the newest iPhone has a thing where they're doing facial tracking. Oh, wow. Okay. Developers have access to the information from that facial tracking. So if they ask a question, I raise an eyebrow, they quickly switch the topic. Well, that would be the idea. But the claim would be is that if they can get enough data, so people play with these- there's games there and things where they ask people to make faces, or look happier, look sad. That's how they get the Big Data. So the thing is if everybody plays along with that, in five years, Apple might have a part of their interface where they say, he's happy now, he's sad now, he's puzzled, he's curious. Then they could adjust the dialogue to that, and there's a question that you can ask- I find it a little bit spooky. But somebody, the other night, was actually telling me, they already have a rudimentary version of that. You can imagine that Watson would maybe already have a rudimentary version like that. IBM Watson in that cognitive. Component. Cognitive computing component. They have stuff in there that recognize personality. They might, very quickly, have stuff in there that are very soothing, they might have stuff in there to actually recognize some things about somebody's emotion as well. Right, and then there's Data fusion. I mean, if I know the personality from what they're saying and I combine it with a face, I can do the data fusion in that. I can get more fine-grained information. So coming back to something very interesting you said, the Turing test. So the Turing test. Turing founding father of computer science in the 1930s it was, right? He wrote this paper that basically outlined what we now understand as a computer. So this is also known as a Turing machine, a universal computer. There's this one, the Turing test, the famous test was, if you can speak with something behind a curtain, and you don't know if it's a person or if it's a machine. If you couldn't distinguish if it's a person or a machine, this machine would have passed the Turing test because it would be an artificial intelligence that gets up to being indistinguishable between us and the machines. So if you now and the Alexa prize, would you say you've already passed the Turing Test? Is that something you already passed? Well, I don't think anybody would confuse any of last year's competitors with a human. Oh, really? No? No. How would you find out and when people ask, what are the most difficult things? Sarcasm, irony, and sometimes things. We, sometimes, don't think up on it, if somebody is sarcastic or ironic. Yes. So recognizing those kinds of things, I think, is still very challenging. We've done work on sarcasm recognition from social media. So where we have thought thousands of examples, that's one of the things that we do. A lot of times we collect annotations for sarcasm. So if you try and then do something like that, crowd sourcing, if you're trying to crowd source annotations for sarcasm, and you're trying to get reliable annotations, what we resorted to was we had nine crowd workers annotate each utterance, because there's a lot of disagreement on whether something's sarcastic or not. That is right. Yes. So we do have a pretty good sarcasm recognizer, but we only use the data where at least six, seven, eight, or nine people said that they thought it was sarcastic. That sounds interesting because, often, we say like, oh, the machine only recognizes three out of six. But if you ask 10 different people, also only half out of them, three out of six people might also agree that it's sarcasm. So it's also we don't even agree on it. We don't agree on it. How could we teach a machine? Actually, sarcasm is a really interesting case, because what I started thinking is that one of the affordances of sarcasm is that you can be off record. So you can deliberately be ambiguous so that you can play with your audience where they don't know whether you're being sarcastic or not. Like if I say, "Oh, this is great." Right now, right? Right. You don't know what really like- How would the machine know? Okay. So you say the Turing test, no, we are not there yet. No, we're not there. The team that won last year was the University of Washington team which is led by Mari Ostendorf who's a long time speech person, and that team won. Some of the things that they did I think were really interesting in terms of the system taking the initiative. So they took data for example, from Reddit, from this thread, this sub-reddit called change my view, where people express opinions, and they recycled a lot of those opinion utterances as Alexa's utterances. So she would express opinions about various topics and they took those, and that was one thing that was considered interesting and novel about that. One of the things that we did that people really liked, although we're not allowed to do it again this year, is we had games in there. We had games in there. So there's all this kind of stuff on Facebook, for example, where you can do a little quizzes and games, where you can say, am I spicy or sweet, and it asks you a bunch of things, and then it says, "Oh, you're spicy."So we put a bunch of that stuff in there just to make it more fun. Actually, on the user evaluations, we've got high scores for those conversations where people played these different games that we had. But this year, and the prize, they've said, you're not allowed to have any of those kinds of fillers, things that people get engaged in that are not really truly conversational. So they've made the prize harder. So they've zeroed in on a smaller number of teams, they've made the prize harder. It's quite challenging to think about. You really have to think about, if you say something like, "I watched The Crown on Netflix and I'm not really into the royal family." Then for me, as the system, to think okay, what's a good follow-on from that? What kinds of things can I say there and how would I continue that topic or change the topic or do something else? It's really, really different than thinking of dialogue is being searched, or thinking of it as being a structured call flow like set a timer. I've heard of it. I've mentioned this. I think that's also why they called you. [inaudible] We will see in a few years we will have it in our pocket before we notice any signs of that.
