Hi, I'm Martha. I'm an assistant professor at the University of Alberta. Hi, I'm Adam. I'm a research professor at the University of Alberta and a senior researcher at deepmind. Adam is well known for his work on predictive knowledge for reinforcement learning some other research groups around the world are building on these ideas. I personally think it is some of the most important work in recent years and will only become more influential. He's also an impressive empiricist a real master designing experiments to cut to the key issues. He's constantly trying to improve scientific understanding in the RL community. Martha's algorithm contributions to reinforce and learning are too many to list here. She has developed several new off policy learning algorithms new approaches to policy gradient and dozens of impressive contributions to representation learning. She is truly one of the young rising stars and reinforce learning. She also happens to be my most frequent co-author and my partner in life. So how did you get into computer science? I grew up in the east coast and high school. I was really interested in the Sciences physics chemistry biology, but I couldn't really find the right fit. So I kind of randomly took a course on computer science and programming actually in gw-basic and it was really just a career-defining moment for me because I just remember the first time, you know, implementing an if statement or for Loop and it was just amazing to me that I could get a computer to execute a sequence of instructions for me. And so from there like there was no hope I was always going to do computer science in college. Okay, and then from there, you know, why did you start working on reinforcement learning? So for Graduate Studies I moved out to Alberta and I came out here to do parallel distributed computing that that's what I kind of focused on in my undergrad. So I would had all my courses loaded up with you know systems courses and I have one more elective to fill so I was looking around for course to take and I did a very reasonable strategy. I looked at the perfect pictures of the professors and I picked a friendliest looking Professor I could find and yeah that turned out to be rich Sutton and it was introduction to reinforce learning and that course was amazing. It was really inspiring. It was my first introduction to artificial intelligence or machine learning of any kind. And so I remember one of the early assignments in that course, we were implementing sarsa to do a grid world. And again, it was another career defining moment for me. Just watching this agent learn to solve this Maze and all they got was this reward signal - one per step and somehow it figured out this amazingly efficient path out of the Maze. And again, it just amazed me and I was hooked and I was going Doing that for the rest of my life. So Martha, how did you end up doing research in machine learning and artificial intelligence? Well, I actually started my undergrad in math. I've always thought functions and dynamical systems are beautiful. And then I just took a CS course for fun and it really was very fun. So I just continued in both math and Cs and those two topics go really well together for AI and machine learning actually. So and then I did some research in AI in the summertime and just continued from there and I really just stayed in the field because over time I realized it was more than just about functions in math. It's really a community where you get to collaborate a lot and you get to make a career where you help people so my day I get to Mentor a lot and I get to work at a big team working on hard problems and it's really just very amazing to get to work on such interesting questions in a team. There's probably some people who are taking this RL course because they want to do Graduate Studies in RL. So what's one of the most important things you learned in your Graduate Studies? Well, maybe surprisingly enough. I learned a lesson that's useful across probably all the sciences. And in fact just any form of research and that's to really love coming to understand the details of things to focus and do small problems and understand the really really well and that that's something that I think is really applicable across a wide range of things and it's something that I pass on in my students in my classes and I really focus on teaching my graduate students that and it's something that I hope to share to the people taking the specialization. So what do you like about your job? Well, I love doing research. But sometimes that can kind of grind you down a little bit. Like it's really long hours. The peer review process can be a little frustrating and so in those times when you kind of step back and look at your research, it can seem kind of small and a little bit insignificant and it's hard to see where that's going how that's going to impact the world and make it a better place. And so in those times you can always fall back and think about teaching and mentorship right teaching is where you get to help people learn about things for the first time to come to understand how your field works and it's just so exciting and you're also being part of the, you know, the beginning of their career is whether they end up in Academia or industry or starting their own companies. It's just such an amazing experience that be part of that process and you know throat my week. My the best day for me is always the days when my schedules full of meetings with my students and I get to help them work on their problems and kind understand reinforcement learning better. So why do you think it's so important to work on reinforcement learning wasn't the promise of RL is really big but there are still a lot of open challenges machine learning has started being used much more widely. But our else kind of only been used in a few areas. But at the same time I think Carl can have a really big impact and I think within the next decade or so, we're going to see more control Engineers actually using RL in their systems. We're really just going to see that it's a tool that can help in automated decision making really within a larger engineered systems, but I think for that to really happen we need to make some advances in RL. We need to start thinking more about how we're going to have robust algorithms and how we're going to get people to be comfortable using these algorithms. So I think alongside adoption in Industry. We're going to need to be thinking about how we can improve our algorithms so that these two things can happen together. So, how about you? Why do you think RL is good to work on? Well, naturally, I think reinforce learning is just the exact right framework to study Ai and to make progress towards solving AI but RL is also really important to me because I'm very interested in developmental Learning System. So you can think about you know humans, you know babies playing on the floor or young animals learning about the world. I want to learn about those systems and how they work and if you think about those systems, they learn an enormous amount in such a tiny amount of time, right? They learn how to crawl. They learn how their bodies work. They learn how to interact with their environment manipulate objects communicate with other agents in the world and there's just generally they're very competent learning systems and they do that over like a couple weeks to a couple months and it's just really astounding and it feels like we have so much work to do to get close to those kind of systems. And so I really want to come to understand the algorithms and the representations that those systems might be using. And I think reinforce alarming is the right lens to look at those things and to come to understand those things. So applications always kind of Drive interest and excitement in the field and we've had a few of them in games and some industrial control problems. But what do you think is going to be the next big thing in reinforcement learning? Well as hard I think RL is just general approached automated decision making it's a tool that's going to help us in a lot of our engineered systems. And so I think the place we're really going to see it take off as an industrial control and Industrial control. We have experts that are really looking for ways to improve the optimal the how well their systems work. And so we're going to see you do things like reduce energy costs or save on other types of costs that we have in these industrial control systems and in the hands of experts we can really make these algorithms work well in the near future, so I really see it as a tool that's going to facilitate experts in their work rather than say doing something like replacing people are automating them away.
