>> Great. Today, we are going to talk with Bryan Larish who's the Technical Director of Enterprise Networks at the NSA and Bryan's been doing a lot of work looking at various SDN technologies and applications to government networks. So we're very lucky to get the inside scoop from Bryan today. So thanks, Bryan. >> Yeah, thanks for having me. This is really exciting. As a former student of your course and went through all this stuff, this is kind of fun. >> Cool. So what would you say your job is in managing the NSA's Enterprise networks? Like what kind of problems do you need to deal with on a day-to-day basis and what do you find yourself doing daily? >> Yeah, so that's a great question. So as Technical Director my role is to look at the networks that we have today, look at problem areas, look at what industry and academia are doing. And see where we can apply new kinds of technologies or new approaches to solve those problems and pave the way forward for the future of NSA Enterprise networks. And so as part of that, I get to see a lot of the problems that we have and places where we can improve. So on a day-to-day basis, it's looking at those problems, looking at the technology and figuring out how we can incorporate those kind of things into the network. >> Cool. So, I guess it sounds like you're looking at a bunch of new technology these days. >> Yes, so when we look at today's technology, some specific kind of problems we're trying to tackle right now and I don't think this is unique to the NSA. I think pretty much every large enterprise that I'm aware of has these problems, but managing these large enterprise networks is really, really difficult with the tools we have today and I think that's for a couple of factors. One of them is just the diversity of users that we have inside of our network. And so, I think a lot of enterprises will run thousands of different applications inside of their network. And each one of those applications requires something special from the network, there's not a lot of uniformity. And that in itself is a management problem, but when that kind of thing comes to me, the network guy, I then to implement that special thing the application needs. I have to go touch lots of different boxes in my network one at a time. Put in one off very specialized configurations, setups on those boxes. And so you can imagine, if I have thousands of applications. Let's just say, hundreds of boxes, lots of different boxes. That's a lot of individual configurations. A lot of opportunity to make errors. Makes it hard to troubleshoot, it's very hard to predict what those changes are going to, kind of impact or how they are going to turn out. So this space today, it's hard to deal with that with the existing force. This is just a flat out numbers problem that we have today, so existing technology doesn't really help with that. So we're on lots of new things. For example, with SDN and this notion with centralized control. Maybe now if I had to do lots of one off things, I can go to one centralized point in my network. Make the changes there and have some confidence that I know the impact of those changes and that those changes will be propagated to my entire network. So the thing you're seeing today, the problems we're trying to solve are just really hard to tackle with technology we have today. >> So you think that SDN is going to help you kind of tackle the challenges with heterogeneity a bit better and then diversity? >> So that's a good question. On the application side, I think it's going to be hard for us to do anything there. As an enterprise, we're always going to have huge diversity and heterogeneity in the applications that run on our networks. I think all enterprises are going to be like that. But within the network, I think SDN has a huge promise there to help with that management challenge. Where if I have a centralized point of control, there's just a lot of things I can do that are very difficult when I'm trying to individually put configurations and settings in lots of different places across my network. >> So it sounds like, I mean, a lot of it is reducing the cost of operations for things you're already doing. Is that fair to say or do you think there's sort of new things that you also want to do that are just way too hard right now or is it a combination of the two? >> So, I think it's a combination. Being any good, large enterprise, we have lots of processes and budget planning and those kind of things. And so getting our foot in the door to do new stuff, it's easiest when we can go in and say, hey, we're going to reduce the operational costs. So there's huge value in reducing the operational costs, that let's us get new technology started and moving. But at the end of the day, our long-term goal is new capabilities. If new capabilities weren't possible, I'm not sure we would be looking at that SDN and these kind of approaches as strongly as we are. >> What do you think is like, the single biggest missing piece of functionality that you want to see addressed, that you think SDN can help with? >> Wow. You're asking me the hard ones. >> Yeah. >> [LAUGH] So let me answer that, but maybe not in the way you were thinking. So when I look at enterprises today, the applications and the workloads that we run on our networks are very different than when the internet first started. So if you look at something like the department of defense or NSA, we have time critical applications that have to work otherwise people's lives might be in jeopardy. If you look at a large bank or large financial institution, it's a similar kind of thing where applications have to work have to behave the way they expect otherwise people lose lots and lots of money. And that's totally different than when the internet first came about and it was ad hoc. Let's have people come together and share files and emails. It's important to get there, but there's not that sense of urgency. And so the biggest piece of functionality I want is a network that is built for those kind of mission critical, have to have applications with dependable performance and predictable outages and for that kind of environment, as opposed to let's build a network that just lets everybody come and communicate with sort of a best effort kind of service and whoever wants to come and play can come and play. >> Sort of a really high assurance, high reliability guarantees, as opposed to just a best effort kind of model. >> Yeah or maybe the ability to, in the cases where I need high availability and guarantees, the ability to ensure that with the network. We don't need necessarily need that everywhere, but on the places we need it, we need to be able to enforce it, ensure that it's going to happen and guarantee that level of performance. I see, so it's funny because it's sort of like in a way kind of great, choosing the internet all the way back to switch. It's survivability except it's not in a same sense that it was originally envisioned and it's like a needing to talk to B in spite of arbitrary failures, but A really having good assurance and performance guarantees when its talking to be. >> Absolutely and I think its a great testament to how sort of the designers of the internet and those pioneers developed it is that it you know it has morphed into that initial intent and now people are trying to take and use it in all these different ways I think that's outstanding. We're just trying to figure out how can we do these new ways possibly better with the new technology approach. >> Huh. So I guess like another thing you probably deal with in your role as a technical director is a lot of legacy. Shall we say both in terms of equipments as well as expertise. So for example, I guess you have like a large install base of hardware switches and things like that. Now that you kind of need to figure out what to do with in terms of migration and things like that. And as well as operators, right? Who are likely to say, yay we can do all that now with the equipment that we have. Are you facing those kind of challenges? And how do you sort of deal with both the capital and the human expertise transitions, if you will. >> It's not easy. I'll start on the technical side, because that's got a little bit cleaner story. So the good news is that when you run a large enterprise network, you have lots of hardware that needs to be refreshed just kind of continuously, right? So if you have a three year life cycle on a network switch or a router and you run lots of routers, you're pretty much replacing switches all the time. So, from a technology perspective, we do have a huge legacy install base but we also have great opportunities to inject new technology when we're doing hardware refresh. So that's the good part on that. The other kind of tact we've taken is if you look at any enterprise network, you have out on the edges, branch offices, campus networks, data centers all connected to a wide area backbone that allows them all to communicate. And so the approach we have taken is, start at the edges of our network, and replace, more or less, isolated areas of the network with SDN that can operate sort of independently of the other ones. So maybe go to a single branch office, replace the entire network there. And as long as it can connect to the LAN back home that we have, we're good from a functionality perspective that we can have SDN just in that isolated place. So we've taken that approach of like starting at the edges of our network, replace around and then as we get more comfortable with technology, start looking at how we can replace that core backbone. >> That's cool so like in you have these sort of islands of SDN doing things like branches and that allows you some ability to test the waters. >> Exactly. Exactly. And without having to rip out every single legacy device that we have and right? That's just not a feasible approach. So it's nice because we have these independent places where we can start replacing piece by piece and then hopefully have the critical mass where everything can just be SDN. >> Are you finding that like the operators who run the network are picking up the new technology with some ease? I mean certainly the sort of traditional network operations has not really involved writing programs, so there's a bit of a learning curve there I imagine. Yeah, so this actually has been a really challenging area for us. Because a lot of times, when the existing network operators hear about SDN, it's said in a way that they immediately think their job is going away. And if you think a certain technology is going to make your job go away you're not necessarily going to be very receptive to learning it and figuring out how to use it. And so, internally, we've had a pretty difficult challenge getting the existing folks up to speed and participating in our SDN efforts. Which is a shame, because the thing we try to tell them is your jobs not going away, the skills you have aren't irrelevant, but we want you to do new things that make better use of your time, better use of your skills. So for example, I want somebody who understand how routing protocols work. Right? And so how OSPF works or Dexter algorithm works. But I don't necessarily want somebody who knows how to configure that on a Cisco or Juniper router. I want somebody who has those network fundamentals but can help us figure out how to implement that in a different way. So that's what we've been trying to tell them. On the flip side, we've actually been going to folks who would not traditionally think about networking as a career field or a career path. And telling them, look, here are all these great new opportunities, right? So if you're a software developer, we're trying to implement new things in software. Here's an opportunity to do new and exciting stuff that nobody else out there is doing. And you're just going to be applying that in this specific network technology area. That has been the most successful approach we've taken. Because kids just coming out of college, or who just graduated and started that [INAUDIBLE] say they wanted you to do exciting, and interesting stuff. And so SDN has been the perfect carrot so to speak. To say hey, [LAUGH] here's something really cool that you can come work on, that you probably never would've thought of. >> That's a good resume lined out for them too so. >> Yes. So then we have the whole challenge of, once we have these SDN experts, how do we keep them around? But, that's a different problem. [LAUGH] >> Right, right, right. But I like the point you made about sort of, the domain expertise is kind of indispensable. Right? I mean you kind of need both, right? You can't just take a software developer and have them just write your new network control application, they need to know OSPF or GGPD or many other networking fundamentals as well. So it's a matter of kind of adaptation or at least for today's operators. Right? >> Yes. So you can't do it in a vacuum. And there's so many lessons learned from this entire history of the internet, right? And if you don't have that domain expertise you're going to relive all those lessons learned in a very painful way. Naturally this was actually a funny experience we had yesterday. So we have a insulated place in our network where we're implementing SDN. It's mainly for our SDN team, providing the day to day connectivity. So luckily there was no business or mission impact, but the folks didn't quite understand all the impacts of Spanning Tree, and they turned on an SDN application and within a couple of hours the entire network fell over because they weren't handling Spanning Tree appropriately. That was the perfect example because if we had the engineer there who understood what we were trying to do with the STN, they could have said, whoa, time out guys, this is going to be not good. >> Right, right, right. From that perspective, it seems like with this sort of centralization of policy, there's definitely promise there but also, do you think you're going to need more software developers or more people? I mean, in some sense are you taking money that would have been, shall we say, mark up on certain vendor hardware and throwing that at software engineers or how do you sort of reason about that? Do you think you're going to, do you sort of iust creating a bigger software engineering enterprise, or how do you view that? >> Yeah, so that's another really tough problem, so we have the challenge that when we tell our leadership about these potential cost savings, they look at the cost of legacy hardware and go, oh, well, that's going to reduce by 50% >> I can just cut your budget by 50% >> [LAUGH] >> Not buying that anymore right and so we have to go well guys time out right you can cut that part of the budget by 50%, but we're going to need to augment that or replace that with software developers because we're now implementing the capability there. It's definitely not a cut all the hardware budget you have and use that money elsewhere. But, what we don't know is how much of that software talent we need to have. So for every dollar of hardware reduction, is that a dollar of increase in software developer? I hope not. But what that trade off is, right, we don't know. Is it for every $10 I lose on the hardware side, it's $1 increase on the software developer side? >> And hopefully over time, that exchange rate will change, right, because as the ecosystem gets more mature, there'll be more tools and hopefully more software platforms out there that make some of these things a bit easier. >> Absolutely, right? So the promise of doing a lot of this open source the same way server guys manage servers. I think is something where really hopes come to fruition because that's, I think, one key way to get that exchange at a more favorable level. But even just expanding the market place in general I think will help. All right, so I know a lot of people have talked about the way you have an app store for your iPhone or your Andriod phone, I want the same thing for my SDN controller. That's absolutely what we want, because I think that's really where you're going to see that exchange take a more favorable turn. >> Mm-hm. So, what do you think about, in terms of just the whole ecosystem as a it evolves? And even this compared to a year ago, there were a bunch of technologies that were arguably there, but much more mature now than they were a year ago. Like Docker and Openstack, and even controllers like RIO seem to be quite a bit further along. How do you sort of keep pace with the sort of rapid changes? On one hand, its good right? Because you have better tools to work with, but also maybe there is some integration challenges that you face. >> Yeah. To be honest, it's hard to keep up with all the things that are going on. There are just so many moving parts. It's like every day there's something new coming out that people are asking, oh did you see this and did you see that? And so, to be blunt, I don't think we have a good answer at this point. The one thing we have tried to do is layout a set of principles that we're trying to follow. So things like, any new technology we do is going to simplify our network. Or, we're going to reduce hardware cost or operational cost. We have these set of objectives. And so anytime we get something new, we're evaluating those against the objectives so we can make a relatively quick decision on whether we need to explore it further or just kind of clear it off to the side. So that's at a first pass what we're doing. The integration question is a great one. And that's actually something that internally we're having to debate right now is what kind of role do we want to have there? Do we, the NSA, want to get a bunch of open source or proprietary components and integrate them ourselves? Or do we want to look to a third party vendor to do that integration on our behalf? And again, I don't have an answer for you. But that's some of the thought process that we're going through right now. >> In terms of costs, savings, and things like that, are there any particular functionalities or application areas that definitely look like a clear win? I mean for example, containerization seems to be pretty talked about these days. Docker, and Openstack, and some of these things that. Do you see cost savings in your network as a result of things like that? >> I think, potentially. So, for me, something like Docker or Virtualization. The reason that would, in my opinion, where the cost benefit would be, is it lowers the barrier to entry for new members, right? So instead of a new vendor having to build a hardware platform from the ground up, and then layer their network, their firewall software, or their IDS or IPS software on top, they can go right to the software layer. I'm hoping that lower barrier to entry will increase the amount of innovation, the amount of companies in the marketplace, and that competition is what will drive the new capabilities we want and drive the cost down. >> And that's certainly good for you, right? Like as you've said, it would certainly drive down the cost. Right. And also the switching costs, right, might actually be an important thing, too, right. Because if all goes well you could presumably swap out Docker container firewall A for vendor B without too much trouble. At least you wouldn't have to be swapping out hardware. >> Absolutely. So that swapping out hardware is a huge problem for us. It is incredibly difficult to get new hardware deployed in our enterprise. And so, I think to your original question, the other thing we're trying to do is commoditize every single bit of hardware we can. >> Because it is just too timely, takes too much time, too costly to go replace hardware every time you need a new capability. I mean there's like the blanket statement that the IT infrastructure is too expensive and too slow. And a large part of that is because of the hardware. So we're hoping that commoditize every single bit of hardware we can, encourage the innovation of the new capabilities in the software level, and then that allows us to be more flexible, more agile when we want to do something new or we have to do something new. >> In terms of just operational kind of tasks and things like that too, standard things like troubleshooting and access control and things like that come to mind. I mean, at the beginning you spoke broadly about, sort of, the advantages of sort of centralizing management. Certainly, at least in my own experience, looking at operational config, access control is just a huge beast in terms of both just the number of lines to config it requires and the amount of changes that are happening all the time. And Just the sort of overall complexity there. That seems like kind of a beast. I'm wondering if that one's a particular problem for you. If you see sort of big wins in SDN there. Or if there are other things that are bigger. >> So I think, for our, for us. And from the other enterprises I talked to. I think those two particular areas will be huge wins. Because at least, the way in talking to folks, there's access controllers everywhere in the network. And if somebody has a new device for a new subnet that they need to put in the branch office or the campus. It takes a network engineer going in and modifying ACL's and making sure subnet masks are wind up on lots and lots and lots of different devices. And it's sort of like the stars have to align just perfectly for that new connectivity to take place. And inevitably after the first pass of changes, they don't align perfectly. And then you have the application owners that were deploying the new system. Blaming the network people and there's a lot of finger pointing because it's really hard to figure out where the problem occurred. And, so, at least in our network operators day to day business, I think those, the troubleshooting and the ACL configuration would be huge one's for us. The troubleshooting goes beyond ACL's because every day we get outage reports on our network. And it's amazing to me how you see sort of outage happened here and then we get periodic updates and many times it's six or seven iterations of people trouble shooting this or that just to find where the problem is. Let alone how to fix it. And, if you're having three or four people look in six or seven different places to try to just find the problem, that's bad news. There's gotta be a better way. >> Yeah, it seems like there's huge problems there. I don't know what the solution is. [LAUGH] >> Maybe some of your students in the class could take that one on. >> I think so. So yeah. I guess that would be my last question, which would be given that you did a PhD yourself and lots of students are likely watching and hearing all kinds of great problems to work on in the NSA's enterprise network. What kinds of things do you think people. What would really impress you? Like, it a graduating student came to you looking for a job. What could they say that they've worked on? Or what problem would you like to see solved that would just really impress you? >> So let me, I'll give you just a general statement at first. And then we'll see if I can get creative, and think of something specific. Especially for grad students, and especially for people who are trying to build their resume or find a new job. Not necessarily people who are looking at an existing company trying to figure out how to modify their product to make more money. Because I think those are two separate issues. For the people who want to do something impressive on their resume. Take a fundamental look at how networking is done today and try to find a way to do it better. So instead of putting a band aid or somehow tweaking today's solutions. Look at how to do it better. So don't give me a graphical user interface that lets me configure OSPF on a bunch of boxes from that one interface. Write me an SDM control application that says I don't have to use OSPF anywhere on my network. That's the kind of thing, at least, we're trying to do is, is instead of mask the complexity in today's networks, or make the existing way of doing business easier. We're trying to see are there ways that we can fundamentally re-engineer our network and just take the complexity out and get rid of it. So that we're addressing the fundamental issue instead of piling layers of abstraction on top of it. >> Mm-hm. So basically find something that is complex or hard. Some problem that needs to be solved. And then think about how to refact or rearchitect things such that you can take that complexity away. Simplify. And remove complexity from the network. >> Yes, that's exactly the kind of thing we're trying to challenge the people we have. If somebody brought me a resume that said here's a specific example of them doing that, I would offer them a job on the spot if I was allowed to. [LAUGH] >> Yeah, it's definitely a good thing to work on it. It seems also almost to the way networking has been done. Where it's just check box, after check box of feature. So it's, even just working on simplifying is kind of a fundamentally different way of thinking about it so. >> Yes. Yup. [LAUGH] >> Cool. Well thanks for your time today. And we'll check back in a few years and see what's going on in your network. >> Okay, well thanks for inviting me, this was great. Hopefully your students can start solving some of these problems, and in a couple years we'll be light years ahead of where we are now. >> Cool. Thanks a lot. >> Thanks Nick.
