Hello everyone. We are very, very lucky this week to be chatting with Amin Vahdat. Amin is the tech lead for networking at Google. He's currently on leave from the faculty at US San Diego and he's done a lot of very interesting work, both on STN and wide area networking at Google as well as some earlier work on project called Portland which, which we looked at. In which one might view as sort of a, an early days STN for, for data center network architecture. So thanks for much Amin for spending time with us today. >> Yeah, my pleasure. I'm really excited to be here. >> Cool. So I actually wanted to sort of break, break our chat in to two parts. Like one about, about your, your, about your B4 work with your STN work, and then the second part about the data center networking work. Definitely some of the, the highest visibility work you've had at least externally at Google, has been the work on the B4 system at Google which, which also is, is going to have a sitcom made this year. I wanted to ask you a few things about that. As I was reading through it, I actually found the insight that you could drive your network to 100% utilization, really interesting. It's just totally different than anything I had seen before, in terms of you know, just how to run a network where you know, as you acknowledge to like typically operators want to keep the utilization at 30 to 40%. I was wondering, you know, it seems like the, you've gotta consider a whole number of different, you know, new problems such as you know, reliability and dealing with congestion during, you know, overload or failure. that, that, that might occur if you're, if you're running your network, like, almost, almost to the, you know, to the pegs. And I was, I was wondering do these problems that you know, that operators have to deal with in terms of congestion, failures, etc. Do they become more difficult as as the network utilization gets driven towards a hundred percent? Or do you think that, just, just with the B4 architecture in place that you've got ways of dealing with it that make things easier? >> Yeah I think that's a great question. I mean, I think there's two sides to this that we were going after. One is with traditional decentralized, sort of traffic engineering and provisioning models. You can't tell how the network is going to converge after a particular failure. It just depends on the particular interweaving of events, et cetera. So you sort of have to, especially when your giving SLA's and guarantees to your customers. You have to plan for the worst case. And for you know, distributed system, the worst case could be, substantially different than what a centralized system can do. So, one side of it is that actually since we have a centralized traffic engineering and traffic monitoring infrastructure. We can do much better and basically be much more predictable in our response for failure which allows us, on the one hand, to push up the level of utilization. So that's one side of it. But still going all the way to near 100% is scary, as, as you point out. The basic way that we deal with that, is, we have hooks all the way to the edge. In other words, a key aspect of our architecture we got into it a little bit in our sitcom paper, our upcoming sitcom paper. There is more to the story that we're going to be writing about in the future. But basically, we can differentiate traffic all the way to the edge. And preferentially weight limit the less important traffic ramped up to the more important traffic. So, we are explicitly targeting a network where we have essentially sources with infinite demand. They'll take as much bandwidth as they can get in pushing data from one place to another. They don't necessarily have a strict deadline. And what they're willing to do is, rather than have hard SLAs for worst case performance, over a week or a month or a year, they're more than happy to trade 2X, 3X average bandwidth for precipitous drops in bandwidth for an hour or two. >> I see. So it's, like, this will work really well for specific types of applications. >> And the and it turns out that's the bulk of our traffic, actually, is pushing huge amounts of data from one place to another. >> Makes sense. >> And there is some other traffic, which is actual actually time critical. It might be on the path of a user request. Which needs those hard SLA's. The point here is that our traffic and engineering system in the face of failure can, one, ensure that appropriate mixes are traffic are in any particular link, high priority and low priority, such that a failure does occur, we can do the right thing with the low priority traffic. Right. And so that's one side of it. And the other side of it is, should a failure occur you're going to lose bandwidth. But we're going to make sure that only the quote unquote right applications lose bandwidth. >> That makes sense, yeah. I think if you're sort of prepared to drop the right stuff in the event of failure then that, that makes sense. >> And to be clear, we're not dropping so much as appropriately restricting the bandwidth available. So it's not like all of a sudden we going to do 50% traffic loss just because the link fails. But it means maybe there, there would be a burst of loss. But we'll detect that and then put in the appropriate mechanisms to basically say well, okay, you have less bandwidth now. Stop transmitting at X megabits or Y gigabits per second. Slow down. >> That's, yeah that's really interesting I mean it seems like one of things you were able to take advantage of with the, with, with before is that not all the traffic is kind of one size fits all or you know >> Exactly. >> Be treated in the same way and, and and by sort of appropriately provisioning or, or accounting for the failures but then re, like basically releasing what was going to happen and the events of spe, specific kind of failure. We can then say, okay, we're not going to treat this as sort of normal, like the way normal networking protocols would, we, we have different ways of doing that. >> Not all bits are created equally and you have to pay a lot, and add a lot of complexity if you're going to treat all bits as if they were created equally. >> Yeah. Another thing I found really interesting in, in, in the work was was this idea of how having a baseline operating mode that was, that was overwritten by more sophisticated traffic engineering features. Actually that was something that, it came up like when we were, when we were working on RCP and stuff, too. People were like, what happens when the controller fails. And we were like, well. Certain, certainly you'd like to sort of fall back to regular PGP or something like that if, if, if a failure occurs. But it seems like you've really like, worked out a lot of the details there and, and I thought that was really cool. I, I was wondering like, what are the, what are some of the more common cases where you end up using these override facilities? Like you've got the default behavior and you want to basically. You want to inst, instantiate or impose some more specific behavior in that overlay. I guess the, some of the TE stuff you mentioned a lot. Are there specific types of applications or destinations or things where you find yourself needing to do that override more than others? Well so I think there's again, a good question with lots of different facets to it One is this sort of concern about introducing new functionality and new protocols into a large scale network that people care a lot about. You know you've got these questions with your very forward looking work on rcp. We had these same concerns in introducing this ST N concept and this traffic engineering concept into our data center interconnect. I you know, I want to point out that one of the main reasons that we were able to be successful with this work is that we have an operations team at Google that is willing to take risks. And sort of bet on new technologies. But we're also running. Mission critical traffic on our network, so we have to have a failsafe if you will. I think the baseline B.G.P. and I.S.I.S. was basically there to say, if we have these S.T.N. mechanisms, can we make it look like it is exactly like the network was previously? Basically pulling the protocols out of the routers and the switches, and putting them onto servers. >> So your worst case basically looks no worse than, than today's network [CROSSTALK]. >> Well, that's true, but even, even more than that, before we go debugging our traffic engineering system, and we want to establish baseline confidence in STN. And STN let's say I've got eight sites, what I want to do is I want to just quote unquote upgrade one of them to STN. The other seven are slow running bgm and isis. So from the perimeter of this one site, that's been upgraded, I want to export bgpn isis. Oh I see, yeah. Yeah. >> And to the whole rest of the world now it looks like the network that it did before and in some sense, I mean this is an interesting journey because you take a step back. You do all this work, huge amount of work to create an MSTN infrastructure and all you've done is be compatible with the legacy infrastructure. That is actually really interesting though, it seems like, yeah, you are spending a ton of ton of work just to be backwards compatible, before you can kind of take a leap forward. >> But that also allows us, and the critical thing here is that it allows us to make sure that our baseline or liabilities is as good as the existing network. Right. >> With STN had one side without putting sort of the mission critical applications on the line. [CROSSTALK]. >> And this was actually key to our success. >> Interesting. It seems like another place where that might show up. I was going to ask you a question on it later, too, about you know, if you've got wide area networking being deployed at exchange points or carrying points. You would also have to think about, sort of speaking B.D.P. or being backwards compatible. Being backwards compatible with neighbors in that kind of scenario as well. I guess maybe I asked that question now. Supposing you have the B4, mostly focuses on deploying WAN protocols inside the Google's background network, but doesn't really. Talk about for instance deploying interdomain routing STN based interdomain routing at peering points. And I, I was wondering, you know, supposing you had that. In the, in the course you talked a little bit about, you know, putting software client networking at an exchange point. Let's suppose you had that. Are there things that. You know you want to do with that if you had it right away is that, is that something that seems interesting or? >> Oh I think, I think it's a huge opportunity and something that I'm personally very excited about let me first finish answering your last question and that is, we don't actually want to actually run BGP and ISIS essentially ever. Is what runs almost all the time. But the point here is that if there is a failure of the traffic engineering infrastructure. And that should be rare. In the paper we talk about one instance where a pretty scary failure did take place. We can then fall back to the baseline writing infrastructure. Over time though I think one thing to note is you can, you can demonstrate 99.99 or 99.999% availability only through deployment experience over a very long period of time. In other words, I can't bring my system up and say hey, it's 5 9s availability, right? I, I, I can't say my t server. Can be just as good as BGP, and ISIS. Literally, it takes years, if you're going to do it from statistical perspective, to say. I no longer need the base line. Because I've demonstrated, that my new thing can be as good as the base line. So from our perspective. we, we want to be running the advanced functionality. We are running the advanced functionality. Essentially all the time, if there's a failure, we fall back. Okay, so sorry, going back to your second really interesting point, I'm very excited about introducing S-T-N functionality into peering points and exchanges I actually think, and maybe this is a subject of a different interview, that B-G-P was built for a particular set of deployments... Among mutually distrustful sort of ISPs. We're now in a world where a lot of the pairing isn't among mutually distrustful ISPs. It's sort of among partners, that are cooperating to deliver applications to end users. >> Mm. >> It's not just ISP to ISP as much in many cases. It's sort of big provider like Google or Facebook or Amazon or whoever, to an ISP. In this world, you actually might want to exchange a lot more information. You might want to be selectively much more trusting of your peers then what BGP would do by default, BGP is about connectivity, it's about sort of best path slash shortest path for all traffic from one peer to another. And you can play all kinds of ugly tricks, I would call them, you're, you're as familiar with them as I am, to sort of fool BGP. Into doing what you think is the right thing for the global internet, but they are the in the in hacks. If you had an STN infrastructure, for peering you'd start with BGP, I mean, you'd be crazy not to, with reliability, and baseline functionality, but then you'd probably augment it with a some purpose build protocols that actually expose a lot more information about. Maybe not your fault typology, but you know, perhaps a prior prefix basis, maybe a slash 24 or a slash 28, you know, here is where you want to give me traffic, or for this type of traffic, here is where I have a lot of bandwidth, but the latency might be longer, et cetera, et cetera. The kind of stuff that BGP, isn't at all good at and it wasn't designed to do. >> Like finer grained control over, over traffic flows, or decisions based on more, more like a more sophisticated set of attributes, and, and >> Exactly, and an extensible sets. That's might be decided upon even on a, a sort of peer peer-peer basis. >> Interesting, yeah. yeah, no that seems like there's just a whole lot one could do once, once you sort of free yourself from AS pathfree pending and local reference and. >> Exactly. >> E-books reading so so yeah. I guess another, another question I had actually going back a little bit to the point that you had about demonstrating 99 or five nines of reliability if you will. One of the things I found really cool about the P4 works and he point that, that, you made in the, in the paper and then public talks, is about testability. Yes. >> This I thought was awesome, because I made the point in the class too, right? You can write a S.C.N. controller, or what have you, and that is the real binary that you might run in a production network, in a real network, even though you are running an emulator on your laptop. Of course you take that software and drop it down elsewhere. Seems like you could take huge advantage of that in, in, in your test setup as well. I was wondering if if you had, if you wanted to say anything about that. >> Yeah, this again also has been really important to our success is that we can essentially take the entire software stack, and run it on a few servers in emulation, and subject it to a range of failure scenarios. re, recreating the entire WAN topology if you will again, on a small number of servers, the nice thing is that the amount of bandwidth and computation power available in our cluster it substantially exceeds what you might have in, in the WAN, even though the, our WAN is quite large, clusters of course outstrip it. So the sort of predictability, the failure behavior, et cetera. The hardware, the servers are the same, we have been able to really push our confidence in, basically our software quality, by having the simulation environments, and we have our whole test suites, basically every bug we discover, results in a new unit test. The earlier conditions, we have real hardware in our, in our test labs. So we're really pushing on automation such that, we can essentially push new, images, to our controllers and even to our switches on a pretty fine grained basis. Rolling out upgrades, new functionality very, very quickly. We have some of that data in our paper. But we really are targeting certainly on the server side new functionality even as fast as every few weeks rolled out with a high degree of confidence in its stability. >> That's very cool. I guess another, another thing that occurred to me as I was looking at, at, at what you described there was just ability to do. Forecasting and predictions, so in, in addition to testing and features, I was kind of imaging like, well there's certain types of traffic patterns that are, that are some what regular, I mean there are batch jobs, there are diurnal cycles, and you can kind of sort of predict certain things about traffic loads along along certain paths. And one of the things that, that that today's aren't very good at doing, right, is sort of adapting to those kinds of changes. >> Right. >> But in this case, it just seems like a huge opportunity, like if you know that a spike is coming at a certain time or if you know that a batch job is going to be run, you could use. The STN controls to take advantage of that in your, in your TE. And I, I guess I was, I was wondering. I mean, it seems like you, it, it seems like a logical thing to do. But the paper doesn't talk too much about, about forecasting or, or taking advantage of predictability. I was, I was, I was wondering if, if you had you know, if your. If you're taking advantage of the some, those regular patterns that. >> Forecasting is a very important area and one that we are using our framework for because we do have this nice measurements infrastructure that we're running continuous optimization algorithms over. So at the rate that our WAN and really the global internet is growing, one of the key problems that everyone faces in the internet is. How to build out the network to keep up with the demand. So we, we do have a fair amount of software invested in this. And actually thi this will also be something that we talk about more. In whatever the months and years ahead. On the predictability side we've done probably less to date in, in trying to. Basically I think what you're saying is can the controller take advantage of things that it knows are coming to prepare for them. >> exactly, yeah. >> And yeah.We, we've definitely thought about that. There's, there's more to be done. Right now we, we take advantage of the fact that we know what the high priority tropic is. And how much demand in going to happen. The sort of growth. Is and consumption is pretty stable. So high priority, low priority, medium priority, that doesn't change that dramatically. it, I think the, I think the note about our data center note book, different from public facing win is that the diurnal patterns are actually not nearly as pronounced. They are computers pushing data. Not human being consuming data. >> Right. They are there but nearly as pronounced. So it's, it's not like we have big shifts that okay it's two p.m. on Tuesday so we can, we know this is coming and it's going to be a big shift. It actually tends to be a lot smoother than that. So our core screen say minutes optimization algorithm has been quite good. At adapting to whatever shifts might come. >> Very cool. So I guess in terms of the traffic engineering we were talking about, I guess one of the, I, I wanted to kind of make sure that I under, I understood a little bit. About exactly what was going on there. My read of the B4 work is that you. You kind of identify groups of traffic flows. And then you assign these groups to, to different groups of tunnels, based on your optimization criteria. Setting priorities for different, for different groups of traffic, as you mentioned. I guess, in, in. One can kind of surmise a bit about how you, how you assign those priorities, but it seems like setting up some of these groups, and, and tunnels and stuff can be some what of a mammoth task in and of it's self, I mean, certainly like one of the ways that one can do this with out STN, is like with MPNS For example, an. I guess, well, I, I was wondering does STN make some of these you know, grouping and tunneling types of things easier? Or do you, do you still face a lot of the same problems in terms of like figuring out how to assign different flows to groups of traffic? And figuring out how to assign a, a group of traffic to a tunnel. Is that better? >> Yes >> That problem seems to, it certainly not a new problem, and I was wondering, does it go away at all or is it kind of the same? >> I think the problem, it's quite similar. The nice thing here though is that we have a lot more flexibility in tuning the number of tunnels, and priority of different tunnels and the mapping of flow groups to tunnel groups on the fly. You know NPLS is one mechanism for doing tunneling. It's a good mechanism for doing tunneling. We chose a different one. But from my perspective IP in IP puts a label on a flow group that happens to get mapped to a path. And actually that's exactly how it's implemented in our switches, if you look look in the paper. NPLS has a different mechanism for assigning labels. So there actually is no substantial difference other than expediency. In other words IP and IP is more broadly supported in the range of hardware relative to to NPLS. And then going to the, the second part auto band with the happens to create these tunnels in a distributed manner. We again have the benefit of doing it with our centralized server. >> That makes sense. So, like in the same way as you were describing like convergence problems become easier because you don't have to explore all these you don't have to iterate through all these sub optimal solutions. You could, you could probably take advantage of other things [INAUDIBLE] and provisioning [INAUDIBLE]. >> Exactly right. >> cool. I guess one more question about B4 which was sort of, in the paper you talk a lot about problems that arise with the ordering of dual installation and so forth. And I guess there's been some work in the, in the academic community on consistent updates and networks and, and you know, how to basically take things that are expressed in high policy languages and compile them down into, into rule sets. I think there's another paper on sitcom as well talking about, you know, guaranteeing, you know correct ordering of the installation. I was wondering do you see any of that playing a role here or do you think that the problems with the role installation order in, in that it, in that it, Google wire are unique enough that there's a new set of problems still, to still be solved there? >> No I don't think that our problems are particularly unique, actually. I'm a big fan for example net core work which I think you're referring to. Frenetic, et cetera I think these efforts are. Going to be playing a really big role in STN in general. Because, this is a big opportunity to free ourselves from having to essentially, we've written the equivalent of the compiler. In a very sort of app specific way. We have nothing as glorious as the netcore frenetic one. Make that clear, but we have that functionality tailored to our system. I do CS generalizing along these lines hoping to generalize along these lines in the future. >> Yeah, what other futures in a northbound API would you like to see? I mean, it's clear that I guess from, from what you're saying that. Yeah, the net core and frenetic stuff definitely has some use. But other features, wish lists if you will. >> Yeah so I purposefully stayed away from thinking too much about the northbound API particular open flow controller because I actually think that's the wrong place to stitch things together. Standardizing that right now seems early, so my take is that actually, there needs to be some other software system that stitches together the multiple open flow controllers, and it can export A API to all applications that want to run on top of the network, and modify the network state in particular ways. So they are, I don't truly have anything. Interesting to say, yet, but it is the standard stuff of introspecting on, sort of, utilization failures and having a convenient way, as we were discussing with phonetic or et cetera to specify global behavior and have that be translated into a. Consistence if you will. Ordering of application to individual switches. >> Cool. Yeah, no, that seems like a really, a really fun thing to work on, trying to get multiple controllers orchestrated to, to To do the right thing. >> Yeah, I think a number of people pushing this direction oh what the right set of abstractions are for building, sort of the equivalent of an operation system for your network fabric. Right, we've done a pretty good job. Figuring out how to take a collection of disks, and make them look like one storage system, their collection of processors, and make them look like one big computer. We've done less of a good job taking a collection of switches and routers and making them look like a fabric. And that's a big opportunity in front of us working community. With respect to STN actually. >> Mm-hm, mm-hm. Interesting, yeah. Actually, that, that's a good transition into I wanted to ask a few questions about STN in, in data centers. And certainly I, I actually periodically reread your Portland paper when when I teach it. and, and we, we had the chance to do that again. It's really interesting to me because the fabric manager to me looks so much like an STN controller but they just haven't used that phrasing since STN wasn't in the vernacular but it's basically, as far as I can see one of the. Best fits, or like, in terms of just an example of, of applying STN type concepts in a data center. I think the paper offers just a, a very nice example like that. I guess what the, since we were talking about obstractions of fabric I wanted to kind of jump to. A question about like building the virtual switches, and it seems to be a big trend in data center networks these days, just to build these big virtual switches, like Cisco's virtual switch system, and this extraction kind of creates the illusion of a number of switches merged into a single layer 2 domain. So I guess this you know supposedly makes configuration easier does, does the, does an STN based data center architecture like Portland does it compare to this in any way does it, does it compliment it? >> I think this is the direction exactly the direction that we were trying to go toward. Is basically an I think that was explicitly stated you've probably read actually the paper more recently than I have but. You know it's like, what we were trying to build a single layer two switched to a composition of switches. So I think it's very complimentary to some of the directions in the industry. Working and doing network virtualization, is also in this direction. Basically, you really benefit from having a barrack manager, STN controller, whatever you want to call it, that can keep track of where and when things are virtual end points, virtual machines, where they're running, where they're moving to, and how they mapped to say protection domains. Right, if I have a number of virtual networks, that I'm. Hosting on a shared fabric. Who is allowed to talk to whom, maybe at what rates, doing this, keeping track of this centrally and being able to turn up and turn down configurations on the fly, I think is a big opportunity for S.T.N.. And logically centralizing the functionality any way, if not just physically on a single server, seems to make a ton of sense. I'll mention one thing since you also alluded to it, and that's configuration. I think that there is really an opportunity to make configuration easier, but I'd say that's probably one of the big unexplored spaces from a research and development perspective. How to do config and an open flow in an STN like world. >> And that problem applies to both the data center and the WAN, you'd say. >> Yes. >> Okay. >> Definitely. >> What do you think is a good way to get started on that? I mean that's certainly something that I've looked at in the past in context of just legacy protocols if you will. And certainly there's students out there that would like to make some headway on this. I mean, where do you start in sort of thinking about how to make config easier? Do you suggest looking at user cases or common things that operators do or common mistakes or, bottom up, top down? What are the approaches you take there? >> I, I think use cases is likely the right way to go basically how is config managed right now. In even some medium sized or small deployments and I think what you'll quickly find is that it is in general really messy and you know, text files with specific languages et cetera that describes things, I think again the biggest issue is that config is handled on a per forwarding element basis right? >> Yeah, exactly, it's super hard, I mean, like definitely one of my students has been looking at this like we look at change logs, and, and at campus network configs. And you know, we are basically looking at CBS, CBS tips, like you said, because it is device by device, like, well, are these 5 devices, are they changing together or is that part of one task or is it 5 different things going on? Oh, someone just backed something out or do these lines relate to one another or not, it's like very, very tough. To just sort of back out what's really going on. >> Yeah so, so I mean I'll ask a rhetorical what should be a simple question. In your enterprise or your data center or your WAN what's the typology of your network? What's connected to what? And do you have a way of knowing the answer to that? Right, it's 12:30 p.m, do you know the topology of your network? And I'll guess the answer's no and for many people cases because it's not stored in any one place. And that they're desperate databases and files and maybe sticky notes that track this information. And that shouldn't be that hard. So, I guess the other view of it is, how should configuration, how should network states, be managed? And it's a rhetorical question because I think there, there's some pretty nice obvious starting points for how it should be done if you're trying to manage a fabric as opposed to a collection of individual switches and routers and metal boxes and servers. >> It seems to me that if you, if you were starting from a, from a fabric from managing your network as a fabric, you'd be anal to kind of start with treating your, your config as maybe one of these higher level programs or, you know, something that would be written in SEN folder and then you can basically be. Because you're starting from some logically centralized program. Maybe it's easier to kind of you know, track changes or make sense of what's going on. Is that, would that be a reasonable starting point or? >> Yeah. Seems, seems eminently reasonable to me. Yeah. >> Yeah. Yeah. I mean, the other, the other thing too is I guess. One might, I, in terms of approaches one might start bottom up, right? You could look at todays networks, right? You could sort of see like, okay what are people doing across their distributed network of switches device by device and try to, try to back things out. Or you could maybe start top down and ou could sort of say. Well here we have an STN controller and here the, you know, here the types of things that you want from express and then you take some from frenetic for example and you say here are the types of things that one can do and we're going to design like a, you know, configuration, that kind of texting approach based on that, that you know, these are the types of things that operators are going to write. And this is what they're going to look like. I mean, do, do you. You think that one approach is, is better than the other, I mean they're certainly very different. I, I just wondering what you think might be, might be more fruitful, and in terms of looking at the big there. >> I, I think you have to do both honestly. And I know that, that's a cop out answer, but I think that without knowing what the problems are on the ground. You can usually solve the wrong problems with you glorious top down, clean slate solution. I think both are actually the wrong way to go, you have to in isolation you have to do STN easily. Interesting yeah, no it's it's it's dirty business I guess. [LAUGH] Working with [CROSSTALK]. >> It is, it is but I actually it's, I think that's maybe some of the STN work is really going after the capex I will capital expenditures in the network. But as important as the operation expenditures, and so again, if you don't know what the typology of your network is, how are you going to run it? So, rhetorical question, but figuring this out I would say is is the maybe highest impact thing that network community can be working on in terms of impact. >> That's that's exciting actually, because it is something I put aside, like several years ago and it is good to hear that there is, not only work to be done there but certainly, in your view, one of the more important things they have to do at the moment. >> Yeah I would say it is because it is so open, in other words there is lots of important work to be doing on the sort of STN technology front, but I think there has also been a lot of good work. Already done the STN sort of topol, sorry in the protocol controller full tolerance et cetera, et cetera space. Where as this other domain is relative untapped. >> Cool. Yeah, how do you, how do you evaluate good solution in that, in that case? I mean you can say that here's my way of sort of understanding configure, managing it, or testing, like doing regression testing and so forth. I mean it seems like, there's certain things you might do like pop a number of bugs in a file under, but then in terms of like, let's say that I had something that I wanted to show you a solution to managing a big, what would convince you that it was a good solution. >> I think that based on the bottom up analysis that you'd look at you know the three or four operations, in other words, it is so easy to do performance work because it is so easy to define the benchmarks. Right, okay, the performance is X, I made it X over 2, so what is the equivalent on the operations side, and this is the harder problem, but what are the 4 or 5 really hard, grungy, labor intensive. Error prone tasks that people have in running the network and so here's the before way with doing it with config and here's the after way and since, and the other great thing is that even though this is a hard problem. There's so little there right now that you know you can come up with a straw man and it'll be way better than anything else. >> Mm-hm, like any step forward is a big step forward. That's cool [INAUDIBLE]. Great. Well it's been fun. Like I actually definitely learned a lot. And I actually feel a whole lot better about you know, working on things like config now. I'm, I'm very optimistic there's very good stuff to be done there. >> Absolutely. >> So thanks, thanks a lot for your time, if there's any, if there's anything else you want to wrap up with feel free but otherwise, really, really appreciate you your taking the time to speak to us. >> Yeah, no, this is this is great I'll say what hopefully is obvious to everyone taking your nice course and that's I think it is inevitable that we are going to have more and more STN, maybe all STN in our network infrastructure, but we really are at the beginning, in other words there is a long path ahead of us in getting there, and it is not a one year thing or a two year thing this is a pretty big, evolutionary shift in network architecture that is going to play out over the next few years, so it is an exciting time to be in the space. >> Definitely. It's a fun time to be a PhD student. [LAUGH]. >> I think so yeah. Cool. >> All right thanks a lot Nick. >> Thanks a lot.
