An important evolution that DevOps has introduced is really getting teams and organizations moving away from talking about output, to focus more on outcomes instead. That's what we're going to talk about in this video. By the end of this lesson, you'll be able to differentiate between outputs and outcomes, discuss why it's important to shift to focusing more on outcomes, identify and explain the two major outcomes with which most businesses are concerned, define and explain several key metrics used to measure outcomes including, deployment pain, deployment frequency, change failure rate, meantime to restore service, eNPS which I'll explain that's Employee net promoter score, and a few others. We have a lot to cover here, so let's begin by discussing the difference between outputs and outcomes. So, an output is what we, as an organization, actually deliver. It's the new key feature that's been deployed in a piece of software. It's the number of bugs that had been squashed. In some cases, it's the number of tickets that have been answered successfully. Outcome, on the other hand, is what our organization gains from our output. For example, the revenue generated from the new key feature we released, or customer retention related to improving the software by fixing bugs and answering tickets. Okay. So, do you see the difference? The focus should be more on the outcome than it should be on the output. Really, in most cases, there are only two outcomes that a business is trying to achieve. One, is of course, increasing revenue, and the other is reducing costs by improving efficiency. These two outcomes are definitely tied to DevOps practices and techniques, and as you continue in the course, you will see that as you start to practice these concepts. Now, there are any number of ways that these outcomes can be measured in an organization, and we're going to spend the rest of the video discussing the most common metrics that are used in the state of DevOps report, to understand the dimensions of high-performing organizations. I'm going to touch on each of these, and give you a little bit of a window into how high-performing organizations, are performing on each of these metrics. Deployment Pain is one metric that is used. Deployment Pain refers to how difficult it is for code to be deployed into production. This is looked at because Deployment Pain can often be an indicator of how high performing an organization is. On the other hand, it can also be an indicator of employee burnout and other components of the culture. Let me illustrate with an example. When I was supporting our website's development team at Nordstrom, whenever we did a release to production, it would take a war room of between 30 to 50 people staffed 24 by seven up to 72 hours to release the new code to production. So, that is really extremely high deployment pain. In fact, it really wasn't sustainable at all. So, the team focused on strategies for getting the deployment window down to normal business hours, without such a large number of people, and resources tied to achieve the production deployment. In the end, we were able to get the deployment down to a normal work day and it took 10 to 15 people. We accomplish this by focusing on automation and breaking the releases down to smaller batches. Another metric to monitor is the Change Failure Rate. This refers to how often a change sent to production fails in some way. So, high-performing organizations have a really low change failure rate, somewhere on the order between zero percent and 15 percent. Of course, the closer to zero percent the better because of course, it's desirable to have as few failures as possible. In my experience focusing on deploying smaller changes more frequently, is a great way to reduce the change failure rate. Also, having a way to roll back easily and leveraging feature flags. At Nordstrom, the mobile app team did all three of these things, if we introduced a change that cause an issue, we could turn it off with a feature flag. If we didn't have the feature flag configured, then we had an automated way to roll back any changes. Deployment Frequency is another metric to pay attention to. So, how often are you able to deploy? High-performing organizations are deploying on-demand, and delivering multiple deploys per day. So, obviously, that's directly correlated with the deployment pain, because you probably aren't deploying frequently if your deployment pain is high. As I mentioned above, we were able to reduce the deployment pain. Deployment Frequency did not increase though until we started migrating to the cloud. Once we started putting capabilities in the cloud, we were able to deploy on demand. At Nordstrom, we moved our outfits page to the cloud first, it was low risk and also a good way for us to learn before we went big. Prior to moving to the cloud, the page was on the same release cycle as the rest of the website, every five weeks. When we moved it to the cloud, we deploy changes multiple times per day. We had blue-green deployments, and could validate our changes before taking them live to the customer. Once we validated that this all worked, we moved on to our product page, the Moneymaker. We weren't iterating fast enough on that experience and once we move to the cloud, we could make changes multiple times per day, and this allowed us to test and learn, and also respond to market needs quickly. Lead Time for making changes is another metric you could look at. What it's focused on, is how long it takes once you've identified the need for a change, for that change to be fully integrated and in your customer's hands. In some high-performing organizations, their Lead Time for changes is less than an hour, which is pretty amazing and challenging, but if that's even somewhat realistic in your organization that is definitely something to strive for. Another metric is the Mean Time To Restore service, known in the industry simply as MTTR. So, when you have an issue in production, or an outage of service, how long does it take to identify the issue, fix the problem, and ultimately restore the service on average? In really high-performing organizations, their MTTR is also less than an hour which is very impressive. The percentage of Unplanned Work is a key indicator of organizational performance as well. So, if your team is frequently disrupted with things that were not planned in a scope of work, naturally that keeps you from delivering your project against your current roadmap. Related to this is dealing with incidence. So, if you're not planning capacity for an incident or an outage, then that can turn into Unplanned Work, and coincidentally likely results in a much higher MTTR than is desired. So, Unplanned Work is a metric that's really important to watch, and see how much of that is going on within your organization. We'll discuss Unplanned Work in more detail in another video. The last metric that I want to touch on in this lesson is literally eNPS. This stands for Employee Net Promoter Score. So, a Net Promoter Score is a pretty widely known industry metric. Customers are usually asked how likely they would be to refer someone to this business. So, for example, if you just went shopping and completed some purchase at a retailer, they may ask, on a scale of 0-10 with zero being not at all likely, and 10 being very likely. How likely would you refer a friend to our store? An Employee Net Promoter Score version takes shape in two ways. There's actually two questions that get asked of the team. One, is how likely you would be to refer someone to your team? Then also, how likely would you be to refer someone to the overall organization? Again, typically with the answers being between zero and 10. This gives you an understanding of how many of your employees are engaged, and having a good experience within their team, and within the organization. It's about culture, right? Culture is extremely important within any team or organization, and so that's a metric that you can use. I've used it in the past and found that teams that were outcome-based, had line of sight to the value they were delivering, had continuity with their team, weren't getting contact switch lot, owned their entire stack. Everything from you build it, you own it, you improve it, had the highest eNPS of other teams that might not have those same attributes. On the other hand, teams that we're still focused on output, but didn't own their quality, and didn't know their release capabilities, often did not have high eNPS scores. So, eNPS is a really good way for you to get an indicator of culture, and also of the capabilities that you're using within the organization. All right, that's enough for this lesson. To quickly review, we covered the difference between outputs and outcomes, discussed why shift to outcomes as necessary, and discuss the following key metrics used to measure how well an organization is performing, Deployment Pain, Deployment Frequency, Change Failure Rate, MTTR, Lead Time for making changes, the percentage of Unplanned Work, and eNPS. We'll wrap up the lesson in the next video by discussing some case studies from my work on how DevOps can have a powerful impact on teams.
