How To Use DORA Metrics

So, you're measuring DORA metrics to understand your team’s efficiency. You’ve read the State of DevOps reports, maybe even the “Accelerate” book. You totally buy into DORA metrics. What do you do next? How do you use the metrics to help you change and improve your organization?

We offer four things that DORA metrics can be good for.

But first, here’s a super quick crash course in DORA metrics. It starts with the State of DevOps Report — an annual survey where researchers tried to understand what makes the best companies work. They learned a few things that further defined four metrics that strongly correlate to high performing teams, called the DORA metrics. (DORA stands for the DevOps Research and Assessment, a startup group of researchers who identified the metrics.)

DORA found that if a team scores highly on the four metrics, then that statistically correlates to high performing teams. The four DORA metrics are:

1. Deployment Frequency: how often you deploy
2. Change Lead Time: when you deploy, how long it takes to go from first commit to deploying
3. Time to restore service, also known as Mean Time to Recovery: when you fail in production, how long it takes to restore health
4. Change Failure Rate: when you deploy something to production, the percentage of those deployments that fail

In their study of over 30,000 organizations, the DORA team found that there were four categories of performers for the metrics — elite, high, medium, and low — which allows for comparisons with other teams. For example, if you want to be in the elite group for Deployment Frequency, then according to DORA you should be deploying multiple times a day. If you are in the lowest group, then it's fewer than one time per six months.

If you'd like to hear Sleuth's CTO Don Brown's take on what DORA metrics are good for in the real world, check out this video from episode 2 of Sleuth TV Live.

Four reasons to measure DORA metrics

1. Assessment

What's nice about being categorized into one of the four performance categories from the State of DevOps Report is, when you are a development team, surrounded by your own problem, sometimes you don't understand how your team fits in. And that's where an assessment can be useful.

If you're thinking your team is executing efficiently, and then you measure the DORA metrics and compare your results to the categories, you can see where you really fall. The DORA metrics can be a really useful way to benchmark yourself against other teams.

Of course, you can't really compare deployment frequency of a mobile app that's deployed every couple weeks to a SaaS app or a microservice. But generally, you can start to put yourself in the categories.

As DevOps has been adopted by the industry, the State of DevOps study shows more and more teams are moving into the elite category. This not only indicates what the standard is in the industry, but it also offers a little bit of a warning to what your competitors might be doing.

Of course, just because that’s where the industry is going doesn’t mean that's where you have to go or that it’s relevant to your team’s situation.

Again, metrics are a tool. They are not the answer. If you don't score well on a metric, it doesn't mean you're doing it wrong.

It means that this is indicating something which may or may not be relevant to you. So like everything in software, like everything in technology, it all depends on context.

One example: When our CTO and co-founder Don Brown was at Atlassian, his team was deploying Confluence daily and seeing real momentum. However, the team started to have incidents.

But instead of improving things while keeping the same deployment frequency, a group of people decided moving fast was causing the problems and they needed to slow down. However, assessment of the DORA metrics and performance categories would have offered some context to say that actually, the industry is going faster and elite teams are the ones achieving their organizational goals while decreasing incidents. In that particular circumstance, maybe slowing down was the right call, but again, looking at the industry at large could have played into a decision.

2. DORA metrics help measure progress on initiatives

Let’s take the example of getting teams to deploy more frequently – going from deploying every three months to once a day. Anyone who's done this in a large organization knows a lot of energy goes into keeping momentum going for the project and justifying it again and again.

It takes a lot of resources because you have to spend engineering effort to speed up deployments, and to communicate the more frequent changes to all stakeholders, like sales, marketing, support, and other teams. It’s a lot of work, and you need to measure it to justify it.

The DORA metrics offer a way to automatically measure these things over time so you can see how things change. And that's rewarding. That's exciting for anyone who's interested in the project to see the progress. That's what helps you maintain that momentum.

It’s important to measure at the beginning of a project so that as you're doing your transformation, your stakeholders can see the progress and have their decision confirmed again.

3. Track the impact of an unrelated change through DORA metrics

Now, let's say you get to where you want to be. You got your deployment frequency to six times a day. You’re done, right? Well, things change all the time in engineering, so it's not enough to hit your goal.

It's about measuring over time so you stay where you want to be over the long haul. So when it comes to tracking the impact of unrelated change, an example might be moving a monolith to microservices. In order to know how a project like this is going, you need proof because people will ask for it.

Where's your data? Where's your metrics? You need to have an understanding of the progress of unrelated things. As you moved to microservices, how did that change your Deployment Frequency? How did that change your Change Lead Time?

By measuring, you can put some of the numbers behind it to help you with the unrelated change. As the change is happening, you can verify that it's not breaking something. DORA metrics can be like unit tests. They're an insurance policy you put in place so that as you do things later, you make sure you're not breaking something.

4. Track the impact of team scale events

What about changes to your team size and structure? How does adding people to your team or scaling down affect your software delivery performance?

As your team grows, you want to make sure your software delivery performance is at best unchanged and ideally improves over time. You want to see that the onboarding processes you put in place are helping people move faster.

If you want to understand how new developers are being onboarded, review time will be one of the key metrics to pay attention to. If it gets bigger, that could tell you as a manager that you need to improve onboarding new developers so they can be successful quicker and create less stress for experienced developers.

DORA metrics can come into play here because you can look at them over time and see how they change as your team changes. This is where metrics can go from a measurement to actionable information.

One more thing about DORA metrics

Metrics are good to know, but metrics should not be the answer. Any metrics, but especially DORA metrics, should help you ask the right questions. And then, your responses to these questions help improve your team.

For example, the DORA metrics tell you your Change Lead Time has gone up. That lets you ask, “Why has it gone up? Why has that happened?” Maybe your onboarding isn’t up to par. Why is your onboarding not good? Then you can come up with strategies to improve your onboarding.

So, it goes from a theoretical high level number that you think you just needed to report to management, and it turns into something that a developer, team lead, or engineer can use to ask the right questions and make changes that help their team in very concrete ways. This is where metrics become real.

Final thoughts

With any kind of tool you adopt, be ruthless. If it doesn't do any value for you, it might not be the right tool for you and DORA metrics should be no different. The answer is not to measure metrics. That's not what helps you become an efficient team. The value lies in using DORA metrics as a tool to ask better questions, to make real change in your team, and improve your efficiency.

What's next?

We can’t forget that there is an elephant in the room when we talk about DORA metrics. Two metrics, MTTR and Change Failure Rate, don't ever define what failure is. How do you define failure? If you're measuring failure, you need to have a discussion about what failure is, why it's important to measure it, and how you measure it. Check out our next article for you on how to learn from software failure.