Measuring Developer Productivity: Can, How, and Should You Do It?

Productivity is a big topic. We all want to be more productive — and software developers in particular get put under the microscope. Interestingly, their work is also particularly difficult to measure and assess what “productive” even is.

But we need to do it because we want developers to be more productive — and happier — because we want to achieve business goals together, better. But if developers’ productivity doesn’t meet expectations, does that mean they’re lazy? Does it mean they don’t have the right tools to aid their productivity?

How do we get to a good place with measuring developer productivity?

We could follow Amazon, which is notorious for using algorithms and metrics to measure warehouse workers’ productivity, and applying metrics to the entire company. But believe it or not, metrics aren't the full solution.

Sleuth’s CTO and co-founder, Don Brown, and CEO and co-founder, Dylan Etkin, explore this complex topic in episode 4 of Sleuth TV Live and in the Q&A below.

Don Brown: Let’s get right into it. Should you measure developer productivity?

Dylan Etkin: Yes, of course you should measure developer productivity. I think we've reached a stage where suddenly it’s possible. How are CEOs going out into the world and saying their teams aren't productive enough? How exactly do they know that? If you're not measuring, you don't know.

Don: So let's go to that next logical question: How do you measure developer productivity?

Dylan: You want to base developer productivity measurements off of the actual work that you’re doing. Companies are already investing in measurement tools that capture work, so we have the ability to key into these systems, glean information from them, and get a measure of productivity.

Don: Now, some would argue that it's actually just Jira that brings all this information together. And if you track how many issues people do, maybe assign some story points to it, then that's what ties it all together. Would you agree with that statement?

Dylan: No, I wouldn't. And look, I love Jira probably more than the next person, but the thing about Jira is it's very manual. There are some automations out there that will help you transition things, but it's very easy to be incorrect about what state something is in or to not have the full context.

If you liken it to a telescope, you don't go to the middle of the telescope and try to look on either end and get a sense of how something's going. It's in the wrong spot of the workflow. It’s where you're tracking information and there's a lot going on to the right of it. There's a lot going on to the left of it. But very often once somebody wants to be actionable and really tie into the real work that people are doing, that's where something like a Jira falls down.

Is there a sole measure of developer productivity?

Don: So you talk about wanting to see the information at the right spot, not just in the middle, but the whole thing. But no matter how comprehensive the tools are that you’re using, you’re still distilling it down to what a developer did. Is that the sole measure of developer productivity?

Dylan: I don't think so. When you just look at things broadly, there's certainly an element of how tooling and workflow allow developers to get things done at certain speeds. But we live in a really complicated world, so there's a bunch of pre-production environments and production environments. There's all sorts of reasons why things might go slower other than how fast the individual works. There's this great quote I always think of that basically says if you try to make improvements to efficiency on anything that isn't the actual bottleneck, it's an illusion.

So, the question really becomes, how broad is my aperture? How much of the entire delivery process, from concept all the way through to in users’ hands and functioning correctly, are we taking into account? Bottlenecks could live anywhere inside of that entire flow. And if you think shaving 30 seconds off of running tests, for example, points to increased productivity, but you have a two-day wait built in for no apparent reason to promote something to production, then what did the 30 seconds really buy you?

How about you? What do you think?

Don: I would say no, but I think my objection is slightly different than yours. When you're talking about developer productivity, you're also talking about “the” developer’s productivity, with this idea that we can measure individual developers’ output, compare them, and see who’s the “more productive” or “better” developer. I think that's such a poor proxy for this big problem, which is performance.

You could have a developer who focuses on digging deep into complicated problems that takes two to three days, and the final output is a line of code change. If you looked only at output metrics, that developer would seem unproductive. But when you look at the impact on the company that that change made, it’s a much different story.

I would argue that you can't really measure developers specifically because that will put you down a dangerous path, and it gets people gaming things.

How DORA metrics measure developer productivity

Dylan: Productivity is a team sport, software engineering is a team sport. We work on things individually, but this is a team effort, especially as teams grow. So absolutely, I think you need to be focusing on how your team is performing. Individuals make up the team dynamic, but it should be a blameless culture.

Don: If you just look at a developer, you're looking at a two-week period, which is really such a small part of the productivity picture. So that's where I agree with you. There’s not one single measure, but just because you cannot measure something perfectly doesn't mean you can't get close. And when you talked about team metrics, you were alluding to DORA metrics and how they might help.

Dylan: Right. So for those who don't know, the DORA metrics come from the DORA organization, the folks that were building the State of DevOps reports. And they did a lot of research that boiled down into four key metrics that correlate really well with organizational performance.

Don: Okay. So DORA metrics are ways that you can measure team productivity. Our topic here is developer productivity. How do you feel this is a better answer than trying to measure commits or tickets merged, things like that.

Dylan: What I love about the DORA metrics is they take into account real work, and they have a synergy that seems to capture a broader view of how a team is truly doing. And you can compare the DORA metrics of an apple to an orange and at least get a sense of some degree of efficiency. Maybe there are some lessons that this team's using that I'd like to move over onto this team.

Don: I found in my personal experience that there are some kinds of aspects that the DORA metrics track and encourage, such as more deployments and deploying more frequently, that really do flow down into productivity and happiness. For example, I had a situation where my team was four or five people and we were deploying seven to eight changes every day. The DORA metrics would say our team was doing well. And if you also looked at happiness and the developers’ ability to feel that they're making an impact and actually make changes, I would say our team was really healthy.

Let's be honest, metrics encourage you to work in a certain way that shows up well on the metrics. But I would say in my experience it does tend to back with more productive and happier developers. And it's not just about hitting organizational goals or hitting your numbers. It's about also empowering each individual developer.

Dylan: I can't agree more. We can only understand so much, and we want to break a problem down into really small chunks and ship those chunks so that we can build on top of each one, like a really well built home with a solid foundation. I feel like my whole career has been a progression of the industry trying to give ourselves better tools to just do the right thing.

Don: And DORA metrics give you that indication if you're going toward that direction. But let's be honest, as we know, there is no silver bullet. There is no one thing you do and everything will be good. So, what are some things that just focusing on DORA metrics will make you miss out on when it comes to tackling developer productivity?

Dylan: One of the things that we talk about all the time over here at Sleuth is how to make these things actionable. It’s great to have numbers, but it can be not that great, especially if you see a scary number and you have no idea why that number is the way it is.

Sometimes actionability is interpersonal. So, sometimes the answer to a problem is automation. And sometimes the answer is getting on a call as individuals and talking about the way we work — things like how many unit tests do we want, what's our quality bar. These are things that I don't think the DORA metrics do a great job of capturing. But I think they also have a huge influence on those metrics.

What about the SPACE framework?

Don: I wanted to take a minute here and talk about the SPACE metrics and how this kind of fits in with developer productivity. Nicole Forsgren is one of the authors of the State of DevOps report and one of the co-founders of the DORA group that created the DORA metrics. She looked at the problem of developer productivity and produced this paper.

A couple of interesting things: you can't take all of productivity and reduce it down to a single metric. Productivity is not just about what you do, your activity. It's about other aspects as well. It's about your satisfaction and well-being. How happy are the developers? It's about their performance. How are they able to achieve goals and hit deadlines? How well are developers working together? Are they communicating and working on problems as a team or are they siloed and wanting to do everything themselves? How well can a developer get into the rhythm?

Anyone who's gone into development knows that one of the biggest ways to screw up your productivity is to have lots of little meetings, all spaced out. Developers need to be able to focus on a problem for a long period of time, and they will be more productive.

What's interesting about this SPACE framework is it’s not a new set of four metrics. It's more of a way to think about all aspects of what makes a developer productive. If you look at one thing that has the most impact, it's satisfaction. It's that how happy the developer is will tend to better correlate to them being more productive.

It's actually how you frame the whole concept of productivity, looking at all the different dimensions of it. So with all of this in mind, I wanted to get your thoughts, Dylan, on this. Do you think the SPACE framework is the new DORA? You don't need DORA anymore, now you need SPACE, or do you think that we're missing the point when we say that?

Dylan: I think they're complimentary. It’s so interesting because developer happiness is key, but is the developer happy because they work in an environment that allows them to deploy quickly with safety and see the fruits of their labor very quickly, and when they make mistakes to correct those mistakes very quickly and efficiently in a blameless culture? Most developers I've worked with love working in that environment. They’re really satisfied when they have that ability to execute, and I feel like DORA is that.

DORA metrics are a measure of how we are producing, how we function as a team, how our flow is set up, and how our environment has safety nets in place to make sure we can move fast with confidence. Whereas the SPACE framework feels like it's more of a sense of how satisfied or how productive are individuals, which starts to track softer things, like the number of meetings you’re in, and how your team communicates, which are absolutely important. But I'd argue that when you have the right levers to get that culture right, you're going to see that reflected in your DORA metrics. When you have it wrong, you're also going to see that reflected in your DORA metrics.

Don: I think what stands out to me about SPACE is it shows how this topic of developer productivity is more complex than just one thing. If you ever ask an experienced engineer something, the answer is almost always, “it depends.” Is it good to have a monolith? It depends. Is a microservice better? Is Java the right answer? It depends. Measuring one thing doesn’t mean you understand all of it.

For example, take communication. You can't put a number to that. You can't say Dylan is a four on a communication scale and Bob is a 2.3 and Sarah is a one. It's so subjective, but just because you can't put a number to it doesn't mean that managers shouldn't be thinking about it. You should be asking yourself, “Does my team communicate well? Does it not? Does this person seem to struggle to speak up in meetings? How can I facilitate that better?”

Dylan: It's always great to keep improving. The concept of DORA metrics is that we have a hard number, a hard measure that we can derive from actual work. And I think this is the first time we've been able to do that in software development. We tried it by counting the number of punch cards, and then there were too many punch cards. And then it became lines of code or number of PRs.

And those first attempts to measure productivity were driven by this desire to quantify it in some way. Organizations have sweeping initiatives and it’s important to prove to people that they're actually working, and then reverse decisions if they aren’t. And that's what I believe the DORA metrics help provide. I see SPACE as providing insight where, if we can’t fix something through some form of automation, then maybe it’s an underlying issue, like your developers aren't happy or productive, and you can dig into the why around there.

Balancing out developer productivity measurements

Don: There’s a balance that needs to happen. You can't just say it's hard to measure productivity, so we're not gonna measure anything. I think your point is there are really good things that are backed up by science that show these types of measurements do correspond to high performing teams that's really powerful, but then it's also important not to walk away and kind of take a lesson from Amazon and say, it's all about measuring, because you can’t measure everything.

And sometimes when a measure becomes a goal, it ceases to be a good measure, because people start targeting their behavior to whatever makes that number improve to get a promotion, to get the accolades.

Dylan: I think one of the harder things in software engineering is knowing what are the trust-and-verify moments — like having any amount of lead time on understanding when something's going wrong and it's going wrong in a systemic way. And so, you just need tooling to help understand when these circumstances are arising.

As an engineering manager, back when I ran the Statuspage team, there were 35 devs or something like that. And so my job was not to get in the weeds of things, but occasionally to call my engineering managers that worked under me, and have them tell me about what happened, and use that to understand how the team was doing as a whole. But that's really hard when you're further away from things. And so, metrics and tooling can call these things out and help give us time to focus on them.

Advice for junior developers

Don: So you've been in the industry for a while. When you think back on your career, if you had brand-new Dylan in your grips who was just starting out in the industry and wanted to be a really good developer, and really make a difference, what would your advice to that Dylan be?

Dylan: I always used to joke that junior devs solve a thing in 20 lines of code, and then you become a mid and you decide to solve it in one line of code, but nobody can read it. And then you become a senior and you solve it in three lines of code. And then you become a principal and you go back to 20 lines of code because that's maintainable. So to the junior folks out there, I would just say don't be afraid to just get code done.

And then the very best thing you can do is get involved in a great engineering project. I was really lucky early in my career to get involved in early Java stuff. And that was where the cool things were happening with ORMs and injection frameworks, and all these things that were going to start to shape our industry. And there were people that were tremendously smarter than I was that I got to work with and learn from. So, surround yourself by a great process, go to some place that cares about the DORA metrics and is trying to get better and better, so you can learn good habits.

Advice for new team leads

Don: So we talked about your advice to junior Dylan. What's your advice to Dylan who just became a team lead. You want to make an impression, you want your team to be the best team in the company. What would you recommend to that person?

Dylan: Being a new team lead is tough because usually they're making that transition from individual contributor to management. And so my advice is, don't try to get your happiness from the same sort of source that you used to, because if you're going to miss cutting code, you're going to be sad and not very good at your job.

Your job now is to make sure that the individuals on your team are working well together. It's about finding that flow for the individuals who are a part of your team, getting people unblocked, making sure you're celebrating as a team, and improving as a team — and team leads should learn to get excited about that.

Don: Right. And then you go back to our topic about productivity. It's not about your productivity anymore. It's about the productivity of the whole team. And so that's where a lot of what we've talked about today about measuring output, using the DORA metrics, looking at communication via the SPACE framework, and focus on all of that, instead of just how much you can sling code and make the deadline and save your team's reputation.

And I think this is a lesson, not just for the developers out there or even just for managers. It's really anyone who wants to be a leader in their team. To go to that next level, try to think about the team as a whole and where you can be within it. The point is, things do change and it's about adjusting to that change and seeing where you can go.

So, to summarize, can you measure developer productivity? I think my answer for that is no, but you can measure team productivity and you can get some good signals with DORA metrics.

Where we go from here

We talked about how you measure with metrics, and the other question is, does accuracy matter when you're doing the measurement? Is it enough to look at it and say, things are going well. Are you missing things by not measuring accurately enough, specifically talking about DORA metrics? Read the next article to find out why accuracy matters in DORA metrics tracking.