Ep. 151: Rethinking How We Measure Developer Experience

In this episode of Definitely Maybe Agile, hosts Peter Maddison and David Sharrock dive into the complex world of developer experience and its impact on organizational productivity. They explore the various methods of measuring developer experience, from telemetry and flow metrics to impact metrics and qualitative data. The conversation touches on the challenges of interpreting this data, the importance of trust in the workplace, and the long-term implications of focusing on developer experience, including employee retention and engagement.

This week´s takeaways:

• Developer experience is becoming a crucial focus for organizations, driven by factors such as the adoption of AI tools like GitHub Copilot and the need to improve productivity and retention.
• Measuring developer experience requires a multifaceted approach, including telemetry, flow metrics, impact metrics, and qualitative data gathered through surveys and direct conversations with employees.
• Long-term considerations, such as employee retention and engagement, are essential aspects of developer experience that require careful attention and cannot be addressed solely through short-term productivity metrics.

Tune in, and rethink how you measure and optimize developer experience for sustained innovation and long-term organizational growth.

Chapters

• 0:04 Understanding Developer Experience and Metrics
• 8:59 Measuring Developer Impact Beyond Productivity

Transcript

Peter: 0:04

Welcome to Definitely Maybe Agile, the podcast where Peter Maddison and David Sharrock discuss the complexities of adopting new ways of working at scale. Hello.

Dave: 0:12

Dave, how are you doing, peter? Doing very, very well. Good to see you again. Yeah, good to see you.

Peter: 0:18

So what are we talking about?

Dave: 0:18

today you mentioned something developer experience and I think, as we're trying to make some notes around this, this is a pretty interesting topic, right? So maybe, if I describe what I understand it to be and you can clarify. So, more and more organizations, they're looking beyond, well, they're looking at productivity, and it's the problem of our time. Let me say that it seems to be very visible in a number of markets that we're working in, and one of the keys is they're trying to increase retention and engagement and involvement of workers, whatever that is. So developers in the field of IT. So we're now beginning to look at developer experience, how to measure it, so that our engagement scores can go up and innovation can go up and productivity can go up.

Peter: 1:03

Yes.

Peter: 1:04

So I think, there's a few different pieces there, if I impact it. One is the uh developer experience being a leading indicator of productivity. So if, uh, if your developers are happy, then productivity goes up. So and it'd be, which is, uh, a sort of colloquial way potentially looking at it. Another piece that is interesting and and some of this came back, like from everyone, I think, is at this point largely aware of the work done by uh, nicole forescreen and uh and the accelerate book and dorametrics, and there's the fact that dorametrics is there's way more than the four that everybody knows about and that talks about on page 19 or whatever it is of that book, but the um, they they also. About a year or two later, nicole came out with something called Space Framework, which is actually a framework for selecting metrics for different use cases. That introduces sentiment and other pieces as well, and Dora could be thought of as an implementation of space, and so there's this sort of interesting sort of space there around.

Peter: 2:05

What sorts of things do we measure in organizations to start to understand this concept of developer experience? So there's lots of things happening at the moment, but one of the primary drivers I'm seeing in the marketplace right now is this onset of Gen AI tools and things like co-pilot for GitHub in particular, as an example, coming directly into developers' worlds and giving them prompts and helping them code, and so you can very much see well, the CIO is saying, well, I've got a thousand developers. I'm having to spend an extra few hundred thousand a year on giving them access to this tool. And the CEO saying to the CIO well, does that mean you can now fire a bunch of developers? Or, the best case is, do you have the right mix and the right people in the right roles now, given the impact that this potentially now has on your organization, which I think is bringing it to the forefront?

Dave: 3:04

I kind of have a pin in the board to talk to you about space and get more definition there. I think certainly that would be useful. But before you do that, it feels to me like you are doing a great job describing all these wonderful things supporting developers through co-pilot and things like this and then right at the end you said and now we can get rid of everyone who is no longer needed. So does anybody feel that that might have some impact on developer experience? Potentially, and if it does, how do you measure it.

Peter: 3:36

I think it's kind of the elephant in the room for some of these questions that are going to be coming at CIOs, and some of the right way of looking at it is it potentially doesn't impact the makeup of your teams, because tools like Copilot will help your junior developers get better faster, learn faster, they get access to knowledge faster.

Peter: 4:01

It's being shown that it's of less value to your more experienced developers, because they can already understand that and do better than it can most of the time, and your medium developers. They're going to be able to make use of it, too to get better and to start to develop and deliver solutions for the business faster. Now, based on all of that, if this is all true, do I have the right makeup of teams? Do I have the right people in the right roles? Do I need to change my organization at all as a consequence of this? And you've been seeing some of these moves in the market as well, with some large organizations who've sort of shifted makeup of their workforce and saying, hey, we're going to eliminate these roles and hire these roles, and if you look at the makeup of those roles, you'll see that they've kind of shifted, in some cases, the types of technical people that they're hiring.

Dave: 5:00

I feel like we're straying quite dramatically away from our conversation starter around developer experience, right? So I don't disagree with everything that you're saying. There is, I mean, that one is the wave of continuous. It's like a wave breaking on a beach. It's not one wave and it's gone. There's another wave and there's another wave, and right now we're dealing with one, there'll be another one along. All of those things have an impact on are we structured in the right way to deal with what we're looking at, not least of which we talk so much about agile organizations? We see fewer and fewer organizations structured than there were 15, 20 years ago when we kind of got into this industry. Now there's a really clear understanding of sort of what's going on some form of aligned value, cross-functional teams, aligned to business need. How much of it is truly a kind of classic agile cross-functional team that's dedicated. We can leave for the reader or the listener to think about, but that's a structural shift. It's not developer experience.

Peter: 5:58

Well, I think there's two pieces here in that. I think we've got enough evidence now from everything we've just been talking about to say that there's lots of really good reasons that people are wanting to measure and understand developer experience these days, because a lot of these things are now deeply targeting that particular section of the organization. So there's a question around what is developer experience? There's a question around how do we go about measuring it? And then what sort of conclusions should we draw from that? What sort of approaches should we take? How should we come at this so that we do it in a productive and healthy way that helps the organization evolve?

Dave: 6:38

I'm kind of skeptical as I listen to that, mainly because and you and I, we talk about this in many different ways, but if we just look at productivity and stop talking about developer but think of employee productivity, just generalize Productivity is something that, as any organization is always any country, any organization, whatever it might be in industry, is always seeking to improve. That the data is pretty clear that you invest in technology in order to increase productivity, and that one you can kind of go and look at the literature it's not something that people you know. You invest in the people who are doing the work in labor, or you invest in the technologies that they're using, and those are the things which are going to improve productivity. When it comes to measuring that productivity and wanting to be able to see that the technologies are driving that productivity up DORA, a number of others, outcome measurement all of these are very valuable, but now we're stuck with that. As soon as I put a metric up there, people's behavior changes to drive that metric up.

Peter: 7:35

Yeah, the metrics become targets and targets drive behavior, and so you end up with looking to say, okay, how do I start to look at this in different ways? I think the third one. You mentioned two of the things that can be done to increase productivity, productivity being the amount of work delivered for the unit work over time. So one way is you could hire more people, increase your capacity. Another way is to apply technology to automate processes. Another way is to reduce your operational costs in other fashions, and so these kind of give you ways of improving productivity. But understanding how the system is behaving on a macro scale is a critical piece, and one of the pieces that we always talk about in this space is a lot of that sort of has an underlying assumption that output is consistent for those changes, whereas in a delivery system that's very much not the case, and we should be looking more at the outcomes that the system produces and seeing how they work and what you're describing.

Dave: 8:38

My interpretation of that would be or one of the key levers here is people, not the system. If we think about systems, we tend to think about mechanistic systems, and certainly a one of the key levers here is people, not the system. If we think about systems, we tend to think about mechanistic systems, and certainly a lot of the process, improvements and so on are all focused on this sort of mechanistic view of how a system works. But the people in the system are hugely influential about the success or not of that system. And if you look at a lot of different areas people are recognizing you know the research, the kind of analysis of these types of things, are recognizing the influence of people on how these systems work.

Dave: 9:12

And I'm thinking this is a weird one, but bear with me Crowd control in international football matches and one of the recent papers I read around this just because occasionally you've got to read outside your field, I guess.

Dave: 9:25

But one of the interesting things is that there are certain European countries I'm not going to mention it's the UK, because I don't want them to be labeled with the same with this whole kind of individuals within the crowd who are diffusing situations the moment they appear, works way, way better than kind of the command and control. Here's a line of police protecting certain things and his behavior is being imposed on them. So we're beginning to understand these systems from the people side of things. People side of things, and I don't know that we really understand that when we go and work with employees and developers and have them work on a project and then suddenly change direction and wonder what impact that has, or what happens when you know 10% or 5% of your organization suddenly loses their job, how does that impact how those people in the organization behave and present themselves when they come to the office?

Peter: 10:23

Yes, yeah, and I think. Well, I think we've all experienced or seen the effects of some of these things. So we but I don't know that we have a good understanding of how to go about measuring it. And there's three sorts of things that we apply when looking to say how are we going to see how the developer experience is going in the organization? One and this is where people often make them say they start with telemetry.

Peter: 10:48

Telemetry we've got so much telemetry now. The tooling is so much better at giving us information about exactly what's happening. I can look at code commits, I can look at how my CI servers are doing. I can look at automated tests. I can look at all this information, pull it all back deployment scripts and I can start to draw understanding about how work is proceeding in the system and what's sort of happening. That gives me telemetry.

Peter: 11:10

It gives me one view, but it's very mechanistic in terms of what it is.

Peter: 11:13

It doesn't really tell me anything about whether that's effective or achieving the outcomes I want, or if it's how the people doing it are behaving, or if they're just like shoving whatever they can into one end of the pipe and whatever's coming out the other end.

Peter: 11:26

Who cares, it doesn't give me a lot of information from that perspective. Necessarily, I can go up a level and start to look at flow metrics and start to look at like, how is the work proceeding, like, how does it get done? Does it get stuck? Does it flow nicely through the system? Where is it getting stuck? I can start to look at things like that through throughput and other pieces, and that gives me more information. Where I really want to get to is start to look at impact metrics from least to least, so I can start to understand what is the impact of the changes I'm making to my customer and how can I relate those changes back and start to look at it. That requires a lot more, whereas what more thought and work and isn't a static thing that you can just fire and forget Got to start to continually look at that and learn from it.

Dave: 12:07

I still feel like you're not getting to the point there. You're describing it.

Peter: 12:11

I'm going to get there, I'm going to get there, I've got to get it.

Peter: 12:13

Give me a moment. Give me a moment. Beyond that, we start to look at quality. So this is that's what I was saying, no-transcript, but we're very careful what we ask. But this gives us as well, a more qualitative information about the sense of how the system is. Now we can start to look across that, start to try and draw out sentiment or how are things working, and then start to say, well, what actions do we want to take of that as anything? And we'd start to run experiments into the system and start to think about how can we improve the experience, the developers operating in this system, where might there be bottlenecks, where might there be problems? But we need to bring all of this together to start to think about it as a whole.

Dave: 13:27

But is this? I mean, there's something fundamentally I feel flawed about the way you're describing of picking up experience and sentiment, and part of it is, if I've got, you know, the potential of, say, a redefinition of my job hovering over me, how I'm going to answer those particular questions might change and I think that's a very I mean there's a number of surveys in the news that have got out into the public domain which show a very clear indication that the employees are taking their frustration out.

Dave: 13:57

On some of these surveys, I would argue to make a very strong point about what's going on in their eyes, right. So that's one side of it, and I feel the other one is the difference in cycle times, Like when we look at telemetry and we look at, say, even market profitability is measured quarter to quarter. So if I'm going to look at customer experience and so on, that's effectively going to be measured quarter to quarter in some ways, and I'm missing the sort of longer trends of, say, employee retention in an organization or customers sort of shying away from a company because of something that is going on. These are long wavelength changes in a population and I don't know that what you're describing really measures that.

Peter: 14:43

Well, we're going a little beyond developer experience there, which would typically be in like how is the developer experiencing their delivery processes and practices? Are they able to deliver within the organization versus there's the like? Is what I'm producing valuable to the organization? How's the organization experiencing?

Dave: 15:04

I'm not sure that that's quite correct, and what I'm thinking of is the conversations that we've certainly had around engagement, developer experience, employee experience employee engagement at least has a big part. I mean, they may be looking at you know, work in front of employees right now and what that experience is, but a lot of it is looking for these long wavelength things like retention. If we're investing in people, how long can we expect them to stay? Do we have a potential sort of collapse in retention happening because of some of the changes we're making? And I get that many organizations don't think that long-term.

Peter: 15:38

But some do, and I think retention's essential.

Peter: 15:42

I mean, if we're looking at especially the benefits of the tools we were mentioning earlier, retention is a much better benefit to look at than productivity, because productivity nominally drives well, it's a cost savings If a cost savings, cost savings directly translate to people.

Peter: 15:57

So there's a potential difficult conversation to have there, whereas the retention conversation is the cost of replacing a person who leaves is much, much higher than if they stay. So it's much, much better to retain them. And I think when I was going through it I maybe went through it a bit too quickly, but one of the key pieces I said there it's not enough just to use telemetry and surveys. You've got to go and talk to people, you've got to go and run other mechanisms to go and pull that information out and talk to them, because the kinds of things like the sword hanging over people's heads is going to color it. But you're really going to get that from going and talking to people and seeing how they're feeling about how things are going and are things working? And how do I start to get a sense of how the organization is responding to the changes that we're bringing into the organization.

Dave: 16:46

Yeah, and I'm still like cautious of exactly how like it has to be trust there.

Peter: 16:52

So that has to be trust, and you won't have it everywhere, you know. Do you want to find some way of pulling this all together?

Dave: 16:56

Well, I have my perspective. Maybe I'll start with a couple of observations from where I'm sitting or standing and we'll start with that. So what one of them is? I completely agree. There's a whole focus on productivity and we should be watching for that, and things like telemetry and you know the outcome-based metrics.

Dave: 17:14

Describing the system is the more we know about the system, the better, and we're in a world now where we really can look at many, many different sources of data and then use that to interpret how well the system is working.

Dave: 17:26

I think there's tons of value in that. I mean like huge amounts of value, and so you're going to be able to see what happens when certain technologies change and things like this. So that one great, when I look at what I want to see with, say, developer experience or employee experience, is something with a longer an eye to the future, a longer wavelength change, and I think that one's a lot more difficult, not least because it starts clashing with some of the decisions made and they may have to be made at the executive level and how those consequences get communicated or bounced down. And again, I think it's I don't think it's it's rocket science. We know what that communication pathway could be, but often it's not. Always it doesn't leave a good taste. It's like positive atmosphere and therefore if you are trying to look at these longer sort of more future-oriented things like looking at retention and so on, it just takes a lot more work to get into that position to do that.

Peter: 18:19

Yeah, there's, and I think that's one of the other common things that we're seeing in organizations is that this isn't something that you can just do side of the desk. To do this, you need people who are focused on actually pulling all of this together and really thinking about what does this mean for you here? Because it's going to be different for every organization in the context that they're in, and you only need to look across all of the major companies and non-major companies who are working on this type of thing and they all do it differently and they've all got a different uh makeup, a different set of a different approach to measuring things like developer experience and looking at that across the organization. There's different things and different metrics that they look at and and there's lots to learn in that space there's lots of things you can pull together to start to build that picture and and all the evidence and the research shows that there's a lot of benefits from doing that.

Dave: 19:11

Yeah, and I feel, as you're describing that, one of the things that strikes me is A there's never going to be a single way of doing it. So I don't think we're looking at some sort of consolidation of how organizations do that. There will always be many different ways of doing it. And I think the second thing I'd pick up is in years gone by maybe I should say in decades gone by that was the job of leadership. I mean, there were some simple surveys that would go out and a few bits of data that you'd pick up here and there, but basically, leadership were expected to be on top of that sort of employee mindset, if you like, where they sit, and I think it's great that we've got lots of data for that. Now we've got to learn how to work with that data in a way that's successful for everybody involved.

Peter: 19:51

I agree with that. Yeah, definitely All right. Thanks very much, Dave, as always, and you can send us your feedback at feedback@ com a nd don't forget to hit subscribe. Hope you all enjoyed it.

Dave: 20:01

Thanks again, looking forward to hearing from all our subscribers.

Peter: 20:04

You've been listening to Definitely, Maybe Agile, the podcast where your hosts, Peter Maddison and David Sharrock, focus on the art and science of digital agile and DevOps at scale.