Ep. 124: Transforming Leadership with Data-Driven Decision Making

Unlock the power of data in shaping your organization's future as we tear down the myths surrounding data-driven decision-making. We dissect the essential cultural shift necessary for integrating data analytics into the heart of business strategy, steering clear of reliance on gut feelings or misleading numbers. We dive into practical applications, from planning poker to interpreting customer experience metrics. We provide the knowledge to avoid common data interpretation pitfalls, such as those exemplified by the beer game in supply chain management.

For those striving to guide intuition-driven leaders towards a more data-informed approach, our dialogue offers inventive tactics for leveraging trusted allies and fostering a climate of persistent, creative persuasion. 

This week's takeaways:

• Data-driven vs. number-driven decision making: Data alone isn't enough. Context and understanding of statistical deviations are crucial for sound judgments.
• Overcoming resistance to data-driven decisions: Inflexible leaders pose a challenge. Directly challenging their intuition might be ineffective. 
• Empowering teams and managing technical debt: Create short feedback cycles to enable rapid decision-making and results demonstration.

Resources:

Algorithms to Live By: The Computer Science of Human Decisions - https://a.co/d/dzojO5k

Jeff Bezos- https://www.youtube.com/watch?v=Ub585Pn4yro

Tune in to transform how you harness data for smart, insightful leadership decisions. We love to hear your feedback! If you have questions or would like to suggest a topic, please feel free to contact us at feedback@definitelymaybeagile.com.

Transcript

Peter: 0:05

Welcome to Definitely Maybe Agile, the podcast where Peter Maddison and David Sharrock discuss the complexities of adopting new ways of working at scale. Hello Dave, how are you today?

Dave: 0:15

Yeah, very good, very good, and I was just going to say I think the intention this week we're going to try something different. We're going to record a conversation as we kind of get it up and running, rather than sort of pause and do the breakout first. So this is us chatting about data-driven decision making.

Peter: 0:33

Yeah, so listen in, see what you think. This is an experiment and let us know what you think, because we've found that we often have these great conversations leading up to the podcast. So here's one what's been going on for you this week, dave?

Dave: 0:47

Well, maybe what's interesting is everyone's talking about database decision making, and I wonder if there isn't some interesting stuff to do about the foundation before you can really take database decision making in.

Peter: 1:03

So when you say database decision making, I have a database?

Dave: 1:08

I mean data informed oh database, oh, so data-driven. Data-driven. There you go. Yeah, I thought I'd better clean that up.

Dave: 1:17

Yeah, yeah, I was going to say like database, okay, well, and this really is less to do. I mean, part of it is to do with how you inform those decisions and how you use data. That's for sure. But there's a big chunk like the prerequisite that I can imagine us needing it. Well, I've seen needing is number one. You need a cultural or an organization that has a culture of using data. So so many times you actually get data coming in informing your decisions, and then somebody says and I am going to overrule that because my pink, my little toe is itching and therefore we should do this and I just it's not.

Peter: 1:59

It is not a. It's not a new concept, but, yes, that is. That's an interesting one, and one of one of the examples I see of that is and I this came up recently in an engagement where the whole jobs, not just way to show up as job first yeah, and I always describe this as well you can come up with a bunch of numbers that you're going to use to compare things, but you're making this shit up anyway.

Dave: 2:26

Yeah, and this is a really interesting, that isn't data-driven decision making, right, that's number-driven decision making, because I'm putting a number in there that is somehow a proxy for something, whatever that value is. So it's interesting the weighted shortest job, first calculation, is a proxy for cost of delay calculations, but all of the numbers feeding into it are so broad that it's a tool to give confidence to the conversation that we are having. Exactly, and that's what I meant.

Peter: 3:01

I mean you come up with the numbers and you have a conversation around like well, typically I just introduce like so you say that this has dropped their due date. Like okay, explain to me what's actually driving that. What's behind this? Like what is the real cost of delay behind this? Like why is that number so high? Yeah, and then that then it creates the conversation.

Dave: 3:25

And that's not what I think of when I think of data-driven conversations or data-driven culture.

Peter: 3:30

So in those situations.

Dave: 3:32

I think it's a little bit like planning poker. It's useful to have a facilitation tool that allows the conversation to be on what it needs to be, rather than chasing a number because of its perceived accuracy, for example. So these are more. Those things that we're talking about are more facilitation tools that allow us to focus the conversation where it needs to be. Yeah, and all power to them. They're really useful in those many situations.

Peter: 4:01

So, in that case, the data in terms of, I think more of the operational telemetry type stuff where I can make a decision based off a number of different things, and this can also go into customer experience, like where am I? Yeah, where are my users using the website? Like, what are they clicking on? If I've all had an experiment, I put it out there. What am I learning?

Dave: 4:21

Well, and I think that there is there's a big space here of, and I think of, our cars. If any of us are driving a car that still takes fuel of some sort, you've got a fuel tank.

Dave: 4:33

They all take fuel. Okay, but let's. Yes, you are correct, but so a gas powered engine fuel tank, the number on the fuel tank, that is almost not data driven decision making that we're talking about, because nobody is sitting there looking at their tank on empty and arming and arming about making a decision. That's almost like a gate that we have to go through. We know we have a. Well, we talked about time criticality.

Dave: 5:03

I've got a point at which I have to take action. Otherwise the data I'm gathering here is worthless because the car is no longer moving. So those that's not really what I think of when it's data driven, because that one's a you know, if it's the number of units on the shelf or whatever it is, that's, it's not really a data driven decision per se. But the ones that you're describing about, say, customer experience, open to interpretation, there it gets really interesting.

Peter: 5:29

It does. What do you think of something like? I was listening to the fifth discipline, peter Senghi, the one I was out for a run the other day, something like the beer game.

Dave: 5:40

Actually that's an interesting one, so I like to throw out interesting ones occasionally. When you say the beer game, you're talking about the supply chain.

Peter: 5:48

I introduced a little bit of a wobble in my supply chain and it causes whiplash and fluctuations all through it because of the reaction of the system that came to mind when you were talking about inventory, because ultimately that is inventory.

Dave: 6:04

Oh, this is so. Now we're in that gray area because my immediate, as you're saying that I'm thinking, hold on, that's the fuel tank problem. What I mean by that is something we can measure. We can understand that the fuel tank is a very one dimensional, simple understanding. The beer game, that understanding is supply chain. Well, of course, that supply chain is actually very complex, but it certainly drives many. Now it's a good area for us to discuss because the data there Is open to interpretation. It's a complex system so there's always room for interpretation, but we're also getting from that and a system that has heretics that you know. Experience tells us that we should be using data to make good decisions about our supply chain and how we manage it and all the things that the beer game attempts to highlight for us.

Peter: 6:53

Yeah. So I suppose we should probably explain the the listener part of that, the beer game being that you've got the retail store that's selling the beer, you got the wholesaler that's selling the beer to the beer store and buying it from the brewery, and then you've got the brewery that's manufacturing it and basically, under normal circumstances, this small brewery, they sell like four cases of beer, four cases beer, four cases of beer retail store and everything flows through as normal. Then somebody mentions this small breweries beer at the end of a song and the orders double. But the result of the orders doubling in supply chain is the retail store Panics as well.

Peter: 7:35

Orders double because they don't have enough. They had a little bit of inventory, enough to fill it, but they now no longer have that inventory. But then, because it takes time for the brewery to ramp up, they keep ordering more and more because they're trying to satisfy the need. The wholesaler, stuck in the middle, is ordering more and more and more from the beer store and the and trying to deliver it. And what happens is that the retail stores eventually eventually get the amount that they need and supply that, but now they don't need anymore, so these just start ordering zero. Meanwhile, all of the orders that have been ordered by the wholesaler start to come through and all of a sudden the wholesaler has a massive amount of venturi. The breweries production capacity is massively up, but the retail store has as much they need, and for evermore even.

Dave: 8:24

Yeah, yeah, I love that exercise. The learning from it is tremendous and and it's such a simple like the actual shift that causes that imbalance and those sort of whiplash behavior in the supply chain is such a small little blip, it's not even.

Peter: 8:43

Yes, it doubles for a month and doubles for a month and causes this massive overproduction and like hundreds of cases of beer to be sat there which they can't get rid of because there's no demand for it anymore and it's a. It's a great. So, anyway, back to data, decision driven decision.

Dave: 9:02

Well, and it's interesting because this is. You know, supply chains are probably one areas, but you could describe any number of different experiences online. For example, digital experiences, where there's a lot of information about the data that's being made in the world and a lot of data. In theory, therefore, you could have data informed or data driven decisions being made. However, it's not. You kind of need a culture that understands how to work with those decisions. Yeah, and I got you.

Peter: 9:32

That's actually the. I agree. That's actually one of those problems. Where it typically falls apart is that people don't necessarily believe the data. Some people are also more. Some people are more data driven than others. Some people this relying on your gut, like. I've seen executives who've gone like, well, I don't care what the data is saying, I know in my gut that it's actually this and and there's there's another interesting part around. Data can be context as well. Right, so that's the other side of it, like the context of data, because that can also drive you to make the wrong decision. The classic example of that and we might have even been talking about this a couple of episodes ago we were talking about service management and service desks, because this is a classic problem in that is that my service desk numbers may look well, I've met all my SLAs, everything's great. I can, I've hit all of the numbers, the data looks fantastic, but if I go talk to my customers, they're all upset.

Dave: 10:30

Well, and this is so. There's a great short. I'm sure it was a TikTok video originally, but wherever it is coming from, we'll kind of find it and put it in the notes with Jeff Bezos talking about data and anecdotes, and I felt he had a really great. It's a short conversation and broadly speaking, he said you need data everywhere, you need to look at your data, but trust the anecdotes, yes, and though the anecdotes are one or two data points and so on. But to your point about you know if my call center or my operations IT support function has fantastic numbers, but then you go and talk to the people interacting with that group, you'll see the anecdotes carry a lot more weight.

Peter: 11:13

Yeah, you can either call them clients or victims, depending on you.

Dave: 11:16

Yeah, well, and the bit that I find interesting there is the data is a descriptor. It's always a model. It's describing a model. It's almost like it's a discrete function. It's describing that model and it is only as good as that model is close to reality, if that makes sense, and so whenever I think I'm a huge fan of being having data informed, data driven decisions. I think it's nowadays. There's no excuse not to have that going on.

Dave: 11:50

However, I think you also and I was just talking to a client this week about this I think you also have to very definitely focus on where do you draw the anecdotes from? Yeah, or you sense making these things coming in, because data on its own is definitely not going to get you where you want to go.

Peter: 12:08

There's also another element to that, which is you got to make sure you're measuring the right data and that you're to make the right decisions. And because you can, if you end up with the wrong metrics right, then you could start to drive the wrong behaviors as a consequence of that right. It's the combination of those two different pieces, so it's you've really got to be very, very careful, especially if you're actively using the data to drive behavior right.

Dave: 12:42

Yes, yeah, I find that the sort of other thing, and I started this conversation thinking about the prerequisites you need to be able to have data informed decision making. So we've now touched on a number of things that feed into that, but we've not really touched on the sort of foundations that are needed, and the two that strike me as one is that how do you manage, how do you get past the guts, the intuition based leadership?

Dave: 13:09

Let me think of it that way where people are ignoring the data for and this is part of the headache, right for those their own anecdotes, where they're made up of real about what needs to be done. But there's another side of it, and I think this is a lot harder to figure out, which is the understanding of numbers, or, more precisely, probabilities and statistics. Because I think when you what I always struggle with is data gets brought into a conversation but very little is brought in around uncertainty. You know error bars, probability, whatever you want to think of, but that side of things which you know, is the information that's in that data set actually real or is it in this norm?

Peter: 13:53

Right, You're looking at what's it called statistical process control, where we're looking at common cause and special cause and then standing what is the performance of the system over time versus? Are we looking at anomalous behavior? Is it? If you can't look at data in isolation? We were talking about data needs, you need the anecdotes, but there's also data itself. If you've only got a small number of data points, you can't necessarily predict the performance of the system.

Dave: 14:28

Exactly how much information is in that signal, how much of it is signal, how much of it is noise, and this has to do with how many data points I take and so on. So if I'm doing customer experience, I'm looking at the experience that customer may have on a website or an application we're using. There's a difference between 50 data points and 5,000 data points as an example in terms of the confidence we have can have about the messaging there. But I'm even thinking at the very simple level of if we go into a conversation and somebody says we have this number of whatever it is opportunities, kind of customer experience telling me this, this and this. There's never a conversation about the error.

Peter: 15:09

But there's an element of needing to have enough understanding of statistics and math to be able to really grasp what we're referring to there and like how that actually works and how you determine that. So statistical distributions and like standard deviations and mean and understanding what that actually means, means, and I mean, this is ultimately, though, where Six Sigma and stuff comes up. I mean, that's what that means.

Dave: 15:37

Well, of course, yeah, and, but it's also to your. This is the dirty little secret that you can't talk about, which is you want to be able to come into a meeting and say okay, how many people here pass statistics at university? You've got to have this level of understanding when we talk about my insight.

Peter: 15:52

Nobody wants to admit them. Way too done to understand that.

Dave: 15:58

But I think it translates into how do we turn these into heuristics for making decisions, because actually this is where those rules of thumb come in about. You know whatever it might be, but that and there's there's some just really nice examples If you think about sorting problems or you think about explore, exploit, and there are these heuristics that are beginning to come out in the literature, and I'm thinking specifically here. There's a book around algorithms to live by right and there's some great rules.

Peter: 16:27

Yes, that's a really good book on my shelf behind me. Yes, there's some really interesting pieces that that pulls out. What? One of the issues, the heuristics you bring up there, is the, the rule of thumb. Even though it's a rough approximation, it's the good.

Peter: 16:45

Rules of thumb are rules of thumb for a reason because they are right as often enough as you need them to be in those circumstances. And so, even more, even though you can get more complex algorithms, you might not necessarily capture much more of the distribution. So your rule of thumb is going to be, as it's much easier to calculate, much easier to do and much easier to remember, and so you, so you rely on that. It's like, and that's why those sorts of base rules that everybody knows, it's like we know this behaves like that, so that's why we do it this way. And then these people learn these and so that that's an interesting, interesting sort of observation they've learned Some really interesting pieces that come out of this sort of the AI space as well. Right, like machine learning, more. There's a danger these days that if you say AI, everyone thinks you're talking about chat. You can see a large language model of some kind, whereas that and that changed fast, but I'm thinking more of machine language models, models really, and machine learning models.

Dave: 17:55

How do we summarize our conversation?

Peter: 17:58

Well, I think there's some really interesting pieces there. I think data driven decision making versus number driven decision making. I think was an interesting observation that these are two quite different pieces. There's one used to help facilitate a conversation, another key piece being around when we think about the data that we're getting back. Data on its own may not be enough. Having the context that that data exists in is essential. I think that was another key piece of it. And then the third piece, I think, would be around understanding statistical deviation data, which may be too much for some people, but there is this need. I think the piece we didn't necessarily delve into you asked the question, but I didn't really answer it. I'm not sure either is like how you convince the people who are unconvincible because their gut is always right, like they know better than the data.

Dave: 18:50

Oh, that one's a really interesting one. So, as you were going through those key things, I think the one piece we also talked about, which was the fuel gauge, and the way I've thought about that one is is it being a signal like a Kanban card? Right, it's a signal for me to fill my tank, so the number there isn't actually data so much as it's a signal for me to take action.

Dave: 19:12

And I think, if I think of it in that way. All of a sudden it's no longer. It's not a data driven decision, it's a signal that is telling me to go fill my tank. One of the key things that I've seen work and when we're working with teams, with stakeholders or leaders like that, is short feedback cycles to get results coming in. So what I'm thinking there is. So there was one organization I worked with which was like a grocery chain and they're there. The product team is trying they can see some really obvious things to go and check, but they can't get it past the intake process and you know the leaders making decisions based on strategic view and there's these small things they really wanted to get done, and what we ended up looking at there is how do you get the situation where they're given permission to go make those changes?

Dave: 19:59

and technical debt, you could argue, is a very similar context on delivery teams.

Peter: 20:03

How do?

Dave: 20:03

I get permission to go away and do things and then the key being they, that has to show results fast, it has to be tied to something so they can keep getting that permission.

Peter: 20:14

So that's one way.

Dave: 20:16

you can't necessarily push back on the heuristic intuitive driven decision making, but if you can carve out a little bit of space to explore your data driven decisions and make sure those results are clear and coming back fast.

Peter: 20:30

Yeah, I like that.

Dave: 20:32

Any other ways. I don't know that I'm going to come up with that many just off the top of your head. No, no okay, so that's the one I tend to sit down on. I think the because the other ones really involve going somewhat head to head with a leader who is insisting on following their intuition. And yeah, I mean you can do it as a peer, you can do it in certain ways right, but they're all where you're really at that leadership level. If you're not at that leadership level, it's more difficult.

Peter: 21:00

Yeah, you need to be coming from a position of trust. So the other way of doing it is the find the person that the leader does it listen to. Like. The way I'd describe it is if you try the front door, if that doesn't work because they say no, you try the side window, and if that doesn't work, try the back door and maybe break the window. You have to do it. That's like some way there's a terrible analogy really, but the the concept is find out who he's going to listen to, find out who his friends are and influence them so they can influence that him or her to change the way that they're thinking about this. Cool, I think that was awesome. We could probably wrap that up as all and we can talk again next time for sure. Thanks again, thanks, bye. You've listening to the podcast where your hosts, Peter Maddison and David Sharrock, focus on the art and science of digital agile and DevOps at scale.