Finding Your Engineering Bottleneck: The Hierarchy of Engineering Needs

Transcript

Shane Hastie: Good day folks. This is Shane Hastie for the InfoQ Engineering Culture podcast. Today for a change I'm within a single timezone of our guest, Myles Henaghan. Myles, welcome. Thanks for taking the time to talk to us today.

Myles Henaghan: Thanks, Shane. Good to be here. As you say, in the same timezone.

Shane Hastie: My normal starting point with these conversations is who's Myles?

Introductions [00:47]

Myles Henaghan: So I am the founder of an engineering practice called Wires Uncrossed. We're based in Auckland, New Zealand. Before that I'm from Ireland originally. A lot of us are actually from overseas and then there's a clutch of New Zealanders as well, but I've been in software engineering for just over 20 years now. Actually started out in semiconductor manufacturing and electronics and then arrived in New Zealand 2007 and then was just doing straight software engineering from there on and sort of traversed in between engineering and leadership roles.

I was the CTO of a New Zealand company called Accordo and then did a stint as the general manager for engineering at Xero for one of their portfolios and then other ones in between here and there. And we, all of us, three of us have a similar background, that mix of working in engineering teams and helping to troubleshoot them and form them, but also then a lot of deep technical expertise depending on the different organizations.

Shane Hastie: We are talking today because I've come across your Hierarchy of Engineering Needs. Why do we need another one of those?

Why we need the Hierarchy of Engineering Needs [01:58]

Myles Henaghan: Yes, another thing. Why do we need another thing? That is totally valid, yes. Even the books on our respective shelves are full up. Why do you need another thing? Well, this is, in fairness, it's a retake on an old thing or several old things, but the Hierarchy of Engineering Needs is a model that we use and have been using for the past several years and have decided to open source the core of it or the piece of it that is actually designed to help you figure out which of all of those other things are the best ones for you to use now and why. So at the crux of it, the hierarchy supports engineering leaders or people who have to influence engineering teams and organizations as to what's the best sequence of good ideas to actually work on and why.

Shane Hastie: So what's the best sequence of good ideas to work on and why? And how does a hierarchy help?

How the hierarchy helps guide good decision making [02:57]

Myles Henaghan: Well, you might've heard of Maslow's Hierarchy of Needs, so that's the actual inspiration for the crux of it. The reason you kind of need it is that we all work in these software delivery teams, delivery systems, sociotechnical systems, complex adaptive systems, whatever you want to call them.

Years and years of myself and my colleagues, we would continually have this pattern of people will come and ask things of us as an engineering group or there would be things that would be holding us back. And it was often quite frustrating where somebody would have a totally valid request like, oh, you need to sort out your security or you need to sort out your privacy, or here's a new goal setting mechanism, or we're going to go DevOps or we need to template this, that and the other. But the ability of those good ideas to actually land and the teams to actually have the capacity to take them on was very hit and miss.

And when we really stood back and spoke to what is and debriefed and see where things worked, where you had continuous delivery teams that are able to truly just calmly and confidently deploy changes to production on a daily basis, proactively take care of things like disaster recovery. We've actually just tested our database, rolled over and nobody had to ask them, and they're not freaked out about taking on junior engineers. It's like, fine, yes, we have the space to take them on and nurture them. It's not like we're too busy to help anybody else. When you compare the cases where things go well and things do not go well, the pattern was teams were trying to reach for something that was just at a level of maturity that was not appropriate for where they were at. Often a team would be more pressed or concerned or influenced by something much more fundamental that's upsetting them.

The four levels explained - Basic Needs, Managed Work, Effective Ownership, and Flow [04:46]

And so Maslow kind of gives us a head start on modelling that same decision influencing type problem that happens in teams. So looking on screen there, basic needs is a clutch of things that pretty much any software development operation needs to have in place to some degree, just to discern any sort of a change out the door. Obviously, I won't go through them all, but obviously you need people. Those people need to know what they're there to do. Those people need certain technical capability so that most people on the team are able to do most jobs. You need a backlog and then there's more from the technical realm. Obviously, you need some kind of compute solution, you need some way to make a change on your local machine and so on and so forth.

Now with those things, a team can and does still, you'd be surprised, just deploy changes from their own machine. Maybe it's like an early startup or there's some ecosystem of tooling that just doesn't suit all the shiny stuff, but you can do it. So that's basic needs.

And then there's managed work, is a layer of things that the team needs in place to some degree to actually stand behind that work over time to repeat the quality at a certain standard and to scale it out with more people.

Effective ownership goes on to... Basic needs and managed work is when the team has been asked or needs to actually make a change to the application. But we know that people also actually have to own services or operate services over long periods of time, and that might be just essentially managing a quality of service or managing consistency and expectations to customers or across teams, and that's what the effective ownership and sustainability is about.

And then ultimately the Flow, what we call the flow level represents kind of what everybody wants in the engineering team to do and what the CFO is hoping they're paying for. Is it true that as a software operation we are efficiently delivering things that our users or customers see as valuable, and are we doing that while still maintaining their trust in what we've already shipped them? So flow is where you're looking to get to. The reality is that moments of flow can be quite fleeting if you don't have these other dependencies actually tuned up in place to the appropriate level. They don't need to be sophisticated. What would be good enough for great or green for any of the needs in the hierarchy is kind of relative to the type and size of the organization, but there is this dependency.

And equally, if you have a team that's doing well, they can be brought to their knees quite quickly if something starts to creep lower down. So even if the team has been running for quite some time, you'll know yourself and we see it every day, if the build system starts to flake out or become much less reliable than it used to be, that will quite quickly dominate everybody in the team's attention, let alone if you just sort of switch off the WiFi or something, or if a key person leaves.

And so the hierarchy plus Maslow does a good job of this stage of understanding the pushing and pulling that happens amongst things, and the consequences of one thing not being at the appropriate level of maturity for another. And then there's another layer of how we use it and how we hope other people can get value out of it, which moves more into the theory of constraints or finding of all of these things, there's too many things, there are always too many things to do. They will all be always a little bit broken, but can we find or stack rank the top five things that are most impactful for us?

Shane Hastie: So that touches on one of the topics that you and I were chatting before we started recording, the systems thinking and the need for that systems thinking and theory of constraints view. You made a statement that most managers are not good at that at the moment.

The Systems Thinking gap - most managers struggle with systematic approaches [08:53]

Myles Henaghan: I made a statement that we believe that the biggest opportunity to uplifting software delivery is getting a base level of systems thinking for as many engineering leaders, managers or key people who have to influence teams, we find that it is quite subjective as to how sophisticated or systematic people are in terms of how they go about their work.

Ultimately, they need a strategy, but if you say that word, people respond differently to it, but there's no other sort of replacement to it at the crux of it. And so, the base level of systems thinking and the Hierarchy of Engineering Needs is sort of designed and intended towards that approach. How can we help put a groundwork in place to actually feed into thinking about your delivery system systematically? That it's not just the people, people are important. There's all the other bits and bobs that pull and tug in it, but then also apply what it is that the organization needs of the delivery system.

And we believe that, yes, if you can get a base level of systems thinking, which myself and others have been privileged to collect over the years, but it took us 20-odd years and a lot of failure and heartache and success and highlights across the way, we believe we can help people avoid as much as possible the angst and delays and misses that we've had to do. And we think that's quite a meaningful thing because you'll know yourself, particularly in the software industry, a lot of really good ideas falter or don't actually see the light or don't get the chance that they could have or should have just because of happenstance of the people and the context and the experience that that opportunity or project was able to actually attract at that moment in time.

Shane Hastie: So let's dig deeper into that. What does systems thinking really mean to us today from the context of that engineering leadership?

Understanding sociotechnical systems and theory of constraints [11:08]

Myles Henaghan: Okay, from the context of engineering leadership, and I'll try not to get too academic on it. The way we look at it is you've got an engineering leader, a leader of engineering or a principal engineer, architect, whatever it might be. They are in a sociotechnical system. People, process, technology, goals, metrics, culture and all the rest of it and all those things are pulling and tugging at each other on a daily basis. And a sociotechnical, we just see it as kind of like a subset of a complex adaptive system and that's that daily pulling and tugging. So, the infuriating thing is you have all of these models or things including our own that try to represent these systems, but you can never really fully represent them and not really understand them for more than a moment in time. However, that's not an excuse not to do it.

From an engineering leader, these teams sometimes, it's quite easy to burn north of a million dollars a year on an engineering team. And then if you sort of multiply that out, these are very big investments and very important investments for their organizations. And so it's dangerous to apply certain techniques to an engineering team, but it's irresponsible not to apply systematic thinking to the wider system that they're actually operating in. There's just too much money and business consequence for the organization at stake.

So what we want to do with the hierarchy is just put a layer of detail further on down to make it more accessible to people. But at the end of the day, what we found most effective is follow the theory of constraints when you're dealing with one of these systems, and that's how we use the model. And we believe other people if they squint, they can do it in a similar way without a bunch of the other tools that we use for more complex or larger situations with our clients.

But of all of those 40 things, at any moment in time, one and only one is the most thing that's limiting the effectiveness of the delivery system, be it productivity or reliability or resilience or team satisfaction, whatever you're optimizing for. Really it's about throughput, quality and resilience. Really everything comes down to those three. One of those is what you need most of and at any one time, there is one thing above all else that is the most limiting. And if the engineering leader is not investing something in that most limiting constraint to either get rid of it or to rally your resources, it's something that can't be tested. It could be regression testing or something like that.

So, well then the whole system is limited by the ability it takes to regression test a monolith or something like this. Then okay, accept that and you need to rally as much resources on making it easier to regression test it or optimizing the upstream stuff so that you're not just fictitiously thinking that we're doing a lot more activity before that constraint, but the work just builds up invisibly and delivery pressure comes on, and then you have to sacrifice the ability to regression test it properly, and then it goes out the door and bad things happen and this, that and the other. And so it's to help people see the chain of consequences of those things.

From a software engineering leader, how do you get to a point where it's easy for them to know, one, but more importantly the rest of their team internally and their colleagues also agree with the thinking of what are the top five things or three things. And if they have that, what we found then the other problem gets a lot easier for the engineering leader, is if you know what your main constraints are and why, then you turn around to all of the good ideas that everybody wants you to do. So, your own internal engineers, if you ask your most trusted engineer, "You've got a week, work on whatever you want". The things that they would choose to work on would be subjective to how much awareness or context or shared alignment they have on what's the most important problem for us.

This process of using the hierarchy helps embed that in people so they have that same context going into it. You have your compliance people or your governance people, so somebody's coming in to tell you to save a lot of money on cloud costs, another person is saying, you got to automate your security controls or this thing has to get patched. Another person is saying, you got to have a data dictionary for GDP or alignment, yada, yada, yada.

But once you have your key constraints, you can put all of those more specific technical initiatives, process initiatives or people or ways of working, then it's a lot easier to stack rank them because each of those things would have a stronger or weaker contribution to your key constraints. And then you can show and communicate and influence everybody, "Here's our top five, everything else, we're saying yes to these few things and no to everything else, but here's why".

And we find they actually respect the homework and they can see that, okay, the answer's maybe not what I want, maybe my thing's not in the top 10, but I see why you're doing it and I can move on. Once you get to a moment of understanding for the engineering leader, that other bit for both them and their peers, having a way to reason with priorities is the second most helpful. And the third is measurement, but that's another whole area.

Shane Hastie: You make the point, there are 40 items in this hierarchy. One of them is going to be the biggest bottleneck. How do I figure it out?

Finding the biggest bottleneck, identify the most limiting constraint [16:57]

Myles Henaghan: You got to do an assessment of them, and that's kind of what we do in my day job. We believe that people can get some of the value on their own if we give them enough pointers on it, but we might also open source the other ways of doing it or figure out ways to do it. But there's three stages, Shane. Yes, you've got 40 things. For those 40 things. There is a clutch of qualitative and quantitative things you can use to figure out where you're at. It might be a particular type of metric, it might be an opinionated survey or a self-rating. I'll give you sort of an example of some of them. And it might be you're asking your subject matter experts in the organization. So here's the thing, so naming things and caching. So we've had to name things. There is a bit of a, people have to go and look at it as to what it is.

But if I would take local dev and IDE for example, we give people as a prompt in each of them if you click into it or you go into the particular need, the description of what's the scope of that particular need. And we also give a kind of a hint as to what might be true if this thing is going well, like a high performance indicator. We call them a signal or a prompt or a question is like strongly disagree, strongly agree that the IDE setup is consistent for new team members and agreed coding conventions are codified in doc files and this, that and the other. We're going to have to probably extend this a bit in the next release to talk more about guardrails for copilots, LLMs and this, that and the other. But if your local dev is going well, people can get up and running within so many hours of a new starter joining.

There's also metrics, you can get ad hoc metrics, you can get surveys. If you take deployment solutions, for example, deployment solutions is build pipelines and everything else that might go in between with it. Things like the DORA metrics are very helpful there, but you can't rely on them on their own.

So, 40 things, some of them you will just switch off right away. For our context, that's not applicable. We don't need it. Chaos engineering perhaps, that's not going to come up with us anytime soon. For the ones that are important, you do the rating qualitative, quantitative, and you be harsh, you be critical and you ask several people and ideally you ask people that think differently to yourself. So, you would get the rating or get the opinion of somebody who's close to it and somebody who's like a customer of it kind of thing.

And then the truth will be somewhere in between, broadly right, precisely wrong as they say in one of the Wardley mapping things. But it doesn't have to be accurate, it just has to feel right. And we will give you nudges or prompts as to if you think you're being a bit too subjective. You got your Christmas tree, imagine green is, yes, it's okay for us, amber, we have something, but it could be better, or red that we don't have that at all as a formal capability, or we have it, but it's just totally, we've totally outgrown the capabilities that it's at.

The next bit is we have to ask what the organization needs, or more importantly, the engineering leader needs to do something that sounds simple but is hard, and they need to decide what's more important for them, like throughput and quality versus resilience and governance versus product impact versus team experience. So all of those things will clash with each other and it's like pick any two, the truth will be somewhere, the engineering leader needs to know that. And sometimes that takes a while. They might start on this and put a certain percentage on throughput and quality and another percentage on team experience, but in reality they'll find out after a while that actually maybe they need to rejig the loading.

So, then you have the two hardest bits done, for all the needs you got what maturity that they're at. Then you have for the system dimensions of throughput, resilience, this, that and the other, you know which ones are more important. And with those two, then you can surface up if this is each need, Shane, has got a different, we put a different weight in on it. We haven't opened up those yet, but each need contributes more or less to say, sending work out the door versus making your systems more fault tolerant versus team experience or standardization of things.

And then that will give you your top five. So, of all of things on the Christmas tree, there'll be 40 items. If really what you're after is resilience or standardization. Standardization sometimes is a big thing we're having to sort of offshore or co-deliver with another organization. Then actually these ones are way more impactful, because they're lower maturity and they're much more contributing to that. I've done a lot of hand waving, but that's the general gist as to how it goes.

Shane Hastie: So really got to think hard about this.

Myles Henaghan: Yes, what we're trying to figure out, Shane, is we can open source more or make it easier, but we don't know where the balance is because at the end of the day, if this thing works, it's like a thought-provoking or it's a way to help structure your thinking about it. And that's actually the outcome that we want to get to. Yes, we're keen for feedback on what level is useful versus just, I don't know what to do with it.

Shane Hastie: As an engineering leader in an organization, at whatever level I'm at, what can I influence that's going to make us better?

What engineering leaders can actually change and control [22:47]

Myles Henaghan: Usually more than you'd think. So, I guess that the engineering leader, fundamentally I believe that their job is to put a big neon sign on what's the best problem to work on and why. And if they're not in a position to move that rock themselves, then their job becomes to lobby everybody else in the organization to agree to fix it with them or for them. And so why we do this kind of work and why we've open sourced the model on that is a large part of those jobs is communication, and so they're holding context and holding continuity as to why something is important. Because the organizations, they get impatient, don't they? People come and go, goals change, budget changes, big customers, small customers, sales event, this, that and the other, but the problems don't go away underneath usually. Often they just kind of fester.

The enduring thing about the leader is regardless of if they're just starting out and they're tussling with the, "Oh, I can't do everything myself anymore. I've got to find a way to deliver with or through others". Or the person who's sort of time poor and then be bombarded by everything, including other executives telling them how to do their job.

The hardest part is to have a snappy common language or framing to bring everybody back to again and again as to we did our homework and when we did the homework, we felt this was our main problems to work on. Sometimes it's changes. Maybe you acquire a company or there's a big change that's happened and it goes, okay, let's do a rethink. Let's do a refresh on it. But we believe what can they influence? A lot more than they think, just never as quickly as they would like it to happen.

The main thing is if the leader can be consistent in what's the best problem to work on and why, and if they demonstrate that way of working about how to find that problem, then they also influence all of the other people in the organization or subject matter experts that need to have some ability to think the same way.

Shane Hastie: What's the important question I haven't asked you?

Myles Henaghan: Why don't people do this anyways? Why wouldn't it work? Who would it not work with or where would it not work?

Shane Hastie: So let's dig into those.

Preconditions and limitations of the framework [25:14]

Myles Henaghan: I think there's a precondition, if this is going to work and it'll fit some people and not others, it kind of has to land with people who we talked about systems thinking, but that is kind of in the realm that it sits. So this will work best with somebody who is not the heroic type person that I'm going to engineer my way out of this, I'm going to code my way out of it, or I've seen this before and I'll copy and paste my solution.

That's at the heroic end. And at the other end is more like the idealist person who has quite up to speed with a lot of theories and practices across people and process and technology, but they haven't had the ability to actually road test them for themselves and their organization. So we look for people in the middle that whether or not they say these words, but they aspire to take in a systematic approach to managing, improving, optimizing the delivery effectiveness, delivery and operational performance effectiveness.

It won't work if somebody is too busy to slow down to speed up. Because often what will come out with this, particularly if it's sort of a low performance or a challenging type organization situation, often what this will do is put a big flashing light and a rationale as to why quite a large problem that's been tiptoed around or hiding in plain sight for a long time needs to get sorted out.

I mentioned the regression testing of the monolith, that's quite a common one. Your main constraint, even if you're planning to get off this thing, your main constraint is the ability to run changes confidently through that thing. Because even if you want to shrink it down, that will aggravate your problem even more.

And so if somebody is looking for a quick fix, there will always be a couple of quick fixes, but if the person is not ready or the organization is not ready to actually start biting into a long overlooked or an oversized problem, rightly or wrongly, it's not necessarily for me to say, but they're probably in that local optimization mode where they're doing things as an engineering leader, which are in the effort of improving stuff, but they're just kind of doing what they can do instead of what they should do.

Often the org chart is the problem. So if you get people who are not in a position to sort of delete the org chart or delete the technology choices, that can also be a challenge.

Glass half full, if there is one of those things that just literally, this is not going to go anywhere because this team over here is highly dependent on that thing and they have no interest in making that thing easier to actually work on. But then at least it is a vehicle to actually escalate. And so in those situations, maybe the person who's using it, they need to take a step back and go maybe there's somebody else in the organization that I need to use this to provoke a conversation. It won't work for quick fixes and won't magic away or silver bullet something that is quite large.

Shane Hastie: Quite a lot to explore here. We'll include the link to the hierarchy in the show notes. If people want to continue the conversation, where do they find you?

Myles Henaghan: Oh, you know the way it is, Shane, these days, best thing is LinkedIn, I think.

They can also ping us on GitHub, so there's a link on the website as to where that goes. We're really keen for feedback on it. As I say, we're trying to get that balance as to what might be useful for people independently versus what's the effort of us maintaining it publicly. So keen for feedback, either at a coding level, just go and send a change or raise an issue, or if you just like this type of conversation or having a discussion or help us create content as to how to apply it or make sense of it, keen to hear all of that.

Shane Hastie: Wonderful. Myles, thanks very much for taking the time to talk to us.

Myles Henaghan: Likewise. It's been a pleasure, Shane. Thank you very much.

Mentioned:

• Hierarchy of Engineering Needs
• Myles Henaghan on LinkedIn