Bottom-up Adoption Through the Prism of Flow
This is (a transcript of a rehearsal of) my talk for Agile Yorkshire 14th October 2014. It goes with the slides here.
By Bottom-up Adoption I’m talking about a team that’s decided to become agile or some local lower-level managers has said “let’s do agile” but the organization doesn’t necessarily know what any of this is. Flow is a concept which has come from a book by Donald Reinertsen and it’s very hip a the moment; lots of people are recommending you should read it. It’s a heavy book and lots of the people who recommend you read it haven’t read it themselves. I would recommend reading it about five times.
Okay, so a little bit about me: I’ve been in the same team project for about seven years and we started doing Scrum, and I was nominated Scrum Master. We’ve tried lots of adjustments to how we work but so far we haven’t had any really good success stories. Without these it’s hard to get management or upstream and downstream partners to agree to behave differently to help us out—to optimize the flow of the whole chain. So I’ve been forced to keep improving my understanding of how this stuff works so I can try to make more persuasive arguments.
So Flow, the book by Donald Reinertsen, gives you a rational underpinning to agile practices, to product development practices. It’s very much the zeitgeist right now. There is not enough time to do justice to the topic, so I am going to fly through some of the concepts of flow and really, I expect you just to get some of the keywords you go away and read about yourself and if you need some of this stuff I do recommend you read the book.
So here goes the Quick and Inadequate explanation of the concepts of flow.
Variability is how the length, time, complexity of your tasks vary. It’s generally bad because it makes it harder to predict what will happen by your deadline. Scrum, and Kanban, which comes from a lean manufacturing background, try to eliminate variability. With flow, there are techniques to acknowledge variability, and make better decisions in the light of that.
What is a queue? The thing about queue lengths is, once you start knowledge work, it ages. Technology and the market move on. It’s better to actually delay starting, than to commit and then delay. If you delay starting then you can use more recent knowledge and avoid going down a blind alley. Queues are internal queues after you start the work, maybe something’s waiting for the testing department or so on.
Queues delay work, but they delay it for all the items in the queue so the cost is worse than linear in the length of the queue.
This curve has a knee when you are in the face of variability. When you get over about 80% utilization your queues start to get longer. Not only that, but if you experience some variability, when utilization is near 90%, a small increase gives you a much longer queue. And as we already know, a much longer queue means much much higher costs.
Batch Size is the concept of how much stuff must you do before it’s “done”. So if you need to do this entire international standard before your customer will review it, then you deny yourself feedback and correction of assumptions.
In the graph, we see effort going in, and the shaded areas are work that is done but not shipped. In large batches, we lose opportunities to charge for, and/or learn from, our work because it’s not in the field. With smaller batches we allow more opportunities for both, and your returns earlier, and good compounding effects.
The other interesting thing about batch size, is that longer projects overrun by more. Not just more in absolute time, but by a greater proportion of the original estimate. So the percentage overrun increases as intended duration increases.
If you’ve got more Work In Progress (WIP) then your utilisation gets pushed higher, which as we know increases your chances of queues, and your queues increase your delay. So limiting WIP is an approach to keeping your queues down.
Yes, in terms of customer feedback, but also feedback like in any control system. So for example you might have a queue that’s getting to a certain length that you’ve decided is too much. You might use this to signal your upstream partners to slow down a little there.
This is known in agile anyway, people say “handoffs are bad, m’kay?”. But there’s this fun diagram in the book where we see a piece of work which is handed off from one stage to the next stage, and each stage has the power to send the item back to the start. The work took a typical 40 days to get through the process. When changed to a regular, cross-functional review, work would make it through the process in 5 days.
So how does Scrum relate to these various topics?
I’m going to run through the elements of ideal scrum, looking at them with our new Flow-based lens. And in italics, I’ve added some of the things that BDOs (Big Dumb Organisations) do that break agile and flow. Remember that we’re adopting agile practices from the bottom up so we don’t necessarily have the influence to change these behaviours.
I’m not going to count this as a queue, because we try to ignore most of the backlog. If you’re doing ideal Scrum, then you’re only estimating the top of the backlog, and you’re only slicing and designing and planning the top of the backlog. And the rest of the backlog can be cheaply switched out and changed as someone changes their priority.
Unless of course, someone has committed to deliver the whole backlog. Then it’s a queue.
We’re reducing batch size, which is good. Batch size causes overruns, which delays feedback. So here we have less variability and more timely feedback. There’s also the concept of variability pooling. The variability of a series of small unconnected items ends up being less than the variability of one large item.
It’s important to remember that the decomposition here is on the level of the behavior of the product. These are the stories that add value to the customer and we say what is the smallest way we can change their life for the better?
In BDO’s it’s common to have already broken down a big feature into lots of tasks that need to be done, and present that to the scrum team.
This is a limit on Batch Size again. If your slicing is working, so you have items small enough to achieve in a sprint; and if you do actually refuse to take in items that are more than half a sprint.
So this can be undermined if a manager keeps negotiating up the sprint commitment so your batch size is going up and your chances of being unpredictable are higher.
It’s also a loose limit on WIP. So the team commits to what they think they can achieve in the timebox.
Again, managers will try to negotiate up the commitment.
Also, if you have to answer support calls during the sprint, your WIP is not really limited.
Sprint Backlog is a queue. So the more stuff you’ve committed to in your Sprint Backlog, the more you’re delaying the completion of that work. The more you risk that you’re now building something that will be out-of-date. So that’s a reason why you keep your sprints short.
“Todo” and “Doing” are queues. “Doing” is WIP. I actually draw a frowny face on the “doing” column of our task board because that’s the riskiest part. It’s where we’re spending all the money and creating uncertainty. Ideal scrum uses the “snowplough” pattern, the pattern on the right, where you focus on finishing stuff before starting other stuff and keep less stuff in progress.
This is undermined by managers going “I just wanna see some progress on topic X, can somebody just start it?” We have experiences where our team’s collaboration gets spoiled by members being assigned to various pet projects.
We’re limiting variability: a good observation in the Flow book is, if you review on a fixed time basis then the reviews always happen. If you review based on completed scope, then the projects in trouble get reviewed less.
This can be undermined by customers saying “we’re not going to review it until it’s complete; don’t waste our time with this half-finished thing”
This allows redeployment of resources to bottlenecks, in flow terms. It’s a cadence on which the whole team plans how to get the stuff done.
A manager can mess this up by “making sure everyone has a job to do”. By increasing utilization until we lose the capacity to handle variability, and start to increase our queues per the J-shaped graph we saw before.
Also it’s a synchronization across functions, so it’s getting rid of the to-and-fro 40-day diagram we saw before.
Unless of course your team is not cross-functional, or the PO is not engaged.
So you demo the behaviour and get fast feedback.
Unless the customer and PO are not present. Or if your customer is remote then you end up with handoffs between the feature team and the customer, delaying feedback.
Shields the team from additional WIP, to try to keep the utilization steady.
Unless it’s just a dev with a baseball cap on and the managers think they’re being stubborn and trying to resist doing any work.
Scrummaster nurtures the adoption of good practice. Which includes “optimize the whole system”.
Unless of course scrum is regarded as “Something that teams do”.
In ideal scrum, colocated, cross-functional, self-organizing team gives you fast feedback, e.g. between architect, tester, dev and tech author, effectively eliminating queues between those disciplines: there are very few questions still awaiting answers.
Additionally, a cross-functional, self-organising team can reallocate what people are doing to get things through bottlenecks.
Unless of course, your organization doesn’t get this and says “use these guys in India to do your testing, they’re cheap!” or you have a manager who says “you’ve got 7 people in the team, you ought to be doing 7 things.” you don’t have a team then. You have 7 soloists sitting near one another.
So that’s a run through Scrum, but what is missing from this picture?
One of the key things that people talk about in agile is suboptimisation. Indeed, Flow says it only really makes sense to optimise the whole system. If you have a high-throughput component, like your dev team, in the middle of a chain that’s running much slower, like a 6-month release cadence or an 8-week quality process, then you’re not really going to get the value out of scrum, but you are going to get the costs of running around making all these internal releases. Feedback is very valuable but if you’re not getting the feedback, then what is the value of a release cadence?
If you’re doing “Agile in a bubble” — if the company is not paying attention to batch size, if people are saying “implement the whole of this international standard and don’t show me until it’s done” then it doesn’t matterr how agile you’re being inside the team; if nobody’s going to help you with more information about how you’re doing, and steer you, then you’re not iterating, you’re just oscillating.
So we do need a culture change if we’re in a BDO. There are significant harmful behaviours that traditional organisations suffer from. For example, when things don’t deliver the right functionality, the instinct is to work harder to perfect the specification.
Or to insert stage gates. Reinertsen comments that, in his informal surveys taken at talks, about 95% of organisations that have stage gate processes also have a totally other, hidden process that actually delivers the product.
Another harmful behavior might be to make sure everyone’s busy. And we’ve seen how that affects the queue.
Can you address all these culture issues from the bottom up? From within the development team?
To the nearest whole answer: No.
You need luck and the ear of a sympathetic manager. You can’t do this from the bottom.
I started to look at what I need to get in place to get Flow working, and realised I could frame that itself as a Flow problem. And there are quite a few problems.
So I talk to my manager, who probably understands about 30% of what I’m trying to do; he talks to his manager who probably gets 30% of what he’s saying, and he talks to the next manager, who has a completely different set of concerns.
Maybe a “cross-functional synchronization” approach can work going up the organization chart? Try getting more layers of management together to present the ideas and meet criticisms efficiently. But if your organization is deep enough, I’m not sure this can be overcome.
If you have a tall organization then you might need to get the ear of someone who has a completely different set of concerns than you do. Their boss’s boss’s boss’s bonus is based on something completely different to your boss’s boss’s boss. You then have an overhead of figuring out what all the people of that division are worried about, and trying to sell your idea as how it’s going to help them too.
It would be great if you could just hand out the book and say “everybody read this book!” But itself it suffers from a large batch size. Though it was recommended to me by someone I respect greatly, it took me a year to get around to starting it, and months to finish. (But it’s well worth it and I will be reading it several times)
Batch Size of Culture Change
What is the batch size of culture change?
Ask yourself “how many elements need to be in place for success?”.
The Scrum literature does contain most of what you need to make Scrum a success, but does miss the cultural aspects.
Ask yourself if you can get better results than now using a few elements of what you know?
(By the way, I should say that visualising the work and reducing your batch sizes should be number 1. I assume you’re already doing that with a backlog and task board.)
Working within suboptimization?
Can we work with suboptimization? What if we say ok, we’re not going to try to convert the organisation, we’re going to try to work within it. You’ve got to recast your situation then, because “lifetime profitability” is not something you can control any more. So maybe go with a campaign of small victories.
Highlight some individual problem that your upstream or downstream partners suffer from, and fix it with some “agile stuff”. If you show successes then you will get the ear of management because they want to know what the special sauce is so they can spread it on everyone else.
But make sure you go into this with your eyes open because if you don’t understand the underlying mechanics, then what works for you might not work for the next project or another team. I think Flow will give you the underpinning to be able to look at a project and assess it, and say “actually, the problem here is not the same as the problem on our team before we were so successful after we fixed it.”
Even armed with all this, your success will depend on your company’s and your customers’ culture.
Go read the book!
Beware of large batches,
Watch your Queues (no, actually watch them)
Keep utilization surprisingly low, 70%-80% busy
Look for small success stories
If your management aren’t getting it, you’ll need lots of luck.