software- and people-guy

Bottom-up Adoption Through the Prism of Flow

This is (a transcript of a rehearsal of) my talk for Agile Yorkshire 14th October 2014. It goes with the slides here.

By Bottom-up Adoption I’m talking about a team that’s decided to become agile or some local lower-level managers has said “let’s do agile” but the organization doesn’t necessarily know what any of this is.  Flow is a concept which has come from a book by Donald Reinertsen and it’s very hip a the moment; lots of people are recommending you should read it. It’s a heavy book and lots of the people who recommend you read it haven’t read it themselves. I would recommend reading it about five times.

Okay, so a little bit about me: I’ve been in the same team project for about seven years and we started doing Scrum, and I was nominated Scrum Master. We’ve tried lots of adjustments to how we work but so far we haven’t had any really good success stories.  Without these it’s hard to get management or upstream and downstream partners to agree to behave differently to help us out—to optimize the flow of the whole chain.  So I’ve been forced to keep improving my understanding of how this stuff works so I can try to make more persuasive arguments.

So Flow, the book by Donald Reinertsen, gives you a rational underpinning to agile practices, to product development practices.  It’s very much the zeitgeist right now.  There is not enough time to do justice to the topic, so I am going to fly through some of the concepts of flow and really,  I expect you just to get some of the keywords you go away and read about yourself and if you need some of this stuff I do recommend you read the book.

So here goes the Quick and Inadequate explanation of the concepts of flow.



Variability is how the length, time, complexity of your tasks vary.  It’s generally bad because it makes it harder to predict what will happen by your deadline.  Scrum, and Kanban, which comes from a lean manufacturing background, try to eliminate variability. With flow, there are techniques to acknowledge variability, and make better decisions in the light of that.

Queue Lengths

What is a queue? The thing about queue lengths is, once you start knowledge work, it ages. Technology and the market move on. It’s better to actually delay starting, than to commit and then delay. If you delay starting then you can use more recent knowledge and avoid going down a blind alley.  Queues are internal queues after you start the work, maybe something’s waiting for the testing department or so on.

Queues delay work, but they delay it for all the items in the queue so the cost is worse than linear in the length of the queue.

Capacity Utilization

This curve has a knee when you are in the face of variability. When you get over about 80% utilization your queues start to get longer. Not only that, but if you experience some variability, when utilization is near 90%, a small increase gives you a much longer queue.  And as we already know, a much longer queue means much much higher costs.

Batch size

Batch Size is the concept of how much stuff must you do before it’s “done”.  So if you need to do this entire international standard before your customer will review it, then you deny yourself feedback and correction of assumptions.

In the graph, we see effort going in, and the shaded areas are work that is done but not shipped. In large batches, we lose opportunities to charge for, and/or learn from, our work because it’s not in the field.  With smaller batches we allow more opportunities for both, and your returns earlier, and good compounding effects.

The other interesting thing about batch size, is that longer projects overrun by more. Not just more in absolute time, but by a greater proportion of the original estimate. So the percentage overrun increases as intended duration increases.


If you’ve got more Work In Progress (WIP) then your utilisation gets pushed higher, which as we know increases your chances of queues, and your queues increase your delay. So limiting WIP is an approach to keeping your queues down.


Yes, in terms of customer feedback, but also feedback like in any control system.  So for example you might have a queue that’s getting to a certain length that you’ve decided is too much. You might use this to signal your upstream partners to slow down a little there.

Cross-functional Synchronisation

This is known in agile anyway, people say “handoffs are bad, m’kay?”. But there’s this fun diagram in the book where we see a piece of work which is handed off from one stage to the next stage, and each stage has the power to send the item back to the start.  The work took a typical 40 days to get through the process. When changed to a regular, cross-functional review, work would make it through the process in 5 days.


So how does Scrum relate to these various topics?

I’m going to run through the elements of ideal scrum, looking at them with our new Flow-based lens. And in italics, I’ve added some of the things that BDOs (Big Dumb Organisations) do that break agile and flow. Remember that we’re adopting agile practices from the bottom up so we don’t necessarily have the influence to change these behaviours.

Product Backlog

I’m not going to count this as a queue, because we try to ignore most of the backlog. If you’re doing ideal Scrum, then you’re only estimating the top of the backlog, and you’re only slicing and designing and planning the top of the backlog. And the rest of the backlog can be cheaply switched out and changed as someone changes their priority.

Unless of course, someone has committed to deliver the whole backlog. Then it’s a queue.

Story Slicing

We’re reducing batch size, which is good. Batch size causes overruns, which delays feedback. So here we have less variability and more timely feedback. There’s also the concept of variability pooling. The variability of a series of small unconnected items ends up being less than the variability of one large item.

It’s important to remember that the decomposition here is on the level of the behavior of the product. These are the stories that add value to the customer and we say what is the smallest way we can change their life for the better?

In BDO’s it’s common to have already broken down a big feature into lots of tasks that need to be done, and present that to the scrum team.

Sprint Backlog

This is a limit on Batch Size again. If your slicing is working, so you have items small enough to achieve in a sprint; and if you do actually refuse to take in items that are more than half a sprint.

So this can be undermined if a manager keeps negotiating up the sprint commitment so your batch size is going up and your chances of being unpredictable are higher.

It’s also a loose limit on WIP. So the team commits to what they think they can achieve in the timebox.

Again, managers will try to negotiate up the commitment.

Also, if you have to answer support calls during the sprint, your WIP is not really limited.

Sprint Backlog is a queue. So the more stuff you’ve committed to in your Sprint Backlog, the more you’re delaying the completion of that work. The more you risk that you’re now building something that will be out-of-date. So that’s a reason why you keep your sprints short.

Task Board

“Todo” and “Doing” are queues. “Doing” is WIP. I actually draw a frowny face on the “doing” column of our task board because that’s the riskiest part. It’s where we’re spending all the money and creating uncertainty. Ideal scrum uses the “snowplough” pattern, the pattern on the right, where you focus on finishing stuff before starting other stuff and keep less stuff in progress.

This is undermined by managers going “I just wanna see some progress on topic X, can somebody just start it?” We have experiences where our team’s collaboration gets spoiled by members being assigned to various pet projects.


We’re limiting variability: a good observation in the Flow book is, if you review on a fixed time basis then the reviews always happen. If you review based on completed scope, then the projects in trouble get reviewed less.

This can be undermined by customers saying “we’re not going to review it until it’s complete; don’t waste our time with this half-finished thing”


This allows redeployment of resources to bottlenecks, in flow terms. It’s a cadence on which the whole team plans how to get the stuff done.

A manager can mess this up by “making sure everyone has a job to do”. By increasing utilization until we lose the capacity to handle variability, and start to increase our queues per the J-shaped graph we saw before.

Also it’s a synchronization across functions, so it’s getting rid of the to-and-fro 40-day diagram we saw before.

Unless of course your team is not cross-functional, or the PO is not engaged.

Sprint Review

So you demo the behaviour and get fast feedback.

Unless the customer and PO are not present. Or if your customer is remote then you end up with handoffs between the feature team and the customer, delaying feedback.

Scrum master

Shields the team from additional WIP, to try to keep the utilization steady.

Unless it’s just a dev with a baseball cap on and the managers think they’re being stubborn and trying to resist doing any work.

Scrummaster nurtures the adoption of good practice. Which includes “optimize the whole system”.

Unless of course scrum is regarded as “Something that teams do”.

The Team

In ideal scrum, colocated, cross-functional, self-organizing team gives you fast feedback, e.g. between architect, tester, dev and tech author, effectively eliminating queues between those disciplines: there are very few questions still awaiting answers.

Additionally, a cross-functional, self-organising team can reallocate what people are doing to get things through bottlenecks.

Unless of course, your organization doesn’t get this and says “use these guys in India to do your testing, they’re cheap!” or you have a manager who says “you’ve got 7 people in the team, you ought to be doing 7 things.” you don’t have a team then. You have 7 soloists sitting near one another.


So that’s a run through Scrum, but what is missing from this picture?

One of the key things that people talk about in agile is suboptimisation.  Indeed, Flow says it only really makes sense to optimise the whole system. If you have a high-throughput component, like your dev team, in the middle of a chain that’s running much slower, like a 6-month release cadence or an 8-week quality process, then you’re not really going to get the value out of scrum, but you are going to get the costs of running around making all these internal releases. Feedback is very valuable but if you’re not getting the feedback, then what is the value of a release cadence?

If you’re doing “Agile in a bubble” — if the company is not paying attention to batch size, if people are saying “implement the whole of this international standard and don’t show me until it’s done” then it doesn’t matterr how agile you’re being inside the team; if nobody’s going to help you with more information about how you’re doing, and steer you, then you’re not iterating, you’re just oscillating.

Culture Change

So we do need a culture change if we’re in a BDO. There are significant harmful behaviours that traditional organisations suffer from. For example, when things don’t deliver the right functionality, the instinct is to work harder to perfect the specification.

Or to insert stage gates. Reinertsen comments that, in his informal surveys taken at talks, about 95% of organisations that have stage gate processes also have a totally other, hidden process that actually delivers the product.

Another harmful behavior might be to make sure everyone’s busy. And we’ve seen how that affects the queue.

Can you address all these culture issues from the bottom up? From within the development team?

To the nearest whole answer: No.

You need luck and the ear of a sympathetic manager. You can’t do this from the bottom.

I started to look at what I need to get in place to get Flow working, and realised I could frame that itself as a Flow problem. And there are quite a few problems.

So I talk to my manager, who probably understands about 30% of what I’m trying to do; he talks to his manager who probably gets 30% of what he’s saying, and he talks to the next manager, who has a completely different set of concerns.

Maybe a “cross-functional synchronization” approach can work going up the organization chart? Try getting more layers of management together to present the ideas and meet criticisms efficiently.  But if your organization is deep enough, I’m not sure this can be overcome.

If you have a tall organization then you might need to get the ear of someone who has a completely different set of concerns than you do. Their boss’s boss’s boss’s bonus is based on something completely different to your boss’s boss’s boss. You then have an overhead of figuring out what all the people of that division are worried about, and trying to sell your idea as how it’s going to help them too.

It would be great if you could just hand out the book and say “everybody read this book!” But itself it suffers from a large batch size. Though it was recommended to me by someone I respect greatly, it took me a year to get around to starting it, and months to finish. (But it’s well worth it and I will be reading it several times)

Batch Size of Culture Change

What is the batch size of culture change?

Ask yourself “how many elements need to be in place for success?”.

The Scrum literature does contain most of what you need to make Scrum a success, but does miss the cultural aspects.

Ask yourself if you can get better results than now using a few elements of what you know?

(By the way, I should say that visualising the work and reducing your batch sizes should be number 1. I assume you’re already doing that with a backlog and task board.)

Working within suboptimization?

Can we work with suboptimization? What if we say ok, we’re not going to try to convert the organisation, we’re going to try to work within it.  You’ve got to recast your situation then, because “lifetime profitability” is not something you can control any more. So maybe go with a campaign of small victories.

Highlight some individual problem that your upstream or downstream partners suffer from, and fix it with some “agile stuff”.  If you show successes then you will get the ear of management because they want to know what the special sauce is so they can spread it on everyone else.

But make sure you go into this with your eyes open because if you don’t understand the underlying mechanics, then what works for you might not work for the next project or another team. I think Flow will give you the underpinning to be able to look at a project and assess it, and say “actually, the problem here is not the same as the problem on our team before we were so successful after we fixed it.”

Even armed with all this, your success will depend on your company’s and your customers’ culture.

Take Aways

Go read the book!

Beware of large batches,

Watch your Queues (no, actually watch them)

Keep utilization surprisingly low, 70%-80% busy

Look for small success stories

If your management aren’t getting it, you’ll need lots of luck.


A couple of design patterns

The “Vader” design pattern

def addChild( self, c ):
    self._children.append( c )
    c.setParent( self )

And the “Luke”

def setParent( self, p )
    raise Noooooo()

Page Up, Page Down, Home, End

Chatting about Macs with a fellow owner, and we agreed on a few minor details where Windows beats Mac OS. Four of them were the behaviour of the Page Up, Page Down, Home and End keys.

In case you’re a Mac owner, I’m talking about the ⇞, ⇟, ↖ and ↘ keys.

We basically found that we didn’t know what these keys really did and ended up not using them deliberately, and occasionally using them by accident, ending up with confusion.

Read the rest of this entry »

Agile is an Adjective

Below is the transcript of my lightning talk, 99% of which I delivered at Agile Yorkshire 10th December 2013. If you want the full interactive experience, open the prezi and click next each time you read [CLICK] in the text.

What is an adjective?

It describes a thing, e.g.


Tall, Green, Sad.

  • A sad tree
  • The person was very green
  • Our project delivery is quite agile

You can’t “do tall” so why do we nod in polite complicity when someone says they “do agile”?

If I say I’m “doing tall”, what do I mean? Am I growing? Faster than usual? Or am I on stilts? Or wearing a big hat?

What do I mean when I say I’m “doing Agile”? Is it the same things as what my product owner hears? Or the budget holders?

I’m embedded in a team trying to go agile in a wider organisation that is in mid-transition, and is somewhat patchily bought into the concepts. One major preoccupation of mine is communication, especially across the agile / BDUF divide.


For example, a conversation that I had with my product owner:

SM: “Our ability to deliver customer value is hurt by the way priorities change every two weeks”
PO: “But that’s exactly what doing Scrum means I CAN do!”

What’s happening here? Well yes, Scrum does promise this capability, to stop work on features that are no longer valued and switch to reflect emerging priorities, with minimal impact on the delivered product or project timescale. But is doesn’t say that’s the best way to do things!

Looking at our work over the past eighteen months, a common pattern was like this:


as each request came in, it became an epic; then we sliced the smallest possible functional increment in order to have something to demonstrate within a sprint, then at the sprint end the epic would be shelved and a new epic was top priority.

By this pattern of preemption, epics were entering the in-progress state then getting starved of resources, time and again. Customers were getting annoyed at the unbounded lead times.

Meanwhile, we would start getting the bug reports in for the features that we shelved 6 months previously because our focus was switched away after a partial implementation.

So when I opened this conversation with the PO, my observation was that despite the team frequently delivering successful sprints, we weren’t burning up the epics, and the product was still not delivering its potential value in many of its several functional areas, even those that had enjoyed periods of being the most important feature one or more times during the year.

When I suggested to the PO that we should discourage the customer from switching out priorities every sprint, I had in mind that we should increase the emphasis on “focus on finishing” by agreeing to hold the tiller steady until meaningful features were landed. The PO was understandably taken aback, because he valued the ability to switch priorities as laid down in the Scrum documentation for all to see.

What “doing Scrum” meant to me and “doing Scrum” meant to him were quite different.

Developers talk about Code Smell, where certain characteristics of code are not themselves wrong, but they usually indicate that there *is* something that *is* wrong. Like code smell, I think there are also culture smells, and I think use of the expression “doing Agile” is one of them. (Another is writing SCRUM in block capitals, like an acronym.)


Why do we say “doing Agile”? Well, it would be too much of a mouthful to say “I’m doing and promoting and constantly seeking to improve a set of good practices drawn from those accepted by the wider community centred around delivering quality software at frequent intervals and making sure that the outcomes and feedback are communicated effectively to all stakeholders”. So we say we’re “doing agile”. It’s like a macro or a tech-term. Or jargon. Or slang.


Jargon is, let’s face it, one of the fun things about software.  It’s a necessary part of discussing software to come up with new terms for complexes of meaning. For example, in my team we have to be clear whether we are talking about a RunnableEntity, an ExecutableEntity or a SchedulableEntity as each idea carries a set of connotations that have consequences for how the software should work. So we get habituated as engineers to adopting and using new terms or shorthands for ideas.

Slang and secret languages also have a cohesive effect in teams. In my team, everybody knows well what Standard Conversations #0, #10, #14 are, and you can tell the team “Oops that would be me, I checked in a #14 then went to lunch”.  [Meaning, I did something that I thought could not possibly have broken anything and I wasn’t strong-willed enough to do adequate, if any, testing to back up that belief, trusting to fortune that I wouldn’t break trunk. (#10 is “it could catch on” which is used when somebody suggests implementing a unit-test, or reading the spec, and #0 is “#7 is in the wrong tense”)]

But there’s a downside to using this kind of shorthand without checking in that you understand and are understood by the person you are communicating with. In a gelled team, jargon is kept alive by the frequency of communication between the members; the meanings are constantly tested and refined.  Using shorthand terms outside your close tribal group can lead to breakdowns in communication and undermine trust. Even something as seemingly innocuous as to say “doing Agile”.

The problem with a phrase like “doing Agile” is that it wraps up a bunch of ideas in a single term. It puts the idea in a black box. The ideas evoked in the speaker and the listener might not be the same.


Thinking back to my conversation with the PO, perhaps “doing Scrum” means to me:

  • Communication, avoid anchoring, avoid negotiating over estimates
  • Every two weeks we call our own bluff regarding whether the software still works
  • Better contextual information from end users about what is important in the non-functional aspects of the software

Maybe to the PO it means:

  • The team can deliver anything we request within two weeks
  • Planning poker means that backlog estimates are both detailed and accurate

As agile practices gain traction and Agile gains a capital A, we see more and more attempts by command and control organisations to “do Agile”. My employer is one of them. In this organisation, there’s a strong bias towards *execute the procedure* rather than *elaborate the outcome*.


I expect many large organisations are biased this way. Waterfall is a recipe of actions which, if you execute them — and take your eye off the outcome — you end up building the wrong thing… and if only you executed the actions more carefully, with more focus, then you might get the right thing. These cultures see “Agile” and read that it deals with the problems of risk and expense and budget overruns, so they want it. But on a fundamental level, they still view Agile as a recipe of actions. That is why they prefer to “do scrum” than to “try to become more agile”. They also still expect to take their eyes off the outcome and get the right thing. Managers got a quieter life throughout the old project lifecycles. Iterative delivery creates pain by eventuating risks, and managers need to unlearn that shouting at underlings until they keep quiet about problems is the most effective way of removing the pain.

What has this all to do with Grammar?

If agile is a noun, like a tax return, or waterfall, then as soon as you *do* it, you expect to check it off. We are doing iterations, so why don’t you have these great features every two weeks? We locked the team in a warehouse across town [away from PO and the testers] and told them to do Scrum for 6 months, and they still delivered something not suitable for deployment. But we’re doing Scrum!

If agile is an adjective, a quality of something, then it can’t be dismissed so easily. You have to unpack it to understand what is meant. The question “Is the team agile?” is so patently meaningless by itself that you are compelled to get a clarification. Do you mean are our iterations regular? Do we succeed most sprints? Are we able to respond to customer feedback in an acceptably timely way? Already the attention is on the outcomes of agile practices, in terms of the capability profile of the development team.

Rather than being content to “do scrum”, ask yourself and your stakeholders what you wish to get out of the team’s agile practices. What capabilities you value in your software development engine. Keep a mental Jenkins server that turns red when they are violated. If the PO likes “turn on a dime from one sprint to the next” then asks for a feature that will take a full sprint just to understand the question and rough out a demo, then this can be flagged and the trade-off understood.

If you treat the development team as the product, then these kind of capabilities are its features. The capacity of the team to act or react in particular ways or to produce a particular quality of work are outcomes for the process of reflection and improvement. This is where the (meta) value is, not in “doing agile”.

In short, come back to outcomes, not activities.

Ok, SAY I buy all this. How do I stop saying “doing agile?” it’s such a useful phrase!

I’m glad you asked me that. You only need to enforce one simple rule to improve your spoken precision:

Make sure you follow “agile” with a noun: ask yourself “agile WHAT?” before you speak.


  • We are adopting agile practices
  • We follow the agile manifesto
  • We try to use agile estimation

This makes you think harder about what facet of software development you are actually talking about at that moment.

And remember:


Outcomes beat activities.
Declarative beats imperative
Adjectives beat verbs and nouns.


Agile is an adjective.