Can we estimate and remain true to Agile?

Few topics in planning with development teams are more controversial than estimation.  We know we are working in a complex environment.  Software development is not wholly predictable nor will ever become so.  There will always be a component of development which is emergent and discovered as the work progresses.  Stakeholders can place too much confidence on the predictions.  Developers, perhaps bitten by this in the past, can be reluctant to generate any numbers knowing that these will not be accurate.

However, estimation is a valuable tool to controlling the uncertain environment of scaling organisations.

What is Estimation?

As always, the first step is to get a common language. Estimation is the sizing of individual pieces of work, here product backlog items.  When a team are estimating a piece of work, they are judging how big it is.  They are not saying anything about when it will be done. 

Colloquially we might talk about “estimating when we will release” but I advise you use the term “Planning” whenever you are talking about a schedule. That avoids confusion between “how big” and “when”.

Size of work is (only) one component part of a plan.  Estimation does not tell you when the work will be done.  We also need to know how much effort the team will put into the item.  And we need to know where it is on the backlog. 

Software cannot be sized in absolute terms, so estimation can be performed in different units. The most popular options are:

• Effort Estimation – assessing the amount of working time (effort) which the team will use.
• Story Points – assessing the complexity of the work relative to other work items

Estimation is sizing at the task level

Planning is deciding what actions we will undertake as a result.

Why do we estimate?

The first and most obvious benefit from estimation is prediction.  In Agile development we are working with fixed timeboxes.  These will represent a release or increment.  We would like to understand what we may be able to achieve within that timebox.  In classical project management we would plan out a schedule of work.  In a complex agile environment we lack the stability to do that.  The Product Owner will need a flexible approach which can cope with constant reordering and change.  Having an idea of size of elements gives us that lightweight guidance on what we might achieve in our timebox.

The second value we see from estimation is in ordering the backlog.  It is important that we maintain an ordered backlog to represent the work.  But we cannot order based only on value.  We need to understand the trade-offs involved in work items.  A medium value, low effort piece of work may well be higher on the list than a high value, high effort piece.  Typically a “cost of delay” measure will be used assessing how much value will be delivered, along with the delay from its size and position on the backlog.  These assessments require at least a loose understanding of sizing.

The final value from estimation is perhaps less often considered.  The process of estimation has an inherent value and generates benefits.  By requiring people to consider the size of a piece of work, they each assess what is involved in the work.  I have often seen that exploring estimates reveals whether there is enough clarity to start work (sometimes referred to as “Definition of Ready”). Often the process of estimation discovers some of the missing pieces in the understanding of the work.

Risks of estimation

There is a risk that estimation can itself push teams down a non-Agile route.  If the work is estimated it may appear more predictable. Of course simply assigning a value does not make it less complex.  But this might deprioritise the agile approaches that we need to manage the high levels of change and uncertainty.  By perceiving the project as more subject to decomposition we could shift towards applying reductionist thinking.  This can lead the organisation to viewing the backlog as a predictive fixed scope of work.  The backlog should be no more than guidance of our current best thinking.  And if we misunderstand that we are setting a fixed scope, not a fixed timebox.  Or worse still, fixing both and having no remaining flexibility.

A second argument often used against estimation is that it stresses teams.  This is often raised as a reason not to estimate, especially by proponents of the “NoEstimates” movement.

I would suggest these concerns are less about the data (estimation) than about how it is used (planning). It is the commitment being made to a specific plan which causes stress.  Any abuse of team involvement in planning will cause stress, and this is not specific to estimation. And yet surely the answer is not to plan without the team being involved.  That is simply a step to disempowerment.

This problem is about management style, not estimation process. This issue exposes poor management style and lack of trust.  It needs to be addressed with psychological safety and a mindshift in team management approach.

Good practices

Ensure that your teams understand what is being asked for when they are asked for an estimate. Make sure they are presented with clarity about the scope for which they are estimating.

Build an environment of trust and psychological safety where the team can openly discuss what is realistic in estimation.

Separate estimation and planning. Make sure the scope, estimate and uncertainty are kept together and used as an input in good planning.

Ensure that teams have coaching and support in estimation