Continuous improvement using DORA metrics

Reading time

5 min

“DevOps Research and Assessment“, better known by their abbreviation DORA, is a team within Google Cloud. They research DevOps good practices and publish annual State of DevOps Reports.

The DORA core model

In the DORA core model there are a set of capabilities which a team possess. The level of capabilities predict team performance. And team performance predicts outcomes.

Capabilities -> Performance -> Outcomes

Outcomes

The outcomes are both organisational performance and team wellbeing. Both business and individuals should thrive. Remember that these are the value areas which we wish to improve and optimise.

Performance

The DORA metrics sit in between the capabilities and the outcomes and fall into two categories.

Reliability metrics or Service Level Objectives for the operational side of the software
Software performance metrics, which cover how software is created and deployed

Capabilities

DORA’s capabilities are based on the “three ways” approach to DevOps proposed in the book “The Phoenix Project“. These can be summarised as:

Fast Flow – emphasise flow and the performance of the entire system.
Fast Feedback – shorten and amplify feedback loops so necessary corrections can be continually made.
Climate for learning – psychological safety, continual experimentation and learning from failure.

There are a set of techniques which will contribute to each of these, many of which overlap with Plays on this site. Some examples are below:

Fast flow is enhanced by continuous delivery, version control, small batches, limiting work in progress, deployment automation and loosely coupled teams.
Fast feedback comes from continuous integration, automated test, work management systems, monitoring and customer feedback .
The climate for learning comes from psychological safety, empowered self-managing teams and effective approaches to documentation, knowledge sharing, technical debt and code maintenance.

The DORA performance metrics are designed to predict (be leading indicators for) business outcomes, which is to say that by observing increasingly good performance measures we expect to see increasingly positive business outcomes.

The metrics are themselves led by capabilities. The numbers increase as the capabilities are improved. You cannot directly address the numbers without improving the capabilities. A focus on simply improving the numbers is likely to create the wrong outcomes.

When you try to measure performance, particularly the performance of knowledge workers, you’re positively courting dysfunction.
“Lean Software Development” – Mary Poppendieck

Productivity

The DORA metrics do not directly measure productivity, which would relate to the outcomes in the DORA model. As we see elsewhere, measuring productivity is extremely challenging to achieve. So if productivity cannot directly be measured, how can the DORA metrics hope to be successful?

to measure software productivity you have to measure the output of software development – the reason we can’t measure productivity is because we can’t measure output.
Martin Fowler 2003

The DORA measures take a lean approach to the problem. Although we cannot easily measure productivity directly, we know that productivity is reduced by waste. If we identify the key areas of waste in software development, we are able to measure that we are keeping these wastes low. We have already identified these key areas of waste from the work of Poppendieck.

DORA’s software delivery metrics

Taking this approach of focussing on waste, the DORA team developed four metrics for software delivery.

These are subdivided into two complementary areas with two metrics in each area.

Throughput measures how fast changes are being made.
Stability measures the quality of the changes being delivered.

These two balance each other in a similar way to a balanced scorecard approach. A pressure to increase speed could impact quality. While a pressure to reduce defects could lead to reduced changes. By ensuring we measure both we keep a broad picture.

Change Lead time

How long it takes for a change to be successfully deployed to production. This focusses on the efficiency of the delivery pipeline.

Lead time can be measured at various points in the value stream. For example, the lead time from a customer making a request to seeing the delivered software is an end-to-end measure. The DORA lead time is specifically around deployment and runs from the code being committed to the change being deployed in production.

Deployment frequency

How often changes are deployed to production. As we’ve seen elsewhere in Agile development, small batch size has advantages. This increases agility and minimised work in progress. Small batch size also makes it much faster to find and recover from any errors.

Change fail percentage

The percentage of deployments which cause failures in production (requiring a hotfix or a rollback). Lower change failue rate implies a more reliable process and less time lost in recovery.

This is measured as a fraction of deployments. Keeping deployments small will make these more reliable. If we measured the absolute number of change failures, this would push the team to less frequent deployment.

Failed deployment recovery time

The time it takes to recover if a deployment does cause a production failure. In a resilient system, recovery time will be low.

Approaches such as feature flags allowing the new capability to be rapidly removed while the root cause is assessed will help here.

Good practices

As an Agile leader, how you use metrics like the DORA metrics is critical. Setting metrics as a direct goal will lead to gaming, especially if you do not use a broad set of metrics.

Management by numerical goal is an attempt to manage without knowledge of what to do, and in fact is usually management by fear
“Out of the Crisis” – W.E. Deming

Use DORA to help a self-managing team improve themselves over time. Comparing teams and applications is not likely to be effective as their circumstances will differ.

Avoid silos. These are not “development metrics” or “operational metrics”. Only by the teams collaborating and working together will they improve.

Remember that the end goal is to maximise value – outcomes in the DORA model. Investing in metrics is a means to a goal, not a goal in itself.