DORA 2024 – Software delivery performance

The influential DORA group have recently released their 2024 “Accelerate – State of DevOps” report.

This is always a fascinating read with a good base of data and some great conclusions and I always read it with much interest.

DevOps (and its more recent extension DevSecOps) is a set of technical practices which complement Agile development extremely well and this report is valuable data for anyone in software development.

In this article I’ll be diving into the details and pulling out some of what I feel are the most interesting conclusions. The full report is 120 pages across a wide range of topics, including significant explanation of the data methodology. In this article I will focus on the key area of Software delivery performance (which has been a staple of all of the DORA reports).

What is DORA?

Let’s start with some background on DORA. DORA (“DevOps Research and Assessment“) is a research group created by Google which has been researching practices and capabilities of high-performing technology-driven teams and organizations for over a decade. It is tightly linked to DevOps good practices

They publish an annual report based on surveys and statistical analysis, backed with follow-up interviews, to look at trends and try to give guidance. The key focus areas are improvements related to DevOps and the overlapping subject of “what makes a high performing team?”

DORA advises a very experimental approach, looping around the steps below:

• Measure the current baseline state
• Propose an improvement hypothesis. 
• Plan the improvement
• Implement the planned improvement

This aligns with approaches like the Shewhart cycle in quality management (Plan-Do-Study-Act) or the Flow-Feedback-Continuous Improvement approach widely used in DevSecOps.

DORA software metrics

DORA is best known for four key software metrics which they proposed to measure the baseline state for improvements in software development.

Two of these align more closely with development and flow (the “Dev” side of “DevOps”)

• Change lead time: the time it takes for a code commit or change to be successfully deployed to production
• Deployment frequency: how often application changes are deployed to production.

Two relate more closely to deployment and stability (the “Ops” side of “DevOps”)

• Change fail rate: the percentage of deployments that cause failures in production, requiring hotfixes or rollbacks.
• Failed deployment recovery time: the time it takes to recover from a failed deployment.

In general, DORA has argued that teams which adopt DevOps principles and work in a SaaS style environment progress in all four of these.  Many organisations use these to assess their progression and improvement around DevOps flow.

In this report, DORA are proposing some changes to these measures. This is quite radical, given how well known the measures are and how widely used.

Revisiting Change Failure Rate

While organisations have tended to progress in parallel in three of these measures, Change Failure Rate has tended to be an outlier which does not correlate directly with the others.

The DORA report finds that organisations tend to cluster into four performance levels – Elite, High, Medium and Low.  About a quarter of organisations fall into each group. As we might expect, a little more (35%) in Medium and less (19%) in Elite.

This covers quite a range of maturity at the two ends. “Elite” teams make multiple deploys a day. Of these, only 5% fail. By contrast, “Low” ranked organisations deliver once a month to once a quarter. And these deliveries are far less successful, with 40% of these failing.

This may seem counterintuitive – “Low” ranked organisations not only deliver less often but also are delayed even more by rework. This fits a general DevOps pattern that by deploying small changes frequently, change is kept to a minimum and so likelihood of failure is lowered.

However, in the middle categories, which account for 60% of organisations, the picture is much more interesting. There are two data clusters here representing different outcomes.

The other cluster deploys between once a day and once a week, with a substantially higher failure rate of 20%.

One cluster deploys between once a week and once a month. The failure rate of these deployments is fairly low at 10%.

Mid-performing organisations

DORA rather arbitrarily refers to the first as “Medium” and the second as “High”, favouring deployment speed (flow) over failure rate (stability). It is clear this really represents two separate routes through improving towards “Elite”.

One group has increased deployment frequency but at the cost of some stability.  While another grouping have the opposite approach – less frequency, more stability.

Either is a valid route, depending on where to focus. However, this has made the DORA team rethink how to categorise the metrics, since it is clear that they do not improve in parallel.

Rather than the original grouping which broadly aligned with “Dev” and “Ops”, DORA has proposed a new grouping:

• Software Delivery Throughput – consisting of Change lead time, Deployment frequency and Failed deployment recovery time
• Software Delivery Stability – consisting of Change failure rate and Rework rate

The two groupings are internally consistent, so an organisation which improves Throughput might expect to see improvements in all three metrics, and one which improves Stability could expect to see improvements in both of these metrics. However, as the data shows, mid-range organisations could prioritise either one over the other.

Rework Rate

As part of this, DORA has introduced a new fifth key metric – Rework rate.

Change failure rate measures how often a deployment fails. When a change fails, this requires operational work to roll back the change. However, it also requires development work to fix the underlying issue.

The new Rework Rate measure acknowledges that change failures spawn new changes and the rework rate reflects the level of reactive deployments introduced to fix defects. This is based on the survey question below:

Good Practices

The key conclusion here goes back to DORA’s focus on continuous improvement. We cannot necessarily compare a team with fast deployment and low stability with one which takes the alternate path.

However, measuring where we are is less critical than measuring how we are improving. Remember to focus on the continuous improvement loop described above.

As an Agile leader you should always focus on how your team can improve, learn and deliver more value.

The best teams are those that achieve elite improvement, not necessarily elite performance

DORA State of DevOps 2024