Using DORA Metrics: What is Change Failure Rate and Why Does it Matter?

Engineering leaders are responsible for guiding teams to ensure efficient, quality software delivery.

An increasingly common starting point for leaders is the four DORA metrics — key engineering metrics established by the DevOps Research and Assessment Group, including Deployment Frequency, Mean Lead Time for Changes, Mean Time to Recovery, and Change Failure Rate. DORA metrics fall under two categories: incident metrics and deploy metrics. These metrics look at critical markers of performance, and help software organizations balance the tradeoff between speed and stability when it comes to software delivery.

On its own, no single metric can fully illustrate the health of your software organization, but it’s still important to understand each metric and how they play a part in improving delivery outcomes and value to customers.

If engineering leaders are looking to better understand what is interrupting workflow for developers, they can look at Change Failure Rate.

What is Change Failure Rate (CFR)?

Change Failure Rate (CFR) is the percentage of deployments causing a failure in production.

Why is Change Failure Rate important?

Change Failure Rate helps teams have confidence that when they deploy to production, they are improving the system, not creating issues or failures.

Bugs or failures in production could lead to a negative experience for end users of your software. It could also mean that developers are spending valuable time fixing issues rather than developing new functionality. If innovation in your software is slowed, your competitors could gain an advantage in the market.

Change Failure Rate is based on a combination of incident and deploy data, providing teams with a unique understanding of how speed and quality are correlated.

How to improve your Change Failure Rate

Even if your team is delivering code quickly and frequently, you will want to make sure that it’s not at the cost of code quality. There are several ways to use data to investigate and improve Change Failure Rate, but the approach may vary depending on the cause.

Engineering leaders can take a look at key metrics side by side to help uncover the reason behind a high Change Failure Rate, and to determine which adjustment to make to their processes first.

Viewing Change Failure Rate with other key engineering metrics

A great starting point for leaders is to view Change Failure Rate alongside Unreviewed PRs, which can be done within Code Climate Velocity’s Analytics module.

If you see that a high Change Failure Rate is occurring while there is a high volume of Unreviewed PRs, it could indicate that the Unreviewed PRs are causing incidents when merged.

This could mean it’s time to look at your team’s code review process. An effective code review process can help catch bugs or potential issues while still being efficient. By revisiting code review with your team, you can decide which approaches can make your code review more effective and implement best practices.

For one, you can designate automatic assignment of PRs for reviews. Additionally, your team can invest in automated code review, which will make it easier to catch any potential issues with a commit before it even makes it to the pull request stage. This adjustment can also help to reduce the overall number of Review Cycles.

Another approach is to reduce work-in-progress (WIP) — the amount of ticketed and unticketed work being carried out by each engineer, so that developers also have enough time to review their teammates' code. This allows developers to spend more time on each change, reducing the number of errors that might otherwise be overlooked, and also reduces context switching. You can also invest in automated testing to review code before it is deployed. 

Velocity users can look at the Code Review page to evaluate their team’s code review process.

What is a good Change Failure Rate?

The DORA researchers create benchmarks annually so that DevOps teams can compare their performance with other teams.

The research group looks at the software delivery performance of teams as seen through their speed and reliability metrics.

From these observations, the DORA group noticed two clusters — “low performing” and high performing” were clearly demonstrated, while “high performing” and “medium performing” teams were not distinct enough to warrant their own categories. As a result, in 2022, the group omitted the “elite” performing category and redefined the “high performing” cluster as a combination of “high performing” and “elite performing” teams from the 2021 research.

According to the 2022 Accelerate State of DevOps report, high-performing teams keep their Change Failure Rate between 0-15%.

How to track Change Failure Rate improvements

Change Failure Rate is a vital metric to help evaluate processes, and know that your team’s deployments are providing value to customers, rather than introducing errors to the codebase.

When viewed in the context of other key metrics, Change Failure Rate can help you analyze and improve processes and meet your overall goals.

Learn more about how to get the most out of DORA metrics like Change Failure Rate with Code Climate Velocity’s Software Engineering Intelligence platform.