How to Use DORA DevOps Metrics in Large Organizations

Understanding the performance of engineering teams at large companies is no easy feat. For many, this is due to the diversity of processes across teams and the resulting complexity of collecting consistent data. Companies need a standard way of measuring and understanding engineering performance and a common language to communicate it to company leaders and individual contributors. In this article, we’ll discuss how large organizations leverage DORA metrics to do just that.

Using DORA DevOps metrics to communicate with leadership

In startups, engineering actions are often more directly linked to business goals, making it possible for leaders to understand what engineering is doing and communicate its impact. For example, if a startup is launching its flagship product, contributors from sales, marketing, and product management collaborate with engineering, often with executive support and oversight, to ensure the business goals are met. They consider what the product does, how it works, why it matters, who will benefit from it, and how it will be sold. Startups often have shared key performance indicators (KPIs) and operate on a single timeline.

Now scale that same workflow across dozens of teams launching and maintaining different products on varying timelines. While engineering will aim to align goals with business objectives, those goals may vary from team to team, and success will look different for each group. That’s why it’s crucial to establish which metrics are important to the company as a whole and create a framework to measure them. Establishing a framework to measure engineering success ensures that managers are measuring teams in a consistent and equitable way so they can identify and resolve bottlenecks to optimize the flow of work.

Using a framework like DORA is a great place to start. The four DORA metrics, Deployment Frequency (DF), Mean Lead Time for Changes (MLTC), Mean Time to Recover (MTTR), and Change Failure Rate (CFR), can be communicated to leadership to give them a holistic view of how the engineering organization is performing.When implementing DORA, it’s important that organizations start by agreeing on how these metrics will be measured. Unified calculations and standards (i.e. company-wide agreement on what is considered an "outage") are critical for measuring effectively throughout an organization. Standardizing on these four metrics and how they will be measured provides uniformity across teams and creates a common language between engineering and company leadership.

DORA metrics help teams balance speed and stability and are good big-picture checks into the health of the organization. Managers can use DORA to see how things are trending over time and spot when a team isn't working as expected. However, they must keep in mind that while it can be instructive to benchmark teams within an organization by identifying what high-performing teams are doing that others can learn from, it's important to note the context. Managers must understand that teams tasked with different kinds of work and different projects will naturally have variations in their DORA metrics, which is normal and expected.

Supporting developers with granular engineering metrics

Using DORA as the foundational framework across teams lets engineering leaders understand how a team is doing within the context of the broader organization and drill down into data from a specific team to learn more about the way it's working. DORA metrics can highlight areas worth attention, serving as a starting point from which managers and their teams can investigate the efficacy of their processes and make changes, then track the impact of those changes.

To do this, they can add context to the four DORA metrics and pair them with complementary metrics to get more insight into what’s happening with individual teams and what improvements might be useful. Common metrics pairings include:

• Change Failure Rate and Unreviewed Pull Requests. If a high CFR is correlated to a high percentage of unreviewed PRs, managers may consider adjusting the process to prevent unreviewed PRs from being merged and causing issues.

• Deployment Frequency and PR Size. If DF is low, managers can use PR size to investigate it. If large PRs are correlated with a low DF, they can coach team members to break work into smaller PRs to see if DF improves.

• Mean Time to Recovery and Revert Rate: A long MTTR could indicate a high Revert Rate, therefore disrupting production and lengthening the time it takes to recover from an unplanned outage or defect. If there’s a correlation, managers can drill down into each revert and see whether the issue is a defect or an undesirable change.

• Mean Lead Time for Changes and Cycle Time. If MLTC is high, it could indicate that Cycle Time is also high. Managers can view these metrics in tandem and dig deeper into related metrics like Time to Open, Time to Merge, and Time to First Review to find the root cause.

How DORA software creates a common language from ICs to the C-suite

Large companies can benefit from a Software Engineering Intelligence (SEI) platform to understand engineering performance at every level of the organization. It allows engineering managers to standardize measurement and reporting on the four DORA metrics to communicate performance to company leadership and ensure that the pace of work meets business needs. Managers can also combine DORA with other engineering metrics in their SEI platform to communicate with their teams to ensure they have what they need to be successful and roadblocks are quickly identified and removed.

Without a strong framework and a centralized platform to measure it, engineering data can become a tangled mess as the number of engineers at a company increases. Measuring DORA and complimentary engineering metrics in an SEI platform helps leaders make sense of their data to ensure that engineering work is optimized and aligned with business objectives.

To find out more about how an SEI platform can benefit leaders at large organizations, speak to a Velocity specialist.