Code Climate

The four DORA metrics — Deployment Frequency, Change Failure Rate, Mean Lead Time for Changes, and Mean Time to Recovery — were identified by the DevOps Research & Assessment group as the four metrics most strongly statistically correlated with success as a company.

Within those four metrics, the institute defined ranges that are correlated with meaningfully different company outcomes. They describe companies based on the outcomes they achieve as “High Performing,” “Medium Performing,” or “Low Performing.”

Moving between categories — for example, improving your Deployment Frequency from “between once per month and once every six months” to “between once per month and once every week” — leads to a statistically significant change in the success of a business. Moving within a bucket (for example, from once per month to twice per month) may be an improvement, but was not shown to drive the same level of shift in outcome.

DORA calculations are used as reference points across the industry, yet, there is no agreed-upon approach for DORA assessment, or accurately measuring DORA metrics. To set the original performance benchmarks, the DORA group surveyed more than 31,000 engineering professionals across the world over a span of six years, but responses were not based on standardized, precise data.

DORA metrics have been interpreted and calculated differently for different organizations, and even for teams within the same organization. This limits leaders’ ability to draw accurate conclusions about speed and stability across teams, organizations, and industries.

Because of this, there are subtle pitfalls to using automated DORA metrics as performance indicators.

Code Climate Velocity measures DORA metrics with real data, as opposed to proxy data, for the most useful understanding of your team health and CI/CD processes.

Code Climate’s Approach to DORA Assessment

As mentioned, there are many different approaches to automating the measurement of the DORA metrics in the market. In order to enable engineering executives to understand how their organization is performing across the four DORA metrics, we wanted to provide the most accurate and actionable measurement of outcomes in Velocity.

Our approach relies on analytical rigor rather than gut feel, so engineering leaders can understand where to investigate issues within their software practices, and demonstrate to executives the impact of engineering on business outcomes.

Using Real Incident Data, Not Proxy Data for DORA Calculations

Not every platform tracks Incident data the same way; many platforms use proxy data, resulting in lower-quality insights. Velocity instead uses actual Incident data, leading to more accurate assessment of your DevOps processes.

Velocity can ingest your team’s Incident data directly from Jira and Velocity’s Incident API. These integrations provide a way for every team to track metrics in the way that most accurately reflects how they work.

The Most Actionable Data

We made it possible for engineering leaders to surface DORA metrics in Velocity’s Analytics module, so that customers can see their DORA metrics alongside other Velocity metrics, and gain a more holistic overview of their SDLC practices.

Teams can evaluate their performance against industry benchmarks, as well as between other teams within the organization, to see which performance bucket they fall under: high, medium, or low. Based on that information, they can scale effective processes across the organization, or change processes and measure their impact.

Balancing Speed with Stability: How Velocity Metrics Contextualize DORA Metrics

If teams evaluated DORA metrics in isolation and discovered that their teams have a high Deployment Frequency or that they deploy multiple times a day, they may still be considered “high performing” — yet we know this does not tell the whole story of their software delivery. Velocity metrics and other DORA metrics within the Analytics module help contextualize the data, so that teams can understand how to balance speed with stability.

For example, the Velocity Metric PR size (number of lines of code added, changed, or removed) can be a useful counterpoint to Deployment Frequency. If you view these metrics together in Velocity’s Analytics module, you can see a correlation between the two — does a low Deployment Frequency often correlate with a larger PR size? If so, leaders now have data-backed reasoning to encourage developers to submit smaller units of work.

This doesn’t necessarily mean that your Deployment Frequency will be improved with smaller PR sizes, but it does provide a starting point to try and improve that metric. Leaders can note when this change was implemented and observe its impact over time. If Deployment Frequency is improved, leaders can scale these best practices across the organization. If not, it’s time to dig deeper.

DORA Metrics Definitions

Deployment Frequency – A measurement of how frequently the engineering team is deploying code to production.

Deployment Frequency helps engineering leadership benchmark how often the team is shipping software to customers, and therefore how quickly they are able to get work out and learn from those customers. The best teams deploy multiple times per day, meaning they deploy on-demand, as code is ready to be shipped. The higher your Deployment Frequency, the more often code is going out to end users. Overall, the goal is to ship small and often as possible.

Mean Lead Time for Changes – A measurement of how long, on average, it takes to go from code committed to code successfully running in production.

Mean Lead Time for Changes helps engineering leadership understand the efficiency of their development process once coding has begun and serves as a way to understand how quickly work, once prioritized, is delivered to customers. The best teams are able to go from code committed to code running in production in less than one day, on average.

Change Failure Rate – The percentage of deployments causing a failure in production. If one or more incidents occur after deployment, that is considered a “failed” deployment.

Change Failure Rate helps engineering leaders understand the stability of the code that is being developed and shipped to customers, and can improve developers’ confidence in deployment. Every failure in production takes away time from developing new features and ultimately has negative impacts on customers.

It’s important, however, that leaders view Change Failure Rate alongside Deployment Frequency and Mean Lead Time for Changes. The less frequently you deploy, the lower (and better) your Change Failure Rate will likely be. Thus, viewing these metrics in conjunction with one another allows you to assess holistically both throughput and stability. Both are important, and high-performing organizations are able to strike a balance of delivering high quality code quickly and frequently.

Mean Time to Recovery – A measurement of how long, on average, it takes to recover from a failure in production.

Even with extensive code review and testing, failures are inevitable. Mean Time to Recovery helps engineering leaders understand how quickly the team is able to recover from failures in production when they do happen. Ensuring that your team has the right processes in place to detect, diagnose, and resolve issues is critical to minimizing downtime for customers.

Additionally, longer recovery times detract from time spent on features, and account for a longer period of time during which your customers are either unable to interact with your product, or are having a sub-optimal experience.

Doing DORA Better

Though there is no industry standard for calculating and optimizing your DORA metrics, Velocity’s use of customers’ actual Incident data, and ability to contextualize that data in our Analytics module, can help teams better understand the strengths and weaknesses of their DevOps process and work towards excelling as an engineering organization.

Interested in learning which performance benchmark your team falls under, and how you can scale or alter your engineering processes? Reach out to a Velocity specialist.

‍

At Code Climate, we value collaboration and growth, and strive for greatness within our product and workplace. For us, this means fostering a supportive, challenging, people-first culture.

Thanks to an emphasis on these values, we’ve earned spots on three of Built In’s 2023 Best Places to Work awards lists, including New York City Best Startups to Work For, New York City Best Places to Work, and U.S. Best Startups to Work For.

This annual awards program recognizes tech companies across the United States, including startups and enterprises. Awards are based on compensation, benefits, and highly searched-for offerings on the Built In platform, like DEI initiatives, flexible work opportunities, and support for professional development.

We work hard to make Code Climate a great place to work, and we’re honored to be on Built In’s lists for the second year in a row. Here’s what employees have to say about why they love working at Code Climate.

Support and collaboration are key

The culture is collaborative, and everyone is encouraged to do their best work. There is a strong focus on enablement and supporting everyone to run with all of their ideas — even if they have never been done before. — Addison LeBeau, Customer Marketing Manager

Our organization is very flat so I feel comfortable reaching out to anyone with a question or concern, from our salespeople to our CEO. — Mike Koeneke, Senior Product Manager

We challenge each other and ourselves to grow

I joined Code Climate because I was seeking a challenging yet fun environment that would push me to grow my technical acumen and operating knowledge. It hasn’t disappointed, and I love that the work environment encourages autonomy, experimentation, and professional development. — Glenn Rothwell, Director of Sales Development

In project work, I feel like I am empowered to take on exciting challenges and learn new things. — Sam Dallas, Senior Software Engineer

We’re building something great together

What I enjoy most about Code Climate is getting to work with an amazing group of talented and passionate coworkers to build innovative products and solve complex problems for our customers. We have a unique opportunity to make an impact across technology departments big and small, helping them move faster and build better products. — Madison Unell, Senior Product Manager

I love our product; as a software engineer it’s a dream being able to build tools to empower other engineers and organizations. Such a privilege. — Javier Murillo, Software Engineer

The people make the place

Each employee truly enjoys what they do and are always willing to help out when you need it. On our in-person days I walk into the office and am surrounded by a great group of people! —Amanda McLaughlin, Employee Experience Generalist

Everyone at Code Climate is delightful. I feel supported in my career and challenged through my work. Our customers are also fantastic! — Jilly Chen, Manager, Solutions Consulting

Code Climate is like my second family. I’ve never felt more comfortable to be myself andto really be able to excel at my work. — Kevin Blackwell, Sales Engineer

Learn more about working at Code Climate and view our open roles.

‍

As the adage goes, the best laid plans go awry, and that also applies to building software. The planning phase, including maintaining alignment, is critical in engineering, but even for the most mature teams, plans can change and evolve as they’re executed.

Engineering leaders need to keep teams aligned with the overall goals of their organization, and that includes setting and managing expectations for stakeholders so they won’t be blindsided when roadmaps need to change. How do CTOs successfully align on strategic priorities, navigate planning, and course-correct when things go off-track?

As part of Code Climate’s Next event, we invited engineering leaders Catherine Miller, CTO at Flatiron Health; Juan Pablo Buriticá, SVP of Engineering at Ritchie Bros.; Ryan Grimard, SVP of Engineering at EverQuote; and D Orlando Keise, Head of Banking Foundational Platform at UBS, to share their experience leading engineering teams while keeping the C-Suite at ease.

Read on for highlights from the panel discussion, led by Peter Bell, CTO and Founder of CTO Connection.

Peter Bell: An engineering leader’s responsibility is keeping the team aligned with a larger organization, and that starts at the top. What do you do to ensure that you’re on the same page with the rest of the C-Suite around things like resourcing, roadmap planning, and strategic priorities?

Ryan Grimard: Our teams use the Scaled Agile Framework, SAFe, for our planning and execution, and we involve our business leaders way in advance. They’re helping us with strategic planning for the company, and then our product and engineering teams are working through portfolio management and bringing things into the conveyor belt. When things are ready for final prioritization for a particular quarter, we use that process to go through “big room planning” and a one-week prioritization planning process. The end of that is effectively a confidence vote for all of the teams. We have 17 Agile teams, and the business leaders are in those meetings and hearing the confidence vote, and giving the thumbs up that they agree that these priorities that the teams have picked actually match up with the OKRs that we set at the company level.

Juan Pablo Buriticá: I have two techniques. One is to force the C-Suite to compromise on a single thing that they care about through a guiding principle. So, do we care about speed, do we care about quality? And then, I use that guiding principle to force decision making on the group for alignment.

The second thing is, a set of cascading strategies: you have business strategy, the product strategy that cascades from the business strategy, and the engineering strategy, which should enable both. And then, it forces resourcing, staffing, and prioritization to be aligned with the guiding principle. That guiding principle is the tiebreaker for everything.

D Orlando Keise: What I’ve found is important is to align on the mission. And I’m using “mission” in a sense that’s more than just the objective. I actually mean it almost in a military sense. What is our target? What are our threats? What’s the landscape and the terrain that we’re operating in? I think we all know that we’re going to come up with a plan, that we have a roadmap, but I find that the plan is not going to survive first light with the outside world.

I want to make sure that when that happens, we’re aligned not only on what we’re trying to do, but why we’re doing it, and the environment that we’re doing it in. Because when we start changing that plan, if we’re not aligned on all those other things, we’re going to run into problems.

Peter: What kind of conversation do you find the most difficult? Is it when you’re in the planning phase, and you have to say, ‘We’re not going to fit all that into this quarter,’ or is it once you’ve said, ‘Sure, we’ll do this by the end of the year,’ and then it’s December 12th and you’re realizing that Christmas might have to be canceled this year?

Catherine Miller: Planning conversations are about feelings, and execution conversations are about data. I like the execution conversations better, because the planning conversation ends up being a very abstract conversation that is based on trust. The more trust you have the better that goes, but you’re all just guessing, and at the end of the day, you are trading on some relationship or some extrapolation, and you know it’s wrong.

Then you get to the execution, and first of all, no one actually expects it to go on track, but what I love about execution is, you can check in along the way, you can see how it’s going, and it forces a conversation. What are you going to do? We literally cannot hit this deadline because of where we are. That is a fact. There is no hoping or wishing that will make that go away. So you’re forced into decision-making in a way that less mature teams often avoid at the planning stage where they just hope everything will be okay.

Ryan: I would say that our company has really evolved over the last two or three years. It used to be a much more difficult conversation upfront when business leaders asked, ‘These are the strategic priorities; how many of these can we complete in a quarter?’ Because they weren’t necessarily involved in the process that I described earlier, they were surprised when we couldn’t do everything at the same time and get everything out the door in that particular quarter. So since we’ve introduced this process, I feel like the more difficult part of that is through the backside, or partway through the quarter when the conversation might be: Why aren’t we executing? And those really get broken down into retrospectives. When something doesn’t quite go right, teams will do a full retrospective. They will come up with a root cause analysis through that retrospective, and line up whatever we need to fix, then present that to those business leaders. I think that builds confidence across the board.

More recently it’s been boiling down to either high levels of tech debt for a particular team, or our team isn’t onboarded to what we call our “paved path,” or our delivery pipeline.

Juan Pablo: I like solving problems by preventing them from happening, so I’m usually more uncomfortable when things have gone off track, because that means that we didn’t prevent what we were trying to prevent. Whereas, when I’m on the planning aspect, I feel very comfortable. Plans are lies we tell ourselves to get to the next stage and motivate our teams and motivate ourselves, so I have a much easier time on that end than the former.

Peter: When you realize for whatever reason that your team isn’t going to hit a date — there’s some commitment you’ve made, it’s clear now that it’s not going to happen — what’s the first thing you do?

Juan Pablo: The moment I find out, I broadcast that state to the organization, because rather than rebuilding trust, I prefer not losing trust. I’ve learned that people will usually handle  transparency better than we think.

Since we’re building software and things tend to go wrong in software, I like building a culture where being off-track is not uncommon, and people don’t react with fear. We’re a good engineering team if we’re good at reacting quickly.

D: I want to underscore something that Juan Pablo said. The very first thing is to communicate it. The second thing is to understand what kind of problem we have. Did we plan poorly for the amount or type of resources that we have? Did we have great resources but the ground shifted under us?

I find that people think they’re solving the real problem but they’re really solving some symptom. So, I try to go a few layers down to find the real problem that caused us to be off track to begin with, and attack that.

Catherine: Part of what I would do is figure out whose problem this is to solve. Is this something that I should really be [calling on] one of my VPs or even directors for, and setting a framework for them to solve, or is this significant enough for me to dig into directly?

Ryan: Keeping it amongst the team is important, at least for a place to start, where the team is a handful of engineers and the product manager or the product owner. There’s a lot of trust that should have been built up over time. We’re pretty big on keeping teams whole and not rotating people around unless there’s some sort of internal mobility opportunity for folks.

Peter: So you found out things are off track, and you have to prepare for a difficult conversation with a senior stakeholder. Are there any things you do when preparing for that to try to make it just a little more likely to go smoothly?

D: A lot of my preparation for moments like that happens long before the moment. A lot of us have used the word trust at various times, and that’s huge. I’m trying to establish trust with stakeholders and partners from day one, because so many of these moments are going to rely on trust to go the way they need to go. The other thing I try to do in the moment, when it comes to explaining a difficult truth, is to take any of the emotions out of it. I try to lay out something that’s very fact-based.

Cat: I very much agree with the emotional regulation piece as a key part. If I have the leeway and the time, the best thing I can do is I can sit down with my coach. As we talk, I lose my emotional attachment to things, until I’m just thinking about options that are good and bad and what the cost to the business is.

Ryan: Data gathering, absolutely. Pre-reads I think are huge, meaning sending out what you plan on talking about beforehand, and making it clear that reading the pre-read is a requirement. You shouldn’t be going into the meeting and then learning about what you’re about to hear about. The pre-reads give you that opportunity to be informed before the meeting, and if successful, meetings are for making decisions.

We’ve run into situations where the pre-reads are pretty in-depth, all of the decision making and our suggestions on where to go next are already laid out, and the meeting is five minutes long.

Juan Pablo: I use paper as my coach. I believe writing is thinking, and I try to anticipate questions that might come from the conversation. If I’m walking into a board meeting, I try to minimize the times I say: I don’t know, let me get back to you.

I have my talking points. What is the outcome of the conversation that I’m seeking? Am I trying to inform? Am I trying to convince? Am I trying to drive to a decision? Based on that, I bring options or I shape my talking points.

Peter: Things didn’t go according to plan, you missed the deadlines, how do you rebuild trust both within your engineering org, and with your executive stakeholders?

Cat: It’s not inherently breaking trust to miss your deadlines if you’re being transparent about your metrics and what you’re seeing as it goes along. It’s mostly that transparent communication that will leave no one surprised.

Ryan: I think rebuilding trust is really about consistency and showing continuous improvement. We’ve been using some of the DORA Metrics that Code Climate has been putting out.

We have custom reports for DORA Metrics, and they’ve been really awesome. Being able to show that the team is performing, that the issue that we ran into was kind of out of our control, and showing there’s an upward progression to how they’re performing…it’s big to show that they’re not not performing.

Juan Pablo: I usually lead product engineering teams, and so the focus is less the plan or the deadline and more the outcome, so ensuring that the teams understand the goal that they’re supposed to be driving. And in product, your goals are so variable, and you can invest 6 months launching a feature that tanks or spend a week and suddenly you have a great product, and so what I care about is… less about times and deadlines and estimates and plans and more about, are you being impactful and are you shipping as fast as you can so we can learn?

D: I would underscore that rebuilding trust is really about being reliable in the things that you say. Communication is important. We had talked about diagnosing the real problem, and now when you’re going and acting on that and executing. The second time around, are you hitting what you aimed to hit based on the problem that you diagnosed? Hopefully you are doing a good job of identifying what that real problem is and then when you drive the team to execute towards that milestone, it actually solves what you said it was going to solve … as you do that you’re rebuilding trust.

Audience member: [To Juan Pablo] Tactically, how often are you resetting your principles because you can’t make the mark or realize priority two has to be priority one?

Juan Pablo: I’m doing a good job if I’m not resetting them that much, or they at least have an 18-24 month lifespan. I do start from a mission. When I was at Splice, the mission of engineering was to enable Splice to learn faster than the market before we ran out of money. So that was driving engineers, that was our mission, anything you do is in service of that, because we’re serving musicians.

And from there, the principles that we set were about sensible choices in technology. We were like, yes it’s fun to learn, but our job is not to learn; our job is to learn faster than the market. I also crowdsourced feedback from the team as far as what resonated with them in regards to our principles, because usually I don’t have the full perspective. If there’s ownership from the group, it’s easier for them to also embrace the principles.

Audience member: There’s a known standard in the tech industry of listening to your customers versus understanding business objectives from leadership. What has been your experience trying to carve in both business objectives and customer feedback?

Cat: More and more, I think every dichotomy in tech is long- and short-term planning. For example, I think that tech debt is long term and product and desires is short term, and I think what you’re talking about a little bit here is what your customers are beating down your door right now, whereas presumably your business metrics are about your long range plans, what your business needs to grow, what our burn rate needs to be… And I don’t necessarily have an answer for you, but I think dispassionately thinking about how much are we investing immediately — how much do we need to do this month or this year, versus three years from now? — is an interesting way to think about your portfolio across a lot of different things like this.

Ryan: I worked at a company where it was very referral based, and the business metrics were: How many referrals did we make, and how much revenue did we make from that referral? The other side of that is, how happy is your customer? Does a happier customer actually produce more revenue? Maybe not in the very short term, but repeat customers, and happier folks in the marketplace of that particular referral business goes a long way. The business is able to interact with a consumer that’s happy and potentially upsell them and so on.

Juan Pablo: Usually we’ve done a good job as a leadership team if we’ve set business metrics that empower teams to take these customer needs and act on them. Because customer needs and business metrics should not be diametrically opposed, and if they are, there’s something wrong.

The most successful engineering leaders incorporate objective data into their leadership strategies. Numbers can’t substitute for a CTO’s experience and instincts, but when it comes to decision-making, leaders can use metrics in software engineering to inform their decisions and align with stakeholders.

Different leaders optimize for a different set of metrics depending on company priorities and the needs of their engineering teams. Yet, if you’re introducing metrics to your team, or refining your approach to metrics, there are key measurements worth considering.

At Code Climate, we’ve worked with thousands of organizations, from startups to enterprises, and we know that there are a few key metrics that have proven time and again to be valuable, even if they’re just a starting point!

Whether you’re just starting to incorporate data into your leadership, or are refining your approach to measurement, these 10 metrics are important ones to consider.

They can help you:

• Get a handle on your organization’s Time to Market,
• Ensure your team is adhering to CI/CD best practices,
• Measure the quality of your code, and
• Assess the efficacy of your planning processes.

To find out which 10 metrics you need to know, how to apply them effectively and resolve bottlenecks, download the ebook.  

In the midst of pandemic lockdowns, VTS, a leading provider of commercial real estate technology, was in a period of rapid growth. In addition to aggressive hiring, VTS grew through acquisitions, adding Rise Buildings and Lane to its portfolio. Soon after onboarding, they discovered the new teams had less effective SDLC processes, which caused Cycle Time to trend toward 80 hours — nearly double the average Cycle Time of the core VTS team.

Engineering leadership leaned heavily on Code Climate as they incorporated the new teams and integrated the new products into the VTS platform. They leveraged Code Climate's partnership to investigate bottlenecks, and discovered the teams were spending Cycle Time resolving issues and needed more efficient workflows.

Fostering an Elite Engineering Culture

Being customer-obsessed and striving for excellence are the core tenets are the foundation of VTS culture. And for engineering, these values drive an ambitious vision of producing elite engineering talent who innovate to serve customers, achieve business outcomes, and positively impact the broader tech industry.

With more than 20 teams and 200-plus engineers, VTS fosters a high-caliber engineering culture built on mutual trust. They have collectively embraced a vision of engineering excellence, and they leverage Code Climate to measure proficiency and success, surface bottlenecks, and actively explore ways to improve. Code Climate's solution delivers end-to-end visibility into the entire development pipeline, which is crucial for tracking engineering progress and achieving OKRs with a large, distributed team.

Prashanth Sanagavarapu, Head of Platform Engineering at VTS, said without these insights, every decision would be a shot in the dark. “As a manager, my worst nightmare is running blind. I need to make decisions based on data and facts, and Code Climate provides exactly what we need.”

Metrics That Matter

For VTS, Code Climate provides visibility into the metrics that matter, and it is more intuitive and robust than what is built into other engineering tools. For example, Jira reporting was inadequate because it lacked context, and engineering leaders couldn’t compare metrics to industry standards.

“An ops team may close 100 tickets, but what does that mean? Someone has to go into each ticket and read the description to understand what is happening, and that just isn’t sustainable,” said Sanagavarapu.

Code Climate allows them to analyze factors like Pull Request (PR) size, frequency, and time to close, enabling them to optimize workflows to consistently deliver incremental value and maintain engineering velocity. Sanagavarapu said he learns quite a lot through the platform: “It’s a fact tool for me. I can see the trends of what is working and what isn’t working for a particular squad and correlate it back to sprint retros.”

Cycle Time is the north star metric at VTS. Measuring Cycle Time every two weeks with Code Climate provides visibility into how fast they are shipping, both organization-wide and at the team level, and it enables them to quickly see when fluctuations occur. Then, within the platform, they can easily drill down to identify choke points and dependencies that may be impacting performance. Understanding if the Cycle Time went up due to outages, open RFCs, or a change in personnel helps leaders to understand trends and better allocate resources to ensure their teams have what they need to be successful.

Sanagavarapu said the ability to drill down to the individual contributor level is very impactful because it allows you to diagnose problems at any level and scale. Since partnering with Code Climate, they have improved Cycle Time by 30% and doubled their deployment frequency.

“Our average Cycle Time tends to be around 35 hours with 48 hours as our max threshold. When we exceed that, we know there is something going on. If it’s not a holiday or another known factor, we can dig to discover and understand the problem and who is being impacted — then, we can solve it.”

Maintaining Excellence During Rapid Growth

Enhanced visibility has been crucial for engineering leadership over the past two years, with company growth accelerating during challenging pandemic lockdowns. Sanagavarapu said more than 60% of the company’s 600-plus employees joined during this time, most of whom were engineers.

Infrastructure stability was a big challenge, so they worked to reduce the number of incidents so that dev teams could spend more time on value-add work. When they discovered a lag time in PRs due to time zone differences, they changed their workflows to reduce the time for feedback and better manage resources across teams. They also added in more test cycles so that rework happened less frequently. Now, the entire engineering organization maintains Cycle time under its 48-hour threshold.

“Code Climate provided insights that helped us accelerate integrating those teams into our culture and processes more quickly and effectively,” Sanagavarapu said.

Measuring Impact

VTS leverages Code Climate's solution to track and quantify impact at all levels. Engineering leadership can measure business impact by translating metrics into stories that show how engineering delivers value. They can understand how quickly teams deliver new features that are important for customers and compare the time spent on new feature work to rework time to ensure engineering time and resources are optimized.

Code Climate surfaces important trends that help engineering managers better understand the impact of process and workflow changes on developer experience. They can drill down to the developer level to diagnose issues and determine what might be consuming a squad’s time. Visibility into engineering capacity helps planning for major initiatives, allowing them to best leverage internal resources and balance workloads with external contractors.

As VTS works continuously to innovate, evolve, and achieve both engineering and business milestones, the insights derived from Code Climate are invaluable, Sanagavarapu explained. “Code Climate is not a reporting tool. It’s the heart of engineering excellence.”

Request a consultation to learn how to maximize engineering impact.  

Metrics are an essential tool for engineering leaders looking to gain objective insights about the teams they manage. In order to improve business outcomes and overall team health, leaders must identify which metrics to track and apply the data with intention.

Code Climate’s Senior Product Manager, Mike Koeneke, sat down with three directors of engineering: Sophie Roberts of Shopify; Mojtaba Hosseini of Zapier; and Romain Dupas of Code Climate, to discuss how they each define and utilize metrics to meet their goals.

Read on for highlights from their conversation. Answers have been shortened for length and clarity.

Defining Metrics: Which metrics do engineering leaders look at, and why?

Sophie: There are four kinds of metrics I look at on a regular basis:

• Operational or system health metrics: These include SLAs, API failure rates, system throughput — the things that tell you, as an engineering leader, if your product is healthy.
• Development metrics: Your average time to commit, your average build time, and your build success failure, which fall under engineering productivity, helping us understand how to improve the productivity of our engineering teams.
• Organizational health metrics: Team composition, retention, employee morale, and ability to actually hit goals that you set. It helps us understand if our people are in a good place and are being effective.
• Business metrics: These metrics show that the business is moving forward. Are we having the impact on the business that we expect to have with regards to things like revenue, customer acquisition, efficiency?

Mojtaba: One thing we’ve started to do at Zapier is to talk about hierarchy of metrics that go beyond engineering, and trying to tie in the engineering metrics into a larger set. As a SaaS company, we’ve got our financial metrics which are very top of mind for the board and the C levels…[so for example,] your customer metrics: your churn and activation…if you’re developing quickly, if you’re getting that feedback, if your teams are high performing and happy and engaged. If so, you probably are creating good products, which bring in users, which then brings in the revenue.

Romain: I look at metrics corresponding to all aspects of our process and our goals: Team goals such as OKRs, financial goals, computing metrics, API metrics, productivity. At the end of the day, any goal or process needs to be measurable. Defining those goals and metrics will make the process more efficient.

Balancing Intuition With Objective Data

Sophie: The superpower of actually using metrics is to combine the data with judgment and instincts. If someone on your team is in the lower bound of what you’ve chosen as a performance metric, and you substitute the metric for your judgment and say they’re underperforming, you’re making a bad decision.

When it comes to things like organizational metrics or individual performance metrics, I don’t even look at those as objective pieces of data. I look at those as pieces of data that allow us to explore the margins and find things to be curious about. That’s where your judgment and instinct come in.

Mojtaba: I have a mental model around using metrics: the dashboard of a car. The car’s purpose is to serve the drivers and the passengers. The dashboard, intrinsically, is of no real value if the humans don’t get where they want to go.

You need the objective data: What is my speed? What is the oil gauge telling me? The speedometer? But you still need the human to decide where to go. [As the driver], I might know the maintenance history of the car, so even if the manufacturer tells me I can go faster, I’m not going to push the car to the maximum speed — because of experience and intuition. The intuition by itself can be too vague and needs the objective data, a reality check. But the reality check without that vision just becomes a bunch of numbers.

Romain: Continuing with the car analogy, on the dashboard is a limited amount of space to show the information that matters. You don’t have anything related to what’s in your trunk; the only information you have is what’s valuable: information about the safety of your car. With metrics, you don’t measure everything; understand what you want to measure for your own goal.

Context is Key: Make Sure Your Team Knows Their Purpose

Sophie: It’s critical for me, and I think for all businesses, for every single engineer to understand how the work contributes to the business’ success. When people understand that, then they understand what [things like] an outage actually mean for our merchants.

My engineering team will sit in on interviews with our customers on a weekly basis and they’ll just chat and understand the pain points, because it’s really important that people can tie the experience of your customers to the metrics of the work that they’re doing.

Mojtaba: The job of the leader is to understand how pieces work together and track the outcome they’re working towards. If a platform team is going very slowly, that’s okay, because they are the platform team. They build, they make it so that other teams can go faster. At a high level, we’re still going fast, but I’m not going to tell the platform team they should be going as fast as these other non-platform teams. They fulfill different functions, and it’s the aggregate of those that really brings value. That interpretation, that context, is what leaders need to bring together.

Be Transparent: Share Metrics With Your Team

Sophie: Don’t hide your business metrics from the engineering team. If I could get anyone to take an action item from this, it would be that if your engineering team isn’t getting a copy of your Monthly Business Review (MBR), make that happen.

Romain: My team and I actually set the OKRs together to be aligned with the business OKRs. I ask the team, What do you think we can do to move the organization in accordance with the business OKRs? And that works very well because it goes back to the idea of making sure that the team understands the metrics and is motivated to reach that goal, instead of imposing from the top down that you have to fit these metrics into your day-to-day work.

Using Metrics for Compensation: What to Avoid and What to Reward

Sophie: You want people to take risks, you want people to fail, and you want the metrics to reflect the fact that things failed, but you don’t want to punish people. So there are a whole bunch of cultural interdependencies there that you have to manage.

The metrics can show us if the work is successful or not, but [for me], the metrics themselves don’t tie into people’s compensation. What ties into people’s compensation is how they approach solving the problem. I think that’s the only way you can do it if you’re going to build a culture where experimentation, risk, and failure are genuinely valid, as opposed to just being talking points on a wall.

Mojtaba: We don’t use the metrics in any way, even in the conversations we have about performance. This might sound controversial, but we don’t even have performance evaluation of individual contributors tied to compensation. We have competency levels tied to compensation.

Romain: Metrics are a foundational bed to talk with my engineers about how we can make our process better. And the compensation aspect would be more about where we are today, and where we want to be. Who are the contributors that are going to lead the team, and what will be the involvement and quality of their contribution? [I base compensation on] what their work brings to the table to help the company, more than on the metrics themselves.

To find out what else Sohpie, Mojtaba, and Romain had to say, listen to the full discussion here.

To find out how you can maximize engineering impact, request a consultation.

When evaluating their team’s health, engineering leaders might look to anecdotal data or rely on instinct. While valuable, these only tell part of the story.

By incorporating metrics into your leadership strategies, you can gain more precise insight into how your teams work. Data provides objective evidence that, if thoughtfully interpreted, can help leaders effectively manage teams to produce better software more efficiently.

Within a business, most departments — including marketing, finance, and HR — use some form of metrics to inform critical decisions. Engineering can also reap the benefits of data: these numbers can provide fact-based support when leaders are looking to create alignment or generate buy-in from key stakeholders. Data also offers engineering leaders a starting point for 1-on-1s, standups, and retros with developers — with metrics, meetings, and coaching conversations remaining objective and rooted in fact.

The idea of tracking and measuring developer productivity can feel like micromanaging, or worse, surveillance, and engineering leaders might be hesitant to introduce metrics to the team. In this ebook, we cover key strategies for introducing metrics in positive ways that avoid blame and promote psychological safety. (Hint: metrics give you insight about the work, not the developers themselves).

You’ll also find:

• How metrics improved delivery for engineering teams and addressed biases for real organizations
• Which metrics to track and how often
• How metrics can both reinforce and challenge your assumptions — and why both are good things

For a deeper dive into the fundamentals of engineering metrics, download our ebook.

Want to learn more about how to best address your team’s challenges? Request a consultation.

What’s the practical value of DORA metrics, and how are real engineering organizations using them? To find out, we invited a panel of engineering leaders & industry experts to share their experiences.  

Code Climate Senior Product Manager Madison Unell moderated the conversation, which featured:

• Scott Aucoin, Technology Director at Liberty Mutual
• Emily Nakashima, VP of Engineering at Honeycomb
• Karthik Chandrashekar, SVP of Customer Organizations at Code Climate

Over 45 minutes, these panelists discussed real-world experiences using DORA metrics to drive success across their engineering organizations.

Below are some of the key takeaways, but first, meet the panelists:

Scott Aucoin: I work at Liberty Mutual. We’ve got about 1,000 technology teams around the world. We’ve been leveraging DORA metrics for a while now. I wouldn’t say it’s perfect across the board, but I’ll talk about how we’ve been leveraging them. The role that I play is across about 250 teams working on our Agile enablement, just improving our agility, improving things like DevOps and the way that we operate in general; and our portfolio management work…from strategy through execution; and user experience to help us build intuitively-designed things, everything from the frontend of different applications to APIs. So finding ways to leverage DORA throughout those three different hats, it’s been awesome, and it’s really educating for me.

Emily Nakashima: I’m the VP of Engineering at a startup called Honeycomb. I manage an engineering team of just about 40 people, and we’re in the observability space. So basically, building for other developers, which is a wonderful thing. And we had been following along with DORA from the early days and have been enthusiasts and just made this switch over to using the metrics ourselves. So I’m excited to talk about that journey.

Karthik Chandrashekar: I’m the Senior VP of our Customer Organization at Code Climate. I have this cool job of working with all our engineering community customers solving and helping them with their data-driven engineering challenges. DORA is fascinating because I started out as a developer myself many years back, but it’s great to see where engineering teams are going today in a measurement and management approach. And DORA is central to that approach in many of the customer organizations I interact with. So I’m happy to share the insights and trends that I see.

Why did your organization decide to start using DORA?

Emily Nakashima: I first came to DORA metrics from a place of wanting to do better because we’re in the DevOps developer tooling space ourselves. Our executive team was familiar with the DORA metrics, and we had used them for years to understand our customers, using them as a tool to understand where people were in their maturity and how ready they would be to adopt our product…we had this common language around DORA…[At the same time,] our engineering team was amazing, and we weren’t getting the credit for it that we deserved. And by starting to frame our performance around the DORA metrics and show that we were DORA Elite on all these axes, I think it was a really valuable tool for helping to paint that story in a way that felt more objective rather than just me going, “We’ve got a great team.” And so far, honestly, it’s been pretty effective.

Scott Aucoin: Liberty Mutual being a 110-year-old insurance company, there are a lot of metrics. There are some metrics that I think we might say, “Okay, those are a little bit outdated now.” And then there are other ones that the teams use because they’re appropriate for the type of work the teams are doing. What we found to be really valuable about DORA metrics is their consistency…and the ability to really meet our customers and their needs through leveraging DORA metrics.

Karthik Chandrashekar: Speaking with a lot of CTOs and VPs of different organizations, I think there’s a desire to be more and more data-driven. And historically, that has been more around people, culture, teams, all of that, but now that’s transcended to processes and to data-driven engineering.

How did you go about securing buy-in?

Scott Aucoin: This has been pretty grassroots for us. We’ve got about 1,000 technology teams across our organization. So making a major shift is going to be a slow process. And in fact, when it’s a top-down shift, sometimes there’s more hesitancy or questioning like, “Why would we do this just because this person said to do this?” Now, all of a sudden, it’s the right thing to do. So instead, what we’ve been doing and what’s happened in different parts of our organization is bringing along the story of what DORA metrics can help us with.  

Emily Nakashima: The thing I love about Scott’s approach is that it was a top-down idea, but he really leveraged this bottom-up approach, starting with practitioners and getting their buy-in and letting them forge the way and help figure out what was working rather than dictating that from above. I think that it’s so important to really start with your engineers and make sure that they understand what and why. And I think a lot of us have seen the engineers get very rightly a little nervous about the idea of being measured. And I think that’s super-legitimate because there’s been so many bad metrics that we’ve used in the past to try to measure engineering productivity, like Lines of code, or PRs Merged. I think we knew we would encounter some of that resistance and then just a little bit of concern from our engineering teams about, what does it mean to be measured? And honestly, that’s something we’re still working through. I think the things that really helped us were, one, being really clear about the connection to individual performance and team performance and saying, we really think about these as KPIs, as health metrics that we’re using to understand the system, rather than something we’re trying to grade you on or assess you on. We also framed it as an experiment, which is something our culture really values.

DORA’s performance buckets are based on industry benchmarks, but you’re all talking about measuring at the company level. How do you think about these measures within your company?

Emily Nakashima: This was absolutely something that was an internal debate for us. When I first proposed using these, actually, our COO Jeff was a proponent of the idea as well. So the two of us were scheming on this, but there was really resistance that people pointed out that the idea of these metrics was about looking at entire cohorts. And there was some real debate as to whether they were meaningful on the individual team or company level. And we are the engineering team that just likes to supplement disagreements with data. So we just said, that might be true, let’s try to measure them and see where it goes. And I will say they are useful for helping us see where we need to look in more detail. They don’t necessarily give you really granular specifics about what’s going wrong with a specific team or why something got better or worse. But I do think that they have had a value just for finding hotspots or seeing trends before you might have an intuition that the trend is taking place. Sometimes you can start to see it in the data, but I think it was indeed a valid critique, ’cause we’re, I think, using them in a way that they’re not designed for.  

Something important about the DORA metrics that I think is cool is that each time they produce the report, the way they set the Elite and High and other tiers can change over time. And I like that. And you also see a lot of movement between the categories…And to me, it’s a really good reminder that as engineering teams, if we just keep doing the same thing over and over and don’t evolve our practices, we fall behind the industry and our past performance.

Scott Aucoin: I look at the DORA metrics with the main intent of ensuring that our teams are learning and improving and having an opportunity to reflect in a different way than they’re used to. But also, because of my competitive nature, I look at it through the lens of how we are doing, not just against other insurance companies, which is critical, but setting the bar even further and saying, technology worldwide, how are we doing against the whole industry? And it’s not to say that the data we can get on that is always perfect, but it helps to set this benchmark and say, how are we doing? Are we good? Are we better than anyone else? Are we behind on certain things?

Karthik Chandrashekar: One thing I see with DORA as a framework is its flexibility. So to the debate that Emily mentioned that they had internally, it’s a very common thing that I see in the community where some organizations essentially look at it as an organizational horizontal view of how the team is doing as a group relative to these benchmarks.

What pitfalls or challenges have you encountered?

Karthik Chandrashekar: From a pure trend perspective, best practice is a framework of “message, measure, and manage.” And not doing that first step of messaging appropriately with the proper context for the organization means that it actually can cause more challenges than not. So a big part of that messaging is psychological safety, bringing the cultural safety of, “this is to your benefit for the teams.” It empowers. The second thing is we all wanna be the best, and here’s our self-empowered way to do that. And then thirdly, I think, “how do we use this to align with the rest of the organization in terms of showcasing the best practices from the engineering org?”  

So the challenges would be the inverse of the three things I mentioned. When you don’t measure, people look at it as, “Oh, I’m being measured. I don’t wanna participate in this.” Or when you measure, you go in with a hammer and say, “Oh, this is not good. Go fix it.” Or then you do measure, and everything is great, but then when you are communicating company-wide or even to the board, then it becomes, hey, everything’s rosy, everything is good, but under the hood, it may not necessarily be…Those are some of the challenges I see.

Emily Nakashima: To me, the biggest pitfall was just, you can spend so much time arguing about how to measure these exact things. DORA has described these metrics with words, but how do you map that to what you’re doing in your development process?  

For us in particular, we have an hour-timed wait for various reasons because things roll to a staging environment first and get through some automated tests. Our deployment process is an hour. We will wait for 60 minutes plus our test runtime. So we can make incredible progress, making our test faster and making the developer experience better. And we can go from 70 to 65 minutes, which doesn’t sound that impressive but is incredibly meaningful to our team.  

And people could get focused on, “Wait, this number doesn’t tell us anything valuable.” And we had to just say, “Hey, this is a baseline. We’re gonna start with it.” We’re gonna just collect this number and look at it for a while and see if it’s meaningful, rather than spend all this time going back and forth on the exact perfect way to measure. It was so much better to just get started and look at it, ’cause I think you learn a lot more by doing than by finding the perfect metric and measuring it the perfect way.

Scott Aucoin: You’re going to have many problems, more than your DevOps practices. And Emily, I think the consistency around how you measure it is something we certainly have struggled with. And I would say in some cases, we still wonder if we’re measuring things the right way, even as we’ve tried to set a standard across our org. I’ll add to that, though, and say the complexity of the technology world, in general, is a significant challenge when you’re trying to introduce something that may feel new or different to the team or just like something else that they need to think about…You have to think about from the standpoint of the priorities of what you’re trying to build, the architecture behind it, security, the ability to just maintain and support your system, your quality, all of the different new technology that we need to consider ways to experiment all of that. And then, and we throw in something else to say, “Okay, make sure you’re looking at this too.” I think just from a time capacity and bandwidth perspective. It can be challenging to get folks to focus and think about, okay, how can we improve on this when we have so many other things we need to think about simultaneously?

What are you doing with DORA metrics now?

Scott Aucoin: It’s a broad spectrum. We’re doing all these fantastic things. Some groups are still not 100% familiar with what it means to look at DORA metrics or how to read them.  

It’s kind of a map and compass approach. You’re not only looking at a number; you’re able to see from that number what questions you have and how you can learn from it to map out the direction you want to go. So if you’re lagging behind in Deployment Frequency, maybe you want to think more about smaller batches, for example. So within our teams, we’re looking at it through that lens.  

And again, it’s not 100% of the teams. In fact, we still have more promotion and adoption to do around that, but we have the data for the entire organization. So we also look at it from the levels of the global CIO and monthly reports that are monthly operational reports that go to the global CIO. And while I can think about someone who I’ve gotten to know over the last few months, Nathen Harvey, who’s a developer advocate for Google’s DORA team, I have him in the back of my mind as I say this, as he would say, “The metrics are really for the teams.”  

We think about the value of it from the executive level as well. And when we think about the throughput metrics of Deployment Frequency and Lead Time for Changes, we can get a little bit muddy when you roll up thousands of applications to this one number for an exact, especially since many of those applications aren’t being worked on regularly. Some are in more of a maintenance mode. But when we can focus on the ones actively being worked on and think about trends, are we improving our Deployment Frequency or not? It can lead the CIO or any of the CIOs in the organization to ask the right questions to think about “what I can do to help this?” Especially when it comes to stability, regardless of whether an application is getting worked on actively today or not, we need stability to be there. So we really are looking at them at multiple levels and trying to be thoughtful about the types of questions that we ask based on the information we’re seeing.

Emily Nakashima: My example is the backward and wrong way to do this. I started by basically just implementing these myself for myself. And the first thing I did with them was to show the stakeholders that I was trying to paint this story too. And I think if you can start with getting practitioners to work with them, getting your software engineers to work with them first, tune them a little bit, and find them relevant, I honestly think that’s the best approach in the organization if you could do it. That wasn’t the situation I happened to be in, but I started with that, used them to radiate these high-level status numbers to other folks on the exec team and the board, and then started to roll them out team by team to allow for that customization.  

So we’re still in that process now, but I started to pull managers in one by one and go, hey, these metrics that I’m tracking, this is what they mean to me. Let’s sit down together and figure out what’s gonna be meaningful for your engineering team and how to build on this baseline here…Let’s build on top of it together.  

And we’re hoping to get into this quarter to have teams start working with them more directly and get more active in tuning and adding their metrics. We think about observability for systems, and we always want people to be adding instrumentation to their systems as they go. Each time you deploy a feature, add instrumentation that tells you whether or not it’s working. And we wanna bring that same approach to our engineering teams where we have these baseline metrics. If you don’t think they’re that good and they don’t tell you that much, then you go ahead and tell us what metric we add, and we’re gonna work together to build this higher fidelity picture that makes sense to you, and then also have that shared baseline across teams.

To hear more from our panelists, watch the full webinar here.

(Updated: February 26, 2024)

Born from frustration at the silos between development and operations teams, the DevOps philosophy encourages trust, collaboration, and the creation of multidisciplinary teams. As DevOps rose in popularity, DevOps Research and Assessment (DORA) began with the goal of gaining a better understanding of the practices, processes, and capabilities that enable teams to achieve a high velocity and performance when it comes to software delivery. The startup identified four key metrics — the “DORA Metrics” — that engineering teams can use to measure their performance in four critical areas.

This empowers engineering leaders, enabling them to benchmark their teams against the rest of the industry, identify opportunities to improve, and make changes to address them.

What is DORA?

DevOps Research and Assessment (DORA) is a startup created by Gene Kim and Jez Humble with Dr. Nicole Forsgren at the helm. Gene Kim and Jez Humble are best known for their best-selling books, such as The DevOps Handbook. Dr. Nicole Forsgren also joined the pair to co-author Accelerate in 2018.

The company provided assessments and reports on organizations’ DevOps capabilities. They aimed to understand what makes a team successful at delivering high-quality software, quickly. The startup was acquired by Google in 2018, and continues to be the largest research program of its kind. Each year, they survey tens of thousands of professionals, gathering data on key drivers of engineering delivery and performance. Their annual reports include key benchmarks, industry trends, and learnings that can help teams improve.

What are DORA metrics?

DORA metrics are a set of four measurements identified by DORA as the metrics most strongly correlated with success — they’re measurements that DevOps teams can use to gauge their performance. The four metrics are: Deployment Frequency, Mean Lead Time for Changes, Mean Time to Recover, and Change Failure Rate. They were identified by analyzing survey responses from over 31,000 professionals worldwide over a period of six years.

The team at DORA also identified performance benchmarks for each metric, outlining characteristics of Elite, High-Performing, Medium, and Low-Performing teams.

Deployment Frequency

Deployment Frequency (DF) measures the frequency at which code is successfully deployed to a production environment. It is a measure of a team’s average throughput over a period of time, and can be used to benchmark how often an engineering team is shipping value to customers.

Engineering teams generally strive to deploy as quickly and frequently as possible, getting new features into the hands of users to improve customer retention and stay ahead of the competition. More successful DevOps teams deliver smaller deployments more frequently, rather than batching everything up into a larger release that is deployed during a fixed window. High-performing teams deploy at least once a week, while teams at the top of their game — peak performers — deploy multiple times per day.

Low performance on this metric can inform teams that they may need to improve their automated testing and validation of new code. Another area to focus on could be breaking changes down into smaller chunks, and creating smaller pull requests (PRs)‌, or improving overall Deploy Volume.

Mean Lead Time for Changes

Mean Lead Time for Changes (MLTC) helps engineering leaders understand the efficiency of their development process once coding has begun. This metric measures how long it takes for a change to make it to a production environment by looking at the average time between the first commit made in a branch and when that branch is successfully running in production. It quantifies how quickly work will be delivered to customers, with the best teams able to go from commit to production in less than a day. Average teams have an MLTC of around one week.

Deployments can be delayed for a variety of reasons, including batching up related features and ongoing incidents, and it’s important that engineering leaders have an accurate understanding of how long it takes their team to get changes into production.

When trying to improve on this metric, leaders can analyze metrics corresponding to the stages of their development pipeline, like Time to Open, Time to First Review, and Time to Merge, to identify bottlenecks in their processes.

Teams looking to improve on this metric might consider breaking work into smaller chunks to reduce the size of PRs, boosting the efficiency of their code review process, or investing in automated testing and deployment processes.

Change Failure Rate

Change Failure Rate (CFR) is a calculation of the percentage of deployments causing a failure in production, and is found by dividing the number of incidents by the total number of deployments. This gives leaders insight into the quality of code being shipped and by extension, the amount of time the team spends fixing failures. Most DevOps teams can achieve a change failure rate between 0% and 15%.

When changes are being frequently deployed to production environments, bugs are all but inevitable. Sometimes these bugs are minor, but in some cases these can lead to major failures. It’s important to bear in mind that these shouldn’t be used as an occasion to place blame on a single person or team; however, it’s also vital that engineering leaders monitor how often these incidents happen.

This metric is an important counterpoint to the DF and MLTC metrics. Your team may be moving quickly, but you also want to ensure they’re delivering quality code — both stability and throughput are important to successful, high-performing DevOps teams.

To improve in this area, teams can look at reducing the work-in-progress (WIP) in their iterations, boosting the efficacy of their code review processes, or investing in automated testing.

Mean Time to Recovery

Mean Time to Recovery (MTTR) measures the ‌time it takes to restore a system to its usual functionality. For elite teams, this looks like being able to recover in under an hour, whereas for many teams, this is more likely to be under a day.

Failures happen, but the ability to quickly recover from a failure in a production environment is key to the success of DevOps teams. Improving MTTR requires DevOps teams to improve their observability so that failures can be identified and resolved quickly.

Additional actions that can improve this metric are: having an action plan for responders to consult, ensuring the team understands the process for addressing failures, and improving MLTC.

Making the most of DORA metrics

The DORA metrics are a great starting point for understanding the current state of an engineering team, or for assessing changes over time. But DORA metrics only tell part of the story.

To gain deeper insight, it’s valuable to view them alongside non-DORA metrics, like PR Size or Cycle Time. Correlations between certain metrics will help teams identify questions to ask, as well as highlight areas for improvement.

And though the DORA metrics are perhaps the most well-known element of the annual DORA report, the research team does frequently branch out to discuss other factors that influence engineering performance. In the 2023 report, for example, they dive into two additional areas of consideration for top-performing teams: code review and team culture. Looking at these dimensions in tandem with DORA metrics can help leaders enhance their understanding of their team.

It’s also important to note that, as there are no standard calculations for the four DORA metrics, they’re frequently measured differently, even among teams in the same organization. In order to draw accurate conclusions about speed and stability across teams, leaders will need to ensure that definitions and calculations for each metric are standardized throughout their organization.

Why should engineering leaders think about DORA metrics?

Making meaningful improvements to anything requires two elements — goals to work towards and evidence to establish progress. By establishing progress, this evidence can motivate teams to continue to work towards the goals they’ve set. DORA benchmarks give engineering leaders concrete objectives, which then break down further into the metrics that can be used for key results.

DORA metrics also provide insight into team performance. Looking at Change Failure Rate and Mean Time to Recover, leaders can help ensure that their teams are building robust services that experience minimal downtime. Similarly, monitoring Deployment Frequency and Mean Lead Time for Changes gives engineering leaders peace of mind that the team is working quickly. Together, the metrics provide insight into the team’s balance of speed and quality.

Learn more about DORA metrics from the expert himself. Watch our on-demand webinar featuring Nathen Harvey, Developer Advocate at DORA and Google Cloud.