Madison Unell

Engineering teams are more distributed than ever. Nearly 70% of active engineering positions are open to remote applicants, and many companies have a mix of remote, hybrid, in-person, and contracted employees. So, how do engineering leaders measure performance uniformly across all of these teams? By creating a framework for understanding performance, asking the right questions, and using data to answer them.

Creating a Performance Framework

Engineering leaders want to mitigate surprises before they impact software delivery or the business. It’s not enough to make decisions based on gut feel or anecdotal performance reviews — especially when an engineering organization is made up of multiple teams with unique working styles and deliverables. To truly understand performance across teams, leaders must establish which metrics are important to their company and create a framework to measure them. Establishing a performance framework ensures that leaders are measuring engineering teams in a consistent and equitable way so they can identify and resolve bottlenecks faster to optimize the flow of work.

Tailoring a Framework for Your Team

Using a common framework like DORA is a great starting point, but leaders must tailor measurement to the needs of their unique team. Traditional engineering metrics, and even frameworks like DORA, can overrotate on the quantity of code that’s produced and underrotate on the quality of that code, how efficiently it was written, or how effectively it solves a specific problem. Solely measuring quantity can result in bloated, buggy code because engineers may prioritize simple features they can get out the door quickly rather than spending time on more complex features that can move the needle for the business.

Adding metrics and context that apply to your specific team can provide a more accurate look at engineering performance. For example, to understand team productivity, leaders may look at engineering metrics like Mean Lead Time for Change (MLTC) alongside Cycle Time. If MLTC is high, it could indicate that Cycle Time is also high. These metrics can be viewed in tandem with other metrics like Time to Open, Time to Merge, and Time to First Review to understand where changes need to be made. These metrics can then be compared across teams to understand which teams are performing well and establish best practices across the organization.

Monthly Engineering Metrics to Understand Team Performance

Data-driven insights can provide engineering leaders with objective ways to evaluate developer competency, assess individual progress, and spot opportunities for improvement. While quarterly KPIs and annual performance reviews are great goalposts, managers are constantly thinking about how their teams are progressing toward those targets. Reviewing engineering metrics on a monthly basis is a good way to assess month-over-month progress and performance fluctuations on an individual level and a team level. Which metrics a team considers depends on its defined framework and overall company goals. Here are a few to consider:

PRs Merged vs. PRs Reviewed

Looking at these metrics together can show how the two key responsibilities of writing and reviewing code are spread across a team.

Review Coverage vs. Review Influence

This helps leaders understand what amount of thoroughness of Code Reviews results in a desired action.

Review Cycles vs. Cycle Time

To understand the effect that back-and-forth cycles in Code Review have on shipping speed, leaders can look at Review Cycles vs. Cycle Time.

Impact vs. Rework

Comparing Impact and Rework will show which teams are making the most significant changes to the codebase and how efficiently they are doing so.

Communicating Engineering Team Performance

Understanding and communicating engineering team performance is an effective way to ensure teams are aligned and that all requirements are understood and met. Making this a standard across the engineering organization — especially in a distributed or hybrid environment — is essential to its success. How leaders communicate their findings is equally important as gathering the information. When feedback is a fundamental part of a blameless team culture, team members understand that feedback is critical to growing as a team and achieving key goals, and will likely feel more secure in sharing ideas, acknowledging weaknesses, and asking for help. Leaders can tailor the questions listed above to meet the unique needs of their organizations and use engineering metrics as a way to understand, communicate, and improve team performance.

Understanding the performance of engineering teams at large companies is no easy feat. For many, this is due to the diversity of processes across teams and the resulting complexity of collecting consistent data. Companies need a standard way of measuring and understanding engineering performance and a common language to communicate it to company leaders and individual contributors. In this article, we’ll discuss how large organizations leverage DORA metrics to do just that.

Using DORA DevOps metrics to communicate with leadership

In startups, engineering actions are often more directly linked to business goals, making it possible for leaders to understand what engineering is doing and communicate its impact. For example, if a startup is launching its flagship product, contributors from sales, marketing, and product management collaborate with engineering, often with executive support and oversight, to ensure the business goals are met. They consider what the product does, how it works, why it matters, who will benefit from it, and how it will be sold. Startups often have shared key performance indicators (KPIs) and operate on a single timeline.

Now scale that same workflow across dozens of teams launching and maintaining different products on varying timelines. While engineering will aim to align goals with business objectives, those goals may vary from team to team, and success will look different for each group. That’s why it’s crucial to establish which metrics are important to the company as a whole and create a framework to measure them. Establishing a framework to measure engineering success ensures that managers are measuring teams in a consistent and equitable way so they can identify and resolve bottlenecks to optimize the flow of work.

Using a framework like DORA is a great place to start. The four DORA metrics, Deployment Frequency (DF), Mean Lead Time for Changes (MLTC), Mean Time to Recover (MTTR), and Change Failure Rate (CFR), can be communicated to leadership to give them a holistic view of how the engineering organization is performing.When implementing DORA, it’s important that organizations start by agreeing on how these metrics will be measured. Unified calculations and standards (i.e. company-wide agreement on what is considered an "outage") are critical for measuring effectively throughout an organization. Standardizing on these four metrics and how they will be measured provides uniformity across teams and creates a common language between engineering and company leadership.

DORA metrics help teams balance speed and stability and are good big-picture checks into the health of the organization. Managers can use DORA to see how things are trending over time and spot when a team isn't working as expected. However, they must keep in mind that while it can be instructive to benchmark teams within an organization by identifying what high-performing teams are doing that others can learn from, it's important to note the context. Managers must understand that teams tasked with different kinds of work and different projects will naturally have variations in their DORA metrics, which is normal and expected.

Supporting developers with granular engineering metrics

Using DORA as the foundational framework across teams lets engineering leaders understand how a team is doing within the context of the broader organization and drill down into data from a specific team to learn more about the way it's working. DORA metrics can highlight areas worth attention, serving as a starting point from which managers and their teams can investigate the efficacy of their processes and make changes, then track the impact of those changes.

To do this, they can add context to the four DORA metrics and pair them with complementary metrics to get more insight into what’s happening with individual teams and what improvements might be useful. Common metrics pairings include:

• Change Failure Rate and Unreviewed Pull Requests. If a high CFR is correlated to a high percentage of unreviewed PRs, managers may consider adjusting the process to prevent unreviewed PRs from being merged and causing issues.
• Deployment Frequency and PR Size. If DF is low, managers can use PR size to investigate it. If large PRs are correlated with a low DF, they can coach team members to break work into smaller PRs to see if DF improves.
• Mean Time to Recovery and Revert Rate: A long MTTR could indicate a high Revert Rate, therefore disrupting production and lengthening the time it takes to recover from an unplanned outage or defect. If there’s a correlation, managers can drill down into each revert and see whether the issue is a defect or an undesirable change.
• Mean Lead Time for Changes and Cycle Time. If MLTC is high, it could indicate that Cycle Time is also high. Managers can view these metrics in tandem and dig deeper into related metrics like Time to Open, Time to Merge, and Time to First Review to find the root cause.

How DORA software creates a common language from ICs to the C-suite

Large companies can benefit from a Software Engineering Intelligence (SEI) platform to understand engineering performance at every level of the organization. It allows engineering managers to standardize measurement and reporting on the four DORA metrics to communicate performance to company leadership and ensure that the pace of work meets business needs. Managers can also combine DORA with other engineering metrics in their SEI platform to communicate with their teams to ensure they have what they need to be successful and roadblocks are quickly identified and removed.

Without a strong framework and a centralized platform to measure it, engineering data can become a tangled mess as the number of engineers at a company increases. Measuring DORA and complimentary engineering metrics in an SEI platform helps leaders make sense of their data to ensure that engineering work is optimized and aligned with business objectives.

To find out more about how an SEI platform can benefit leaders at large organizations, request a consultation.

Google Cloud’s DevOps Research and Assessment (DORA) team’s 2023 Accelerate State of DevOps report examines the relationship between user-facing strategies, process enhancements, and culture and collaboration and its impact on engineering performance.

The DORA team re-emphasizes the importance of the four key DORA metrics, important benchmarks for gauging the speed and stability of an engineering organization. These metrics are a baseline for any engineering team looking to improve, and are a gateway for a more data-driven approach to engineering leadership. Pairing DORA metrics with other engineering metrics can unlock critical insights about a team’s performance.

However, the 2023 report makes significant strides in broadening out their approach to measurement. It recognizes that the four foundational metrics are an essential starting point, but also highlights additional opportunities for enhancing engineering performance. As teams continue on their data-driven journey, there are more dimensions of team health to explore, even in areas that don’t initially seem like they would lend themselves to measurement.

Two such areas highlighted in this year’s report are code review — an important window into a team’s ability to communicate and collaborate — and team culture.

Faster Code Reviews Accelerate Software Delivery

Notably, the report’s most significant finding indicates that accelerating the code review process can lead to a 50% improvement in software delivery performance. While many development teams are disappointed with their code review processes, they simultaneously recognize their importance. Effective code reviews foster collaboration, knowledge sharing, and quality control. And, according to the report, an extended time between code completion and review adversely affects developer efficiency and software quality.

At Code Climate, we’ve identified a few key strategies for establishing an effective code review process. First, it’s important for teams to agree on the objective of review. This ensures they know what type of feedback to provide, whether it’s comments pertaining to bug detection, code maintainability, or code style consistency.

It’s also important for leaders to create a culture that prioritizes code review. Ensure that your teams understand that in addition to ensuring quality, it also facilitates knowledge sharing and collaboration. Rather than working in a silo to ship code, developers work together and help each other. Outlining expectations — developers are expected to review others’ code, in addition to writing their own — and setting targets around code review metrics can help ensure it’s a priority.

Code Review Metrics

Leaders at smaller companies may be able to understand the workings of their code review process by talking to team members. However, leaders at enterprises with large or complex engineering teams can benefit from using a Software Engineering Intelligence (SEI) platform, like Code Climate, to act on DORA’s findings by digging into and improving their code review processes.

An SEI platform offers essential metrics like Review Speed, which tracks the time it takes from opening a pull request to the first review submission, and Time to First Review, which represents the average time between initiating a pull request and receiving the first review. These metrics can help leaders understand the way code is moving through the review process. Are PRs sitting around waiting for review? Are there certain members of the team who consistently and quickly pick PRs up for review?

Reviewing these metrics with the team can help leaders ensure that team members have the mindset — and time in their day — to prioritize code review, and determine whether the review load is balanced appropriately across the team. Review doesn’t have to be completely equally distributed, and it’s not uncommon for more senior team members to pick up a greater proportion of PRs for review, but it’s important to ensure that the review balance meets the team’s expectations.

A Note About Bottlenecks

The DORA report noted that even if code reviews are fast, teams are still unlikely to improve software delivery performance if speed is constrained in other processes. "Improvement work is never done,” the report advises, “find a bottleneck in your system, address it, and repeat the process."

Data from an SEI platform can help leaders continue the work of identifying and removing bottlenecks. Armed with the right information, they can enhance visibility and support informed decision-making, enabling them to detect bottlenecks in the software development pipeline and empower developers to collaborate on effective solutions. Equipped with the right data, leaders can validate assumptions, track changes over time, identify improvement opportunities upstream, scale successful processes, and assist individual engineers in overcoming challenges.

Fostering a Healthy Team Culture

Though the DORA team highlights the importance of effective processes, it also found that culture plays a pivotal role in shaping employee well-being and organizational performance. They found that cultivating a generative culture that emphasizes belonging drives a 30% rise in organizational performance. Additionally, addressing fair work distribution is crucial, as underrepresented groups and women face higher burnout rates due to repetitive work, underscoring the need for more inclusive work cultures. To retain talent, encourage innovation, and deliver more business value, engineering leaders must prioritize a healthy culture.

Just as they can provide visibility into processes, SEI platforms can give leaders insight into factors that shape team health, including leading indicators of burnout, psychological safety, and collaboration, and opportunities for professional development.

It’s fitting that the DORA report identifies code review as a process with a critical link to team performance – it’s a process, but it also provides insight into a team’s ability to collaborate. Metrics like Review Speed, Time to first Review, and Review Coverage, all send signals about a team’s attitude toward and facility with collaboration.

Other data can raise flags about team members who might be headed towards burnout. The Code Climate platform's Coding Balance view, for example, highlights the percentage of the team responsible for 80% of a team’s significant work. If work is uneven — if 10% of the team is carrying 80% of the load — it can indicate that some team members are overburdened while others are not being adequately challenged.

Data is the Key to Acting on DORA Findings

The findings from the DORA report are clear: even those teams that are successfully using the four DORA metrics to improve performance should look at other dimensions as well. Prioritizing process improvements like code reviews and promoting a healthy team culture are instrumental to performance — and data can help leaders learn more about these aspects of their team. Request a consultation to find out more about using an SEI platform to action the 2023 DORA findings.

A good leader can respond to issues and course correct in the moment, but a great leader is proactive about staying ahead of the curve. With the help of data provided by Engineering Intelligence tools, an Engineering Manager can gain visibility into the development pipeline and stay equipped with the knowledge needed to cut problems off at the pass. No matter where an EM is in their professional lifecycle, a proactive one can prioritize more successfully, strengthen coaching strategies, and boost team effectiveness in the short term, mid-term, and long term.  

Short Term Strategy: Spot Risk and Prevent Blockages

A lot of dedication goes into keeping sprints on track and software delivery on schedule. With so many moving parts in the development pipeline, an EM may find it tough to determine what needs their attention first, making it challenging to triage risks. However, by using a Software Engineering Intelligence platform — like Code Climate —  that conveys insights based on aggregated data, an EM can swiftly analyze coding progress and prioritize their task list to focus on what’s most important.  

For example, an EM can assess PR and Commit data to surface at-risk work — work that could benefit from their time and attention. If a Commit has had several days of inactivity and work on the associated PR has seemingly halted, it may indicate that the scope of the task is not clear to ICs, and that they require guidance. Or it may be a sign of task-switching, where too much Work In Progress pulls an IC’s focus and makes it difficult to complete work.

This is where data from a Software Engineering Intelligence platform is critical, as it can signal to a manager that a specific PR or Issue needs attention. Code Climate enables EMs to set Risk Alerts for PRs based on custom criteria, since risk thresholds vary from team to team. From there, the EM can use that information in standups, retros, and other conversations with ICs to help identify the root cause of the blocker and provide coaching where needed.

Mid-term Strategy: Improve Collaboration

As a proactive leader, an EM must understand the nuances of collaboration between all parties to ensure ICs and teams are working together effectively and prioritizing issues that are aligned with company goals. If teams fail to work cohesively, roadmaps may be thrown off course, deadlines may be missed, and knowledge silos may be created. Using their Engineering Intelligence tools, an EM can easily surface the quantitative data needed to gain visibility into team collaboration and interdepartmental alignment.  

When it comes to collaboration on their team, an EM might want to look at review metrics, like Review Count. Viewing the number of PRs reviewed by each contributor helps an EM understand how evenly reviews are distributed amongst their team. Using these insights, a manager can see which contributors are carrying out the most reviews, and redistribute if the burden for some is too high. Doing so will not only help keep work in balance, but the redistribution will expose ICs to different parts of the codebase and help prevent knowledge silos.

To look at collaboration between teams, an EM can rely on quantitative data to help surface signs of misalignment. Looking at coding activity in context with information from Jira can help an EM identify PRs that signal a lack of prioritization, such as untraceable or unplanned PRs. Since these PRs are not linked back to initial project plans, it may indicate possible misalignment.  

Long Term Strategy: Support Professional Growth and Improve Team Health

A proactive EM also needs to identify struggling IC’s, while simultaneously keeping high performers engaged and challenged to prevent boredom. This starts with understanding where each individual IC excels, where they want to go, and where they need to improve.

Using quantitative and qualitative data, an EM can gain a clearer understanding of what keeps each IC engaged, surface coaching opportunities, and improve collective team health. Qualitative data on each IC’s coding history — Commits, Pushes, Rework, Review Speed — can help signal where an IC performs well and surface areas where it might be useful for an EM to provide coaching. An EM can then use qualitative data from 1 on 1’s and retros to contextualize their observations, ask questions about particular units of work, or discuss recurring patterns.

For example, if an EM notes high levels of Rework, this signals an opportunity to open up a meaningful discussion with the IC to surface areas of confusion and help provide clarity. Or, an EM might see that an IC has infrequent code pushes and can coach the IC on good coding hygiene by helping them break work down into smaller, more manageable pieces that can be pushed more frequently.

Using a combination of both data sets, a manager can initiate valuable dialogue and create a professional development roadmap for each IC that will nurture engagement and minimize frustration.

Proactivity as an EM – The Long and Short of It

Proactivity is a skill that can be developed over time and enhanced with the help of data. Once empowered with the proper insights, EMs can more effectively monitor the health of their sprints, meet software delivery deadlines, keep engineers happy, and feel confident that they are well-informed and can make a marked, positive impact.