Objective data is a necessary complement to engineering leadership. By digging into the key metrics of the Software Development Life Cycle (SDLC), leaders can better assess and improve engineering processes and team performance.

Each organization may choose to focus on a different set of engineering metrics based on their goals. Our collaboration with thousands of organizations has revealed a key set of metrics that have proven valuable time and again, including Review Cycles, a key factor in Pull Request (PR) Cycle Time.

In this blog, we’ll dig into the importance of the Review Cycles metric, what it can tell you about the health of your teams and processes, and how to improve it.

What is the Review Cycles metric?

The Review Cycles metric refers to Pull Request reviews, and measures the number of times a PR goes back and forth between an author and reviewer.

The Pull Request review process is an essential component of PR Cycle Time, which measures the time from a first commit in a pull request being authored to when that PR is merged, and looking at Pull Request data can help leaders understand teams’ Time to Market and baseline productivity over time.

Whether the PR is created to ship a new feature or fix a bug, for example, the proposed change needs to be reviewed by a member of the team before it is merged. If that PR gets approved and merged with no further interaction from the author, then the Review Cycle is 1 — the changes were reviewed and approved in a single cycle.

If a PR does not get approval and requires changes, then the author must make an additional commit. The reviewer then checks the new version before it is approved. In this scenario, the number of Review Cycles is 2. Of course, this number increases as the PR is passed back and forth between author and reviewer.

Why should teams measure Review Cycles?

Pull Requests require software engineers to context switch and focus on one particular line of work. When this happens often, and Review Cycles are high, the PR review process can spread engineers’ attention too thin. It can also become a bottleneck to shipment.

Finding the cause of a slow PR review process

There are various reasons why Review Cycles may be high:

• There are differing ideas around what “done” means.
• There’s misalignment around what kind of changes are expected to come out of a review process, or
• There are conflicting opinions about how a solution should be implemented.

If the Review Cycle metric is high for a particular submitter, it could mean that they’re struggling with the codebase or dealing with unclear requirements.

Be mindful of potential biases in the PR review process

While data can offer a concrete number of Review Cycles for a specific PR, it does not tell the whole story. If a specific developer has a high number of Review Cycles tied to their work, engineering leaders should open a dialogue with both the developer and reviewer to pinpoint potential cause.

Sure, they may be struggling with the codebase because they are new to it, but it’s also possible that their teammates may be unfairly scrutinizing their work. There are a number of potential biases that could be skewing perception of ICs and their work. One engineering leader was able to use data from Code Climate's platform to uncover that a woman engineer’s PRs were moving disproportionately slower than those of her male counterparts and concluded that bias was a problem within the team.

To identify what’s affecting your teams’ Review Cycles and PR review process overall, examining the data will give you a starting point to have a conversation with the team and ICs involved so you can align on processes.

Review Cycles can help leaders assess onboarding and Ramp Time

When a developer first joins a team, it may take time for them to get up to speed. Looking at Review Cycles in a Software Engineering Intelligence (SEI) platform allows leaders to observe changes and progress over time. With these insights, you can measure the ramp time for newly onboarded engineers by observing whether their Review Cycles decrease over time. If Review Cycles for new hires are not decreasing at the expected rate, leaders may want to further investigate the efficacy of onboarding processes and ensure that new developers have the tools they need to excel in their roles.

Using data to improve Review Cycles and help engineers get unstuck

When you use a Software Engineering Intelligence (SEI) platform like Code Climate, you can gain visibility into the entire PR review process. The Analytics module in Code Climate's platform is a good place to investigate PR review processes. You’ll want to run a query for Review Cycles, or the number of times a Pull Request has gone back and forth between the author and reviewer. Here’s how:

Click on the arrow next to the Review Cycle number to see all of Hecate’s individual PRs from the selected timeframe. Sort the number of Review Cycles from high to low and start with the ones that have been open the longest. In this case, the top two PRs, which have both undergone 4 Review Cycles and are still open, are worth bringing to a standup, retro, or 1-on-1.

Talk to your team to improve your PR review process

Prepare for a standup, retro, or 1-on-1 with developers by taking a look at Pull Request data. This will allow you to be more informed ahead of a meeting, and be able to focus specifically on units of work rather than developers or teams themselves.

Ask your team questions about specific PRs with high Review Cycles to uncover where the misalignment is happening. Work with the team to find solutions to limit the amount of times a PR is volleyed back and forth by establishing what is expected in a review, how solutions should be implemented, and when a review is complete. Document best practices for the Pull Request review process to use as a reference in the future.

How many Review Cycles should you target in the PR review process?

Code Climate has produced proprietary benchmarks that engineering leaders can use. We have found that the top 25% of organizations have an average of 1.1 Review Cycles or less, whereas the industry average is 1.2 cycles. If Review Cycles are above 1.5, it’s time to investigate why.

Review Cycles are one of many critical metrics that engineering leaders can measure to understand and improve team health and processes. By looking at Review Cycles alongside other Pull Request-related metrics, you can uncover the cause of a slowdown and make informed decisions towards improvement. The data is only a starting point, however — it’s essential that leaders speak directly to teams in order to find a sustainable solution.

Request a consultation to learn how measuring engineering metrics can lead to faster software delivery and healthier teams.

What is engineering team health?

Maximizing engineering team health is critical to the success of the business, and there is a strong correlation between healthy teams and productivity. Team health in engineering is determined by a variety of factors including:

• A belief that work is meaningful, evenly distributed, and not wasted
• Strong team collaboration
• Psychological safety
• Alignment on priorities
• Managers’ support of individual and team development

To improve team health and maximize the engineering teams’ impact on the business, engineering leaders need visibility into key stages of the engineering process and the factors, above, that determine team health.

As part of our survey in collaboration with CTO Connection, we asked more than 200 engineering leaders what areas of team health and performance they are most interested in gaining visibility into.

Here’s a breakdown of those responses:

In this ebook, you’ll learn how to gain visibility into engineering team health and performance to improve efficiency and developer experience to maximize engineering impact on the business.

Download the ebook for free here.

Performance reviews in engineering are often tied to compensation. End-of-year and mid-point check-ins can be great opportunities to discuss individual goals and foster professional growth, but too often they are used as ways to assess whether direct reports are eligible for raises or promotions based on their ability to hit key metrics.

For engineering leaders, a more valuable way to use performance reviews is as a structured opportunity to have a conversation about a developer’s progress and work with them to find ways to grow in their careers through new challenges. This benefits both developers and organizations as a whole — developers are able to advance their skills, and companies are able to retain top engineers who are continually evolving. Even senior engineers look for opportunities for growth, and are more likely to stay at an organization that supports and challenges them.  

The key to achieving this is by focusing on competency, rather than productivity measurements, when it comes to evaluating compensation and performance. But how do we define competency in engineering?

What is competency in engineering?

Where productivity might measure things like lines of code, competency looks at how an engineer approaches a problem, collaborates with their team to help move things forward, and takes on new challenges.

Questions to consider when evaluating a developer’s competency might include:

• How thoughtful are their comments in a Code Review?
• Do they make informed decisions based on data?
• How do they navigate failure?
• How do they anticipate or respond to change?

Engineering leader Dustin Diaz created a reusable template for evaluating competency in engineering, which he links to in this post. The template borrows from concepts in The Manager’s Path, by author and engineering leader Camille Fournier, and is modeled after the SPACE framework, and outlines levels of competency for engineers at different levels of seniority. The matrix can be helpful for leaders looking to hone in on areas like collaboration, performance, and efficiency/quality. It includes markers of competency for different tiers of engineers, including anticipating broad technical change, end-to-end responsibility on projects, and taking initiative to solve issues.

Performance review essentials

We’ve addressed how performance reviews during a difficult year can be especially challenging. Yet no matter the circumstances, there are principles of a successful performance review that will always be relevant.  

Reviews should always include:

• Compassion and empathy — With the understanding that employees could be under a lot of stress, a performance review shouldn’t add to any pressure they’re experiencing.
• Specific units of work — In order to provide useful feedback, bring examples of work to discuss during a review. By focusing on the work itself, the review remains objective, and can open the door for a coaching conversation.
• Objective data to check biases — There are many common performance review biases, as well as ways to correct for them by contextualizing data, checking assumptions, and taking circumstance into account. It’s important not to let unconscious biases inform your analysis of a developer’s work.

Aligning developer and company incentives

When performance reviews are based on hitting productivity benchmarks that are implicitly linked to compensation, developers might be less focused on ambitious goals and more on checking boxes that they believe will earn them a raise; rather than challenging themselves, they will likely be incentivized to play it safe with easy-to-achieve goals.

Encouraging a focus on competency invites engineers to make decisions that have more potential to move the needle in an organization, and to take risks, even if they could lead to failure.

During our webinar on data-driven leadership, Sophie Roberts, Director of Engineering at Shopify shared why rewarding productivity over growth could have adverse effects: “You end up in a situation where people want to get off projects that they don’t really think they have executive buy-in, or try and game the work they’re doing,” Roberts said. “I’ve canceled people’s projects and promoted them the next month, because how they were approaching the work was what we expect from a competency level of people who are more senior…They may try to get work that is a more sure shot of moving a metric because they think that’s what’s going to result in their promotion or their career progression.”

An emphasis on competency can improve the following:

• Risk-taking. If engineering leaders foster psychological safety within teams, developers are more likely to take risks without the fear of being penalized for failure. If metrics are not tied to compensation, developers will likely be more ambitious with their goal-setting.
• Flexibility and adaptability. Failure and change are both inevitable in engineering organizations. Instead of trying to avoid mistakes, emphasize the importance of learning and of implementing thoughtful solutions that can have a lasting impact.
• Upskilling and developer retention. Investing in the professional development of an engineer is a vote of confidence in their abilities. Engineering leaders can boost retention through upskilling or by enabling individuals to grow into more senior positions.

Using metrics to assess competency

Data-driven insights can provide engineering leaders with objective ways to evaluate developer competency, without tying metrics directly to compensation. They can help you assess a developer’s progress, spot opportunities for improvement, and even combat common performance review biases.

One effective way to use metrics in performance reviews is to quantify impact. In our webinar on data-driven performance reviews, Smruti Patel, now VP of Engineering at Apollo, shared how data helps IC’s on her team recognize their impact on the business during self-evaluations.

“It comes down to finding the right engineering data that best articulates impact to the team and business goals. So if you think about it, you can use a very irrefutable factor, say, ‘I shipped X, which reduced the end-to-end API latency from 4 or 5 seconds to 2.5 seconds. And this is how it impacted the business,” she said.

In the same discussion, Katie Wilde, now Senior Director of Engineering at Snyk Cloud, shared how metrics explained a discrepancy between one engineer’s self-evaluation and her peer reviews. This engineer gave herself a strong rating, but her peers did not rate her as highly. When Wilde dug into the data, she found that it was not a case of the individual being overconfident, but a case of hidden bias — the engineer’s PRs were being scrutinized more heavily than those of her male counterparts.

In both instances, data helped provide a more complete picture of a developer’s abilities and impact, without being directly tied to performance benchmarks or compensation.

Defining success in performance reviews

Overall, metrics are able to provide concrete data to counteract assumptions, both on the part of the reviewer and the engineers themselves. By taking a holistic approach to performance reviews and contextualizing qualitative and quantitative data, including having one-on-one conversations with developers, leaders can make more informed decisions about promotions and compensation for their teams.

Keep in mind that performance reviews should be opportunities to encourage growth and career development for engineers, while gaining feedback that can inform engineering practices.

Most importantly, rewarding competency is an effective way to align developer goals with business goals. This way, leaders are invested in the growth of valuable members of their team who make a significant impact, while engineers are recognized for their contributions and able to grow in their careers.

For engineering teams, disruption to the business can have a significant impact on the ability to deliver and meet goals. These disruptions are often a result of reprioritization and budget changes on an organizational level, and are amplified during times of transition or economic instability.

In a survey led by CTO Craft in partnership with Code Climate, engineering leaders were asked to name the main cause for disruption to their businesses in 2022, and to offer their predictions for productivity challenges in 2023. The survey also included questions about engineering leadership in particular, including how leaders intend to keep engineering teams motivated in the coming year, and how their leadership has been impacted by disruptive times.

In total there were 114 respondents, comprised mainly of CTOs, followed by Engineering Managers, then heads of technology, development, and engineering.

Read on for the key takeaways, and see the full survey results on CTO Craft.

Hiring challenges ranked as the #1 cause for disruption

Attracting and retaining talented software engineers is top of mind for many engineering leaders, and with developers in short supply but high demand, this remains a challenge for organizations. Over half of survey respondents said hiring challenges were the leading cause of business disruption in 2022.

Many survey respondents said that recruiting top talent will continue to be a challenge in 2023.

Other common responses were reprioritization of business objectives, followed by a drop in revenue. Over half of respondents (54%), predict that issues with budgets will be a threat to productivity in 2023.

Leaders aim to assign more engaging work in 2023

Nearly half (forty-five percent) of respondents said that they plan to motivate engineers in 2023 with more engaging work, followed by twenty percent of respondents who said they would focus on developers’ career paths. The remaining eight percent said they will use compensation as a motivator next year.

Teams are still getting work done

Despite these challenges, 70% of survey participants said that they almost always deliver on business commitments.

In terms of identifying root causes of going off track and not delivering on commitments, 60% of respondents said they can assess the problem with relative ease, while 25% said it’s difficult for them to do so.

Learn more about CTO Craft on their website.