Stop Code Review from Bottlenecking Shipping

This post is the third article in our Tactical Guide to a Shorter Cycle Time five-part series. Read the previous post here.

Every hour that a Pull Request spends awaiting review represents waste–but also threatens to slow down several subsequent lines of work.

An engineer who’s waiting on a code review will move onto the next Pull Request to feel productive until it too gets stuck in the queue. If this pattern continues, reviewers will be faced with a daunting stockpile of unreviewed Pull Requests, and each developer will be juggling three or four simultaneous lines of work. Code review will become a universally painful experience.

A poor Time to First Review (the time it takes for a reviewer to pick up a review) can jeopardize the effectiveness of your entire code review process. Here’s what you can do to mitigate the first significant bottleneck in the software delivery pipeline.

Benchmarking Success

First, you’ll want to define what a “low” Time to First Review means, across the industry and in the context of your organization.

In our analysis of over 19k contributors’ data we uncovered the following benchmarks for Time to First Review:

• The top 25% of engineering organizations get Pull Requests reviewed in under 4 hours.
• The industry median is about one day.
• The bottom 25% take over a day and a half.

You want to lower this metric as much as you can, without it coming at a cost to the focus and productivity of the reviewers. This delicate tradeoff means that you’ll want to understand the dynamics of your specific team before setting metrics-based goals.

First, look at this metric for your organization and team-by-team. Once you know what’s “normal” for your organization, you can further investigate two kinds of outliers:

• Teams with fast reviewers, who have found a way to create an effective process and juggle a difficult team dynamic. Their processes may be worth sharing with the rest of the organization.
• Teams with inconsistent reviewers, who are frequently blocked and struggling to get to code reviews quickly.

From here, dive into individual and team metrics to better understand precisely why your Time to First Review is high or inconsistent.

Time to First Review Diagnostics

Before diagnosing this particular phase of the software delivery process, make sure that it’s not a symptom of an earlier process. (Read our previous post about leveraging Time to Open to better understand how inefficiencies during the first stage can have significant, negative downstream effects.)

If everything earlier in the PR process is streamlined, however, there are three data points you can look at to fully diagnose this slowdown.

Pull Request Size

Pull Request Size is a byproduct of a high Time to Open and usually points to inconsistencies with how your team manages Pull Request scope.

For many teams, opening a Pull Request doesn’t indicate that the changeset is ready for review. Team members tend to continue making changes before the Pull Request gets into the hands of a reviewer.

At this time, a Pull Request can grow, deterring reviewers from picking it up because either A) they aren’t sure if it’s ready, or B) because it has grown in complexity and is, therefore, more difficult to review.

Velocity surfaces the oldest and most active open PRs, so your frontline managers can mitigate issues before they grow out of hand.

Encourage managers to check in on individual Pull Requests that have been open for longer than a day (but not yet reviewed). If they find patterns among teams or individuals that indicate a lack of scope management, they’ll want to coach their teams on how to control the scope from the onset of the changeset, by, for instance, giving PRs titles that are discrete and specific.

Review Workload Balance

Look at Review Involvement (i.e., the percent of reviews that reviewers participate in) across the team to understand how the burden of code review is distributed. If the same few people are handling all the reviews, work will start piling up, creating a lasting bottleneck.

Velocity shows how involved each team member is on code reviews, next to the outcomes of those reviews.

Poor distribution may represent a difference of enthusiasm about code review. Some team members may be more keen to pick up reviews, while others consistently prioritize their own coding. This issue can be solved with automated code review assignments or a re-emphasis on code review across the team.

Poor distribution of code reviews can also represent a lack of confidence. The proper response will vary from team to team, but you’ll want to make sure there’s clarity around the expectations of a code review and training on how new team members can ramp up and start participating more deeply in the team-wide collaborative processes.

Review Speed

Unlike Time to First Review, which is used to understand team collaboration patterns, Review Speed is an individual metric, indicating how long each person takes to leave a review.

Review Speed helps you understand how well your team is prioritizing code review.

Velocity can show you how any metric is trending as your team scales or changes. 

A high or inconsistent Review Speed can point to cultural or technical blockers.

A cultural blocker can only be uncovered through conversations with specific team members and managers. If you find that a team or a team member is not prioritizing code review, work with them to reprioritize, along with their other responsibilities. Encourage team members to optimize for this metric until Cycle Time decreases and Pull Request authors are quickly unblocked.

A technical barrier at this stage usually indicates that there may be a lack of tooling. Team members, while working in their individual environment, don’t have the signals or alerting systems that inform them of Pull Requests ready for review.

Start Code Review on the Right Foot

Code Review is one of the most difficult processes to get right on a software development team. A different balance between thoroughness and speed exists on most teams– yet few have a solid idea of what their code review objectives should be. Adding an endless queue of unreviewed Pull Requests to this equation makes the entire process unduly more difficult.

If Time to Review is a problem, a small reprioritization or retooling may be necessary– but the effects will be well worth it.

To learn where your team should focus next, look out for the upcoming articles in our Tactical Guide to a Shorter Cycle Time five-part series:

• Part I: The Virtuous Circle of Software Delivery
• Part II: The Single Greatest Lever in Shortening Cycle Time
• Part III: How to Stop Code Review from Bottlenecking Shipping
• Part IV: Keep Code Review from Being a Waste of Everyone’s Time
• Part V: The Last Mile to True Continuous Delivery