The Unexpected Outcomes of Code Review

As modern software developers we have many tools at our disposal to help us feel confident that the code we produce is high-quality and defect free. Among the most valuable of these are tools that aid in inspecting source code written by another developer, or what we commonly call code review.

Often developers think the main reason for code review is to find bugs, but a 2013 study produced by by Alberto Bacchelli and Christian Bird of Microsoft Research concluded that other outcomes are more prevalent – and possibly more valuable!

Bacchelli and Bird surveyed 165 managers and 873 programmers at Microsoft, interviewed 17 developers with various degrees of experience and seniority across 16 separate product teams, and manually reviewed the content of 570 comments from Microsoft CodeFlow, an internal interactive tool that creates a central location where the discussion of code and the subsequent fixes can be discussed either in real-time or asynchronously.

What they found was that while the top motivation of developers, managers, and testers in performing code review is to find bugs, the outcome of most reviews is quite different.

The ‘Motivations’ chart: the ranked motivation categories from the developer segment of the interviews.

The ‘Outcomes’: Ranked categories extracted from a sampling of code review data.

As can be seen from these charts of Motivations and Outcomes, the top motivation for the largest number of developers was “Finding defects,” yet the topic most commonly discussed in code reviews ostensibly pertains to “code improvements:” comments or changes about code in terms of readability, commenting, consistency, dead code removal, etc.

So if code review isn’t giving us what we want, why do we keep doing it? The response of one senior developer in the study sums it up well:

“[code review] also has several beneficial influences: (1) makes people less protective about their code, (2) gives another person insight into the code, so there is (3) better sharing of information across the team, (4) helps support coding conventions on the team, and…(5) helps improve the overall process and quality of code.”

Bacchelli and Bird conclude that, while there’s a significant difference between the expectations and outcomes of code review, this isn’t a bad thing. While we don’t always get the exact value from code reviews that we expected, we often end up getting quite a bit more.

Turns out the Stones were right: You can’t always get what you want, but you just might find you get what you need.

In addition to a modicum of somewhat superficial bug fixes, teams get benefits such as “knowledge transfer, increased team awareness, and improved solutions to problems” from participating in modern code reviews. Additionally, the idea that code review is good at “educating new developers about code writing” is a compelling point.

Bacchelli and Bird recommend embracing these unexpected outcomes, rather than trying to re-focus code reviews on finding bugs, and let code review policies be guided with the explicit intent to improve code style, find alternative solutions, increase learning, and share ownership.

They also recommend automating enforcement of team style and code conventions to free reviewers to look for deeper more subtle defects – something we agree with wholeheartedly!

All quotes and figures are from Alberto Bacchelli and Christian Bird, “Expectations, Outcomes, and Challenges of Modern Code Review”, May 2013, Proceedings of the International Conference on Software Engineering.