The Moneyball Approach to Engineering Metrics

Engineering managers often rely on subjective clues to assess how their team is doing. But decisions made based on gut feel and imperfect measurements are less than ideal — sometimes they result in team success; other times they result in disappointment. Subjective measures need to be supplemented with objective data; with a combination of the two, managers are more likely to make choices that benefit their teams.

When baseball managers learned this lesson, it got the Hollywood treatment. Moneyball depicted the metrics-first approach to baseball management popularized by Billy Beane, former General Manager of the Oakland A’s. Employing three lessons from Beane’s management approach can help engineering leaders drive productivity and efficiency on their teams.

Lesson #1: Use objective metrics

Beane gained notoriety for prioritizing statistics over scouts’ instincts and consistency over flash. This approach is rooted in sabermetrics — the field dedicated to “the search for objective knowledge about baseball” and statistical analysis of the sport.

Though sabermetrics was founded by baseball fans, Beane applied this predilection for objectivity to managing his team. Rather than relying on scouts’ instincts, intangibles like a player’s footwork, or prestigious metrics like batting average, Beane favored a selection of under-appreciated metrics more closely correlated with consistent results.

Historically, software development has been a field lacking in objective measurements. Classic workload estimates like T-shirt sizes can be useful for internal scoping but are too subjective to translate throughout an org; one team’s size “XL” project is another team’s “Medium.”

Engineering leaders need objective measurements. Our version of sabermetrics involves analyzing the data generated by our own VCS, which is full of objective data about the number of Commits logged in a given period, or the length of time a Pull Request stays open before it’s picked up for review. Objective measurements can help engineering leaders measure progress across teams, projects, and quarters, and are instrumental in setting and achieving effective departmental goals.

Lesson #2: Focus on leading indicators

Of course, not all objective data is equally useful; one of the hallmarks of Beane’s strategy involved the careful selection of metrics. When recruiting, most managers looked at a player’s batting average — the number of times they hit a ball into fair territory and successfully reached first base, divided by their number of at-bats. Beane looked at their on-base percentage, or OBP, a measure of how often a batter reaches first base, even if they get there without actually hitting the ball.

A player must be a good hitter to have a good batting average, but what puts them in a position to score a run is getting to first base. From a data-first perspective, the value of an at-bat is not in the hit itself but in the player’s two feet making it safely to first.

Similarly, to the rest of the organization, your team may be defined by their most eye-catching stats and praised when they deploy a much-anticipated revenue-generating feature. But just as a player’s batting average doesn’t account for every time they get on base, a list of features completed can only reveal so much about your team’s productivity.

The true measure of your team’s success will be their ability to deliver value in a predictable, reliable manner. To track that, you’ll need to look past success metrics and dig into health metrics, measurements that can help you assess your team’s progress earlier in the process.

For example, an engineering manager might look at “Time to Open,” which is a measure of the period between when a developer first logs a Commit in a Pull Request and when that Pull Request is finally opened. Code can’t move through your development pipeline until a Pull Request is opened, and smaller Pull Requests will move through your pipeline more quickly.

Time to Open is similar to OBP. It’s not a prestigious metric, but it’s a critical one. Just as you can’t score if you don’t get on base, your team can’t deploy code that never makes it to a Pull Request. As a manager, you need to make sure that developers are opening small, frequent Pull Requests. If they’re not, it may be worth reinforcing good code hygiene practices across your team and speaking directly with developers to determine whether there’s something particular that’s tripping them up in the codebase.

Lesson #3: Use Metrics in Context

Of course, Beane’s approach wasn’t perfect. Metrics alone can’t tell you everything you need to know about a potential player, nor do they hold all of the answers when it comes to engineering. Data is useful, but leaning too strongly on one metric is shortsighted.

Though Beane was famous for seeing the value in a player’s OBP, there’s also evidence that at times, he relied too heavily on that one metric. Not every college player with a strong OBP was cut out to play professional baseball. While a scout might be able to differentiate between a promising young athlete and one who was unlikely to succeed, a metric can’t make that distinction.

As an engineering leader, you need to rely on a combination of instinct and data. Start with how you feel — is your engineering team moving slow? — and then use metrics to confirm or challenge your assumptions. When something is surprising, consider it a signal that you need to investigate further. You might need to re-evaluate a broken process or find new ways to communicate within your team.

Beane’s approach to managing his team was headline-grabbing because it was revolutionary, representing a major shift in the way fans, commentators, and managers thought about the sport of baseball. Many of his groundbreaking strategies are now commonplace, having been adopted across the league.

Data-informed engineering leadership is still at that revolutionary stage. It’s not standard practice to use objective engineering metrics to assess your team’s progress or guide its strategy, but it will be.