The Fundamentals of Data-Driven Engineering Leadership [Webinar]

Metrics are an essential tool for engineering leaders looking to gain objective insights about the teams they manage. In order to improve business outcomes and overall team health, leaders must identify which metrics to track and apply the data with intention. 

Code Climate’s Senior Product Manager, Mike Koeneke, sat down with three directors of engineering: Sophie Roberts of Shopify; Mojtaba Hosseini of Zapier; and Romain Dupas of Code Climate, to discuss how they each define and utilize metrics to meet their goals. 

Read on for highlights from their conversation. Answers have been shortened for length and clarity.

Defining Metrics: Which metrics do engineering leaders look at, and why?

Sophie: There are four kinds of metrics I look at on a regular basis:

• Operational or system health metrics: These include SLAs, API failure rates, system throughput — the things that tell you, as an engineering leader, if your product is healthy.
• Development metrics: Your average time to commit, your average build time, and your build success failure, which fall under engineering productivity, helping us understand how to improve the productivity of our engineering teams.
• Organizational health metrics: Team composition, retention, employee morale, and ability to actually hit goals that you set. It helps us understand if our people are in a good place and are being effective.
• Business metrics: These metrics show that the business is moving forward. Are we having the impact on the business that we expect to have with regards to things like revenue, customer acquisition, efficiency? 

Mojtaba: One thing we’ve started to do at Zapier is to talk about hierarchy of metrics that go beyond engineering, and trying to tie in the engineering metrics into a larger set. As a SaaS company, we’ve got our financial metrics which are very top of mind for the board and the C levels…[so for example,] your customer metrics: your churn and activation…if you’re developing quickly, if you’re getting that feedback, if your teams are high performing and happy and engaged. If so, you probably are creating good products, which bring in users, which then brings in the revenue.

Romain: I look at metrics corresponding to all aspects of our process and our goals: Team goals such as OKRs, financial goals, computing metrics, API metrics, productivity. At the end of the day, any goal or process needs to be measurable. Defining those goals and metrics will make the process more efficient.

Balancing Intuition With Objective Data

Sophie: The superpower of actually using metrics is to combine the data with judgment and instincts. If someone on your team is in the lower bound of what you’ve chosen as a performance metric, and you substitute the metric for your judgment and say they’re underperforming, you’re making a bad decision. 

When it comes to things like organizational metrics or individual performance metrics, I don’t even look at those as objective pieces of data. I look at those as pieces of data that allow us to explore the margins and find things to be curious about. That’s where your judgment and instinct come in. 

Mojtaba: I have a mental model around using metrics: the dashboard of a car. The car’s purpose is to serve the drivers and the passengers. The dashboard, intrinsically, is of no real value if the humans don’t get where they want to go. 

You need the objective data: What is my speed? What is the oil gauge telling me? The speedometer? But you still need the human to decide where to go. [As the driver], I might know the maintenance history of the car, so even if the manufacturer tells me I can go faster, I’m not going to push the car to the maximum speed — because of experience and intuition. The intuition by itself can be too vague and needs the objective data, a reality check. But the reality check without that vision just becomes a bunch of numbers. 

Romain: Continuing with the car analogy, on the dashboard is a limited amount of space to show the information that matters. You don’t have anything related to what’s in your trunk; the only information you have is what’s valuable: information about the safety of your car. With metrics, you don’t measure everything; understand what you want to measure for your own goal.

Context is Key: Make Sure Your Team Knows Their Purpose

Sophie: It’s critical for me, and I think for all businesses, for every single engineer to understand how the work contributes to the business’ success. When people understand that, then they understand what [things like] an outage actually mean for our merchants. 

My engineering team will sit in on interviews with our customers on a weekly basis and they’ll just chat and understand the pain points, because it’s really important that people can tie the experience of your customers to the metrics of the work that they’re doing. 

Mojtaba: The job of the leader is to understand how pieces work together and track the outcome they’re working towards. If a platform team is going very slowly, that’s okay, because they are the platform team. They build, they make it so that other teams can go faster. At a high level, we’re still going fast, but I’m not going to tell the platform team they should be going as fast as these other non-platform teams. They fulfill different functions, and it’s the aggregate of those that really brings value. That interpretation, that context, is what leaders need to bring together. 

Be Transparent: Share Metrics With Your Team

Sophie: Don’t hide your business metrics from the engineering team. If I could get anyone to take an action item from this, it would be that if your engineering team isn’t getting a copy of your Monthly Business Review (MBR), make that happen.

Romain: My team and I actually set the OKRs together to be aligned with the business OKRs. I ask the team, What do you think we can do to move the organization in accordance with the business OKRs? And that works very well because it goes back to the idea of making sure that the team understands the metrics and is motivated to reach that goal, instead of imposing from the top down that you have to fit these metrics into your day-to-day work.

Using Metrics for Compensation: What to Avoid and What to Reward

Sophie: You want people to take risks, you want people to fail, and you want the metrics to reflect the fact that things failed, but you don’t want to punish people. So there are a whole bunch of cultural interdependencies there that you have to manage.

The metrics can show us if the work is successful or not, but [for me], the metrics themselves don’t tie into people’s compensation. What ties into people’s compensation is how they approach solving the problem. I think that’s the only way you can do it if you’re going to build a culture where experimentation, risk, and failure are genuinely valid, as opposed to just being talking points on a wall.

Mojtaba: We don’t use the metrics in any way, even in the conversations we have about performance. This might sound controversial, but we don’t even have performance evaluation of individual contributors tied to compensation. We have competency levels tied to compensation.

Romain: Metrics are a foundational bed to talk with my engineers about how we can make our process better. And the compensation aspect would be more about where we are today, and where we want to be. Who are the contributors that are going to lead the team, and what will be the involvement and quality of their contribution? [I base compensation on] what their work brings to the table to help the company, more than on the metrics themselves. 

To find out what else Sohpie, Mojtaba, and Romain had to say, listen to the full discussion here. 

To find out how you can use metrics to help your team meet their goals, request a free demo of Code Climate Velocity.