Engineering Leadership Summit: Ale Paredes, VP of Engineering at Code Climate

As part of this year’s Engineering Leadership Summit: Virtual Edition, we spoke to Ale Paredes, VP of Engineering at Code Climate. She discussed her strategies for managing with metrics, and the importance of gathering both quantitative and qualitative data. Below is an excerpt from the fireside chat portion of Ale’s session, edited for length and clarity.

Hillary Nussbaum, Content Marketing Manager, Code Climate: Ale, can you introduce yourself and tell us a little bit about what you do and how you got to where you are now?

Alexandra Paredes, VP of Engineering, Code Climate: Sure. Thank you, everyone, for joining us today. I’m really excited to be here and discuss how I incorporate data into management. I am the VP of Engineering here at Code Climate. I joined Code Climate three years ago, and it has been really rewarding to build products for engineers and engineering leaders. I grew up in Venezuela where I studied computer science, and for most of my career I worked for startups remotely in the US and in Europe. Four years ago I moved to New York, and I joined Code Climate shortly after that.

My passion is distributed systems and reliability, and then leadership and engineering management, and growing the engineering team at Code Climate.

You’ve spoken a lot about the way you use data in your management. What’s interesting to me is the fact that you always emphasize that it’s not just about the data. That there’s a human side, too, and it’s about the qualitative and the quantitative information. Can you tell us a bit about your management philosophy and the role that data plays in that?

I start my management relationships with the foundation of trust, respect, and accountability. As a manager, my goal is to make sure that everyone on the team has the best environment to do their best job, so our team is able to deliver on business goals, and business vision. Quantitative data is a tool that I use to support my goals, but it’s not the goal in itself. I usually talk about feelings as much as I talk about metrics. When I am setting goals or I am finding opportunities for improvements, I read the data and I take into account the information that’s given to me, but then I use that as a starting point to gather more context — that could be through one-on-ones with my direct reports, or other meetings.

Where do you start when you’re working with data? How do you know what you’re looking at, if it’s good, if it’s bad? What’s your starting point?

Typically, either I or someone else has questions. Maybe I think we are not moving as quickly as we could, or we are not delivering at the rate that we should be delivering, or we are not achieving what we should be achieving. Usually I start with that first question and then get more information.

A year-and-a-half ago, I was in a position where I needed to understand why our team’s Cycle Time was over time. I started by creating a mental model of all of the steps our team was taking to deploy code into production.

Just creating the mental model is already a really good exercise to make sure that everyone is on the same page, and if there is any confusion, that’s a good moment to talk about it and make sure you make the changes or address the miscommunication. Then once you have that model of how your process flows, how code, for example, gets into production, you can start adding in data.

So the more general metric would be Cycle Time, but then there are multiple steps to how code gets into production, and you can break down that metric into more granular information to identify areas for improvement. For example, if the team is taking a lot of time to open pull requests, you would zoom into this area of the process and try to identify what may be impacting the team.

How do you figure out where you should zoom in, and what’s worth further attention?

I try to start with general metrics, so Cycle Time, for example. For us, the way we get code into production is, an engineer writes code, opens up a pull request, and then that code requires review. Either there is a request for changes, or it’s approved. If it’s approved, then the pull request is merged and deployed. So I kind of break down the process into its sub-processes.

Once I notice areas that have changed over time — like let’s say we used to open PRs very quickly six months ago, but right now it seems to be taking us much, much longer — that tells me maybe there is something that I need to zoom in on, even if there are opportunities to improve other areas of our processes. It seems like something that we used to be doing really well, but it has changed in a way that’s not desirable.

Once you are zooming in, bringing in context by slicing the data can be really helpful. That can help you, for example, understand, if there is a team within your organization that is doing really well, what are the behaviors, what are the practices that they are applying, so you can try to translate that into the entire organization.

Got it. You mentioned before that you work with this combination of quantitative data and feelings, qualitative information. So where does the qualitative part come in, and what do you do once you have that quantitative information?

So once I notice something that requires more of my attention, I will use one-on-ones to try to get more context, so I can understand either what someone is doing that could impact the way they are working or the way they are communicating, and then in our case we also use retros to talk about the way we work. Some teams use retros to talk about the issues they’re working on, but we talk about how we’re working with product and with design, and how we are communicating and managing our processes.

And once you’ve identified an opportunity for improvement, where do you go from there?

Usually we identify more than one area of improvement, but my philosophy is that it’s best to focus on improving one thing first and then move onto the next thing, rather than trying to improve too many things at once. I try to choose the most impactful area for improvement.

So, let’s go back to the idea of wanting to improve our ability to deploy code into production more quickly. If I know we take too long to open pull requests and also that our code review processes need help, opening pull requests happens first. Improving that bottleneck first is more impactful, even if we know we have other areas to improve. So I try to prioritize what would be the most impactful area of improvement, and then I discuss potential changes with senior managers on our team, as well as how we can set targets and measure progress.

You mentioned that when you share the mental model it’s a great opportunity to make sure everyone’s on the same page, and you’re talking now about discussing things with senior members of your team. How do you decide what portion of the process to share with each level? How looped in are the engineers into the fact that you’re looking at metrics, and which metrics you’re looking at?

I’m in a unique position where my team is building a product based on engineering metrics, so I am usually very transparent about the information I’m looking at. They are part of our company KPIs, so not only our team, but other teams have visibility into it. The way I usually share that information is I send a weekly update Monday morning with an update on what happened last week. It includes some quantitative data but it also includes some qualitative data, like what events could have impacted certain areas of our workflow. I send out that information, and if anyone has any questions I share more context.

I use our metrics on a team-level because I think the most important thing, at least for the size of our team right now, is to make sure that we are all improving together. My focus is on team-level metrics, and then in some cases if I need to, I may set individual target goals, but that’s not a practice that I do all the time.

You’ve made it clear that metrics are only relevant in context, and that you need to have conversations — there’s more that’s important than just the number. But how do you make sure that your team feels that way as well? That when you set a target people aren’t just singularly focused on that target? How do you create the right culture around metrics on your team?

The targets usually focus on improving processes and how we are working together, but we still have goals that are not related to a metric in itself, which I think is very important. We want to improve how we are working, and that means over time our metrics will improve in the direction that we’d like. But at the end of the day our focus as a team is to make sure we are delivering a product that is valuable for our customers, so a lot of our goals are based on what are the objectives we’re trying to achieve this month, this quarter, and that translates into business goals and customer goals.

How do you match those business goals with team level goals and team level metrics? What’s that process like?  

Business goals are more related to product deliverables, and then in terms of how I’m using metrics to measure progress, we use KPIs. We have sets of success KPIs, health KPIs. So an example of a health KPI for us is making sure that the system is healthy. So we track the amount of incidents that we’re having, and then if something seems to change on that front, we take action right away. We balance creating goals that are related to a product, while making sure we’re also paying attention to our KPIs help us calibrate how we use our time and which area we’re focusing on.

We’re in a period where there’s a lot of change, there’s a lot going on. How might metrics and the data that you’re looking at have helped you lead your team through those transitions?

To give you a little bit of context on what the team used to look like before the pandemic happened, we had an office in New York and most of our team was in New York, but we also have engineers in Brazil. So when we moved to remote, we thought we were somewhat prepared because we already had people who were working remotely. But we saw that was not the case, and having metrics was really helpful to understand trends. For example, our Rework, which is a metric that lets you see how often you are changing code that was recently changed, had increased. Our Cycle Time was increasing over time.  

So there were little things that told me, even when we were somewhat prepared to work remote, this was still a major shift. That helped me have conversations to understand how we were communicating about projects in general, how well we were helping less-experienced engineers to break down their work. How we were communicating not only in private channels, but also in public channels. So for example, right now we try to have most of our conversations in public channels with the rest of the team, so we can all be on the same page as much as possible.

And so you were able to look to certain metrics and sort of find areas where things weren’t maybe working as well remotely and then work through those. That’s very cool. How involved was your team in helping work through those pain points and figure out solutions?

Very involved, especially since the team grew right at the moment we transitioned into working remotely. I am not involved directly anymore with the engineers. I have two tech leads who are leading their respective teams, so they were very involved in relaying feedback on how the team was doing with working remotely, and sharing some of the communication challenges they were having, and then we worked together to find ways to improve.