Webinars

As part of this year’s Engineering Leadership Summit: Virtual Edition, we spoke to Juan Pablo Buriticá, Head of Engineering, LatAm at Stripe. He discussed strategies for implementing delivery metrics, and shared how he got buy-in from his team. Below is an excerpt from the fireside chat portion of Juan Pablo’s session, edited for length and clarity.

Hillary Nussbaum, Content Marketing Manager, Code Climate: Welcome to the latest session in our Engineering Leadership Summit I’m here with Juan Pablo Buriticá, Head of Engineering, LatAm at Stripe. I’ll let you take it from here, Juan Pablo — introduce yourself, and tell us a little bit about what you do and how you got to where you are.

Juan Pablo Buriticá, Head of Engineering, LatAm at Stripe: Of course. Hello Hillary, and hello everyone. Welcome to my kitchen. I figured since we were going to talk about shipping and delivery and metrics, why not try to ship a product? I’m Juan Pablo Buritica, I live in New York City. I’m from Bogotá, Columbia. I’ve been in the U.S for a little bit over a decade, and in New York for a decade. I’m a former chef, turned engineering leader, and I’ve been leading product engineering organizations for a decade or so as well.

More recently I was at Splice, a music company, for three and a half years. I was the VP of Engineering, and we built a distributed team of about 60 engineers within the Americas. I started working with delivery metrics there, and tried to understand how to make the team more efficient without working longer or harder. That led to a few talks, and also finding Code Climate and actually using it in production.

Then earlier this year, I took a sabbatical. I was ideally going to start a consultancy on delivery engineering metrics, but Stripe came along with an offer of working to expand the product into Latin America. It’s a mission that I am extremely motivated by, and that I’ve really been thinking about a lot. I joined eight weeks ago, so I’m just getting started and trying to figure out how all my experience fits into the picture.

So, welcome to my kitchen and thank you for having me.

Thank you for being here, and for having us in your kitchen. It’s a beautiful kitchen. You touched on this a little bit, but you just started a new role, and you’ve spoken a lot about how you used metrics in your last role. How are you thinking about metrics and similar strategies as you settle into your new job?

All right. Before I answer this question, I’m going to tell you, we’re going to attempt to make a mushroom risotto. There are some things that have been prepped. So you can think of this as work, engineering work that has already been scoped and outlined and is ready to be worked on. And then there’s some mushrooms and some onions that I need to chop and prep — this is work that is unscoped.

Now, let’s go to the question. I think of delivery metrics as visibility. At Splice, it was very useful for me to see the bottlenecks that the organization had in our delivery pipeline, and how we could drive efforts to solve those bottlenecks.

Now I’m with Stripe, a much larger organization that has a ton of metrics all over the place. And I am leading a team that is more on the surface, right? We expand the existing product into the region, at least initially. I’m trying to figure out how to better measure or get insight into what the individuals of LatAM engineering are like, what their role is in the delivery pipeline, and how we can measure the bottlenecks they may have. So I’m trying to figure out how to see the delivery cycle, because I have no visibility yet.

And in the past, is that how you’ve typically approached metrics, as a tool for visibility?

They’ve been a tool for visibility since we integrated the first version of our delivery framework at Splice. I remember my boss asked me, “What are the things that we’re even trying to solve?” And I couldn’t say, “I think we’re slow, it feels slow.” I needed to know why we were slow. So I read a bunch, and I found the book, Accelerate, by Gene Kim, Jez Humble, and Nicole Forsgren. I learned a ton about the capabilities that organizations should have, and how you can use metrics like Cycle Time or Deployment Frequency to evaluate your process.

Having these metrics gave me three things. It gave me the language to talk to the team about our delivery pipeline and things that we should consider. It gave me insight into what to measure and how to measure it, and where it matters. A ton of the research that is out there highlights why it’s important to have a high-performing engineering organization. With metrics, you can actually tie organizational performance to engineering and delivery. And that was really, really useful. Because I could talk to the board about how we think of staffing and whether a team was going fast or slow.

And since you’re heading in this direction a little bit, what strategies do you use for talking about metrics with people who aren’t necessarily familiar with metrics, or who aren’t the ones using them?

Metrics have helped me now illustrate portions of the engineering delivery process that aren’t intuitive. For stakeholders or people outside of engineering, Cycle Time as a concept and as a metric has been extremely useful in talking about staffing and throughput.

We are used to talking about velocity in points, and we’ve taken this metric as an actual metric of performance or delivery. But it is a representation of capacity, and it’s very relative. If we have two teams of four engineers, their velocity is going to be relative to the scale they’re using, as well as their skills, and even the work they’re taking on. So it isn’t a good metric for us to talk about the speed we’re actually delivering at. It’s just a measure of capacity, of how much we can take. It can tell us if we estimated the right amount of work and if we can consistently deliver on our estimates, but it doesn’t tell us how fast we are. That has become very useful in executive conversations with product or finance. We can set a realistic timeline when we have a better understanding of how fast we can go.

That’s how we talk about Cycle Time, as the speedometer for the team. And then how do you talk about metrics with your team, whether your team is new to metrics or familiar with them?

That is the most fun conversation to have, and I’ve found that this is one of the pieces engineering leaders are struggling with the most when they try to implement metrics. It’s been interesting for me to see that we talk a lot about being data-driven in our industry, but when we try to have a conversation about metrics and data, there’s a lot of concern, especially for ICs. And rightfully so, because there’s always the concern of, “How are these metrics going to be used against me?” And there is a general mistrust towards management. The hardest challenge for me in implementing metrics at Splice was getting the buy-in of the team.

I approached it in a couple of ways. First, I did have a ton of trust from them, and I had a lot of leeway to cash in on the trust currency that I had. But I still had to make an argument about how we were going to use metrics, and why. I wrote an entire RFC, and that’s how I convinced my team that treating this as a pilot was useful. The second thing I did was, I didn’t give access to those metrics to anyone other than myself and the Director of Production and Engineering. We could eventually give access to other people, but taking access away would be much harder.

There’s always the chance that metrics can be used against people, and since the purpose of these metrics is to measure the delivery pipeline and not individuals, I wanted to make sure that they knew we were measuring a process and not their individual performance. Some of the tools we were using had information about individuals, so I protected the access to that, and I think they trusted me with that.

Over time, I would share reports and I would share different bits of information that would give them more confidence in how we were using these metrics. It is on us as leaders to earn trust and demonstrate that we’re going to use these metrics for what they’re meant to be used for.

How did you choose where to start, and which metrics to measure?

We started by having an onsite with senior engineering leadership, where we tried to identify what we believed were the biggest problems at Splice at the time. Then we looked at what metrics we thought could drive some of those things.

For example, feeling slow was one of these problems. So we started with Time to Merge, or how long does a Pull Request take to merge? We didn’t know what was good or bad, but we said, let’s say all PRs are going to be merged in 36 hours, and that will help us evaluate whether this happens. And if not, what obstacles do engineers have in getting their work merged?

And the second challenge we had was deployment. Our deployment wasn’t automated, so a merge didn’t necessarily mean that the code was accessible for other people to use. And so we created a deployment metric as well, and then started learning through that. It was a very iterative process.

And then how did you use those metrics to inform your strategy, and how did they impact what you might’ve been doing as a manager with your ICs?

Our strategy was to set some metrics that we believe may help us improve our delivery process, measure them, and get a baseline. From there we could try to understand whether they were really out of a normal range, and whether we really needed to improve. And then we would create projects that would help us drive down those metrics. And there was a production engineering team that had the purpose of doing this.

The second part of being very specific about the strategy was that we had to crawl, walk, and then run into delivery. Primarily because we didn’t want to start running and go fast in the opposite direction, delivering a ton of terrible quality product.

So we had an angle on delivery, Cycle Time, and we had a Deployment Frequency metric as well. And then we had one that helped us monitor the quality of what we were delivering. It took us six months to properly deploy this entire plan. The first few months, while we figured out how to measure things, we deployed our own dashboards and we brought on a tool [Code Climate's solution]. This is not a sponsored talk. Once we had some of that visibility, it was easier for the team to know what they were driving towards. And they took ownership. The production engineering team at Splice took ownership of their own metrics and how to drive them. And they, in the end, made it successful.

As part of this year’s Engineering Leadership Summit: Virtual Edition, we spoke to Dr. Aneika Simmons, PhD, and Anjuan Simmons, Engineering Coach at Help Scout. They discussed strategies for preventing and addressing burnout on engineering teams. Below is an excerpt from the fireside chat portion of their session, edited for length and clarity.

Hillary Nussbaum, Content Marketing Manager, Code Climate: Welcome. We’re with Anjuan and Aneika Simmons. Anjuan is a Technology Translator, and Engineering Coach at Help Scout. Aneika is a Full Professor of Management at Sam Houston State University. Please, introduce yourselves, and tell us a little bit about what you do and how you got to where you are now.

Dr. Aneika Simmons, PhD: I’m a Professor of Management at Sam Houston State University. I am a Texan, so I was born and raised in Texas. And then I went ahead and became a Longhorn at the University of Texas at Austin. From there, I was recruited to Accenture, where I worked as an Information Technology Consultant. And then from there I was recruited at Cap Gemini Ernst & Young. I did work at Enron for two years, but not on their finances, only on SAP, let me be clear.

I worked for about eight years, and I really wanted to do something that better utilized my gifts and my talents and my personality. So I decided to apply for a PhD program, and eventually I was accepted into the Mays program at Texas A&M University. After that, I went to Sam Houston and I’ve loved it. It’s been great for me, for my career, for my family. And it’s something I really enjoy.

Anjuan Simmons, Engineering Coach at Help Scout: Hello, everyone. Again, I’m Anjuan, and I am an Engineering Coach at Help Scout, where we support people who love their customers. I started my career at Accenture as well, working on large enterprise software projects and making sure that they got implemented. I also worked at Deloitte, and then I did a few stints at startups.

One thing I really love about Help Scout is the opportunity to take a lot of the rigor and discipline that I learned at those huge companies, but also to add the personal touch and care that I learned from startups. And so that’s really how I go about my job at Help Scout.

Great, thanks. Now let’s jump right in. We’re talking about burnout today. What do you mean when you say burnout, and why does burnout matter to us as individuals and as leaders?

Aneika: Over a year ago, Anjuan and I — we’ve been married for 18 years, and we have three, basically teenagers (I say basically, because one is 12) — discussed the fact that we were dealing with burnout. So we started doing research on it. We determined that burnout, as defined by World Psychiatry, is a psychological syndrome emerging as a prolonged response to chronic interpersonal stressors on the job. Stressors are the things that cause stress. Stress is what I feel.

There are three dimensions to this. One of them is an overwhelming sense of exhaustion. The second is a feeling of cynicism and detachment, and the third is a sense of ineffectiveness and a lack of accomplishment.

And why should we care about it? At the individual level, it can impact you emotionally, mentally, and physically with all kinds of horrible things like high blood pressure, not being able to sleep at night, or a racing mind. And then from a leader’s perspective, you should care because, number one, I hope we all care about the people that work for us, but number two, it impacts the corporate costs. People are not able to show up or give their best selves. So, this is something with a big impact, and it’s really important.

And how is burnout different for an individual and for a team? How does it impact team dynamics when it’s just one person versus everyone?

Aneika: As an individual, of course it impacts you. A lot of people think that if I’m an individual and I’m engaged, then I’m not going to be burned out. But the research actually says that you can be both engaged and burned out. Meaning that you can be fully involved and burned out at the same time. It’s not just the people who are out the door.

And so how is it different at a team level? Well, for teams there’s emotional contagion, where your emotions can trigger the same emotions in other people. Stress and burnout are not the same thing, but stress is a precursor to burnout. And that can become part of a team dynamic via emotional contagion, and that’s what we need to look out for.

And then one of the big aspects of burnout is depersonalization. If you’re working in a team and you’re depersonalizing your team members, obviously that’s not a good thing. People start depersonalizing because they have nothing left to giv, and they’re just trying to focus in on the minimal aspects of what’s happening. But that’s not healthy, and that is an aspect of burnout.

What strategies can a manager take to try to prevent burnout before it happens?

Aneika: One thing that we need to do is, we need to build our well before we need it. A manager should know their people. They should know the people who work for them, and have an understanding of their baseline. If someone was always a really patient person, and then their patience with people starts waning, that could be a signal.

And then how do you prevent it? Focus on human relationships. Let’s look at one another in the eye, let’s have these meetings, let’s get to know one another. Let’s talk through what we feel. We do a thing where we talk about the importance of eye contact, just engaging in people, understanding the people that you work with. And then we have a cliche that, if you want to go fast, go alone. But if you want to go far, well, let’s go together. If we’re doing things together, we think that that can help mitigate burnout.

You just mentioned it’s easiest to spot signs of burnout if you’re familiar with how a person works. So, what strategies can one use to get that baseline assessment?

Anjuan: One of my main tools as an Engineering Manager responsible for the health and wellbeing of software developers is the one-on-one. A one-on-one is a regular meeting, hopefully weekly, but it can be biweekly, where you meet with someone and talk about what’s going on with them. I really try to make my one-on-ones different from a status update. I have many other ways to get status updates.

They’re a way for me to meet on a regular basis with the people who report to me and really get to know them outside of the work, but to also get that sense of what is their profile, how are they normally presenting themselves? One-on-ones are a crucial way for me to not only be connected to my team, but to get their profile. One other thing that I do is, in meetings, I do a lot of reading facial expressions and body language. And so over time, I just get a mental model for how this person performs. Sometimes they use lots of hand gestures, or present themselves a certain way in meetings. And so that’s a key thing as well, to know how to read a person’s body language, to know that usually they’re really open, but recently they’re closed. And then you can begin to see that they’re outside of their normal profile, and you can begin to interrogate that data.

At Help Scout, we also do self-assessments every six months, and that’s really a way for everyone to look back and reflect. We ask questions like, what has been motivating you over the past six months, or what’s changed, or what’s been draining? And when you get that information, you’ll begin to read that and you begin to see, “Okay, this person, when we had a lot of change happening at the beginning of January, it really impacted them.” And so having that information is really a great way to get the data to understand where individuals are, and then move forward from there.

You mentioned looking at body language and reading those subtle signals. Given that a lot of teams are remote now, how do those strategies differ? Do you find that you need to establish a new baseline before making an assessment?

Anjuan: Help Scout was built from the beginning to be fully remote. Going on more than eight years, the company was built to be global and to work remotely. If your company was built from the ground up to be remote, or you’ve been that way for a long time, then it’s really about continuing those practices. Regular one-on-ones, self-assessments, things like that.

In a remote culture that’s new to a company, if you are maybe thrown into working remotely due to current conditions like the global pandemic, then you have to understand that one, this is not normal. I’m a big fan of working remotely, but I know that a lot of people are not getting a really good first taste of it. Often you’re working remotely with a spouse who’s also been thrown into working remotely. You may have to share a space with that person. There may be children you now need to help with their work. And so I think that being aware, especially now, that this is not normal, and not expecting the team to be normal, is okay.You have to give the team time to figure out their new normal.

One tool that we use at Help Scout that I really like is called Fika. And Fika, loosely translated, is from a Swedish word that means to take a break, get something sweet, like a pastry, and then get a nice beverage, usually coffee or tea. And then we just, we typically use Zoom, hop on a video conference, and we just chat — not about work, but about each other and what’s going on. And so that’s a way that we are able to, even though we’re remote, to connect with each other. There’s also a tool called Donut, which is a Slack app that will randomly pair people for Fikas. And that way, not only with the Engineering Department that I sit in, but with Marketing, Sales, et cetera, I’m able to build those relationships even though we are a remote company.

We are not able to do this for the foreseeable future, but having a retreat is also crucial for a remote company. At Help Scout, we typically have them twice a year.

Are there any other special considerations when it comes to remote teams? We’ve been hearing a lot about Zoom fatigue. Are there other things that maybe you wouldn’t have to think about when you’re all in the same place, but that come up in a remote situation?

Anjuan: Yeah, for sure. Two quick things. I’m very lucky to work at Help Scout where we got a stipend for our home office, because they don’t assume that everyone has the stuff they need at home. If companies can provide that support to help everyone working remotely, especially if they were just thrown into it, that is super helpful. The other thing I would say is that often when you’re working remotely, you don’t have a commute. And most people don’t realize that’s often a really good getting ready time. You may listen to podcasts or to music or do different things, catch up on the news from the radio. And then when you’re leaving work, you have that time to decompress.

But when you’re working remotely, you don’t have that commute. That nice bookend to your work day. So, I would say have some kind of ritual — don’t just spring out of bed and hop on Zoom and start working. Maybe take a walk around the block, or make a nice cup of coffee before you start working, because it’s really easy to work a lot more than you’re used to because you don’t have those natural breaks in your day.

Aneika: I know for myself, I wasn’t really on Zoom that much until everything happened with COVID-19. And so if you could have blocks in your day where you say, “Well, no, I’m not having any Zoom meetings at that point,” that helps. Because Zoom fatigue is real, the stress of making eye contact. When you’re on the phone, you can walk around and do things, but in Zoom meetings, some people may feel a little bit held captive.

That makes a lot of sense. We’ve been talking a little bit about preventing burnout and causes of burnout, but obviously it’s not always preventable. What steps would you recommend that a manager take to address burnout when they become aware of it?

Anjuan: As a manager, it’s really important to understand a person’s baseline profile. And ideally what you’ll be able to do is to say, “Okay, this person reports to me, and they’re beginning to diverge from their baseline profile. They’re acting in a way that is an anomaly from everything that I’ve seen before.” And then the key thing is that don’t diagnose that person. Don’t say, “You have burnout, here’s what we’re going to do.” That is something that a professional should do, and that person may not respond well to you trying to diagnose them.

What you do instead is you share observations. You say, “Hey, I’ve seen you typically log on if you’re remote at nine o’clock in your local time, you’re getting on at around closer to noon.” Or, “You typically don’t introduce a lot of bugs into the system, but I’m seeing more bugs that are introduced by your code.” Or, “You typically are more thoughtful and intentional when you comment on pull requests, and now you’re just writing one sentence.” And so by doing that, you can say, “This is what I’m observing. What do you think is going on?” And often it’s in that conversation that you’ll begin to glean what’s happening. And then as a manager you can do things like, “Okay, let’s look at your workload, and you know what? Out of these five things, only these two are really valuable. Let’s just focus on that.”

And in our framework, we talk about having the need to burn down distractions. And so by doing that, you can focus on what’s really valuable. I find that if I can narrow down what my team has to worry about to just one or two things in a given sprint, that goes a long way toward releasing the stress that builds up when we try to keep a lot in our minds. I really try to make sure that my team only has one or two things that they have to think about, because in a world where there is a global pandemic and there’s a lot of civil unrest and a lot of disruptions to the economy, the last thing I want my people to worry about is work. And so I try to make work as simple as possible.

What would you recommend that people do if they’re feeling burned out? Or if they notice a team member is burned out, but they’re not that person’s manager, so they don’t necessarily have the ability to say, “Let’s focus and take things off your plate?”

Aneika: The first step is acknowledgement. So, for you to acknowledge that you’re burned out. You would ask yourself, are you feeling exhausted? The things that used to really light your fire and make you passionate about work, are they still there? How healthy are your relationships at work? Are you engaged with people? Those types of things are important. One of the main things that we do to mitigate stress is, we go for walks every day. We go for walks in the evening and just kind of decompress and talk. So, the first step is acknowledgement and accepting that there’s nothing wrong with you. We’re human beings. And that we get to a point sometimes where we feel that, “My plate is full, I have nothing else to give, and I have to address this.”

Anjuan: To the latter part of the question, which is, I’m not this person’s manager, but I see that they’re burned out — if you have the right relationship with that person, you can share your observation. Again, don’t try to diagnose them as being burned out, but say, “Hey, I’ve been noticing this.” Like my wife said, “You used to be really energized by working in our CI/CD pipelines, but now you seem to have slowed down there.” Share those observations. And then what you can do is say, “Here are some things that have worked for me when I’ve gone through that.

The ideal is that you’ve already built up what we call burnout resistance in your own life, and you’re able to model that to the other person. If you don’t feel that you have the relationship to be able to share those observations, it’s really important to find someone in the organization who does, and candidly share those observations with that person

Aneika: And I’ll just say real quick that all of these types of things that we’re talking about requires a certain level of psychological safety at your job. So you have to have an organization where you feel that you can be vulnerable. And if you can’t be vulnerable, then maybe you really won’t know what your employees, or the people who work for you, or work with you, are going through. The organization has to have a culture where people feel like they actually can be vulnerable and say how they truly feel.

I’m curious about what you just said, that it is important to be vulnerable as a manager, as a leader. Do you have some tips or strategies for people who aren’t necessarily as comfortable or as practiced with that?

Aneika: Well, there are different types of personalities, like my husband is an introvert, and I’m an extrovert. Of course not all introverts and extroverts are the same, some personality types might be more comfortable sharing things that are close to your heart. So, you would need to keep that in mind. But one of the things that I would say is just make sure you develop a healthy relationship with at least one or two people on your team. I know that as teams get larger, you can’t be close to everyone, but you do want to try to have that interconnectedness. That is really important so that when things do come up, you feel comfortable sharing.

Anjuan: Yeah. And at Help Scout, we use a tool called Know Your Team, which is a website that asks a question every week and then people respond to it. It seems kind of trite, but by just sharing those answers, just sharing more about what you think outside of work…you’ll begin to make the first steps toward having that relationship. And then building that culture where people do feel a sense of a psychological safety. Aneika and I have often said that healthy teams build healthy code, and people can be healthy obviously by taking care of their physical body, by exercising and eating properly, and by having relationships that are strong and meaningful. But I also think if people can bring their authentic self to work, then you’ll get their best work.

As part of this year’s Engineering Leadership Summit: Virtual Edition, we spoke to Ale Paredes, VP of Engineering at Code Climate. She discussed her strategies for managing with metrics, and the importance of gathering both quantitative and qualitative data. Below is an excerpt from the fireside chat portion of Ale’s session, edited for length and clarity.

Hillary Nussbaum, Content Marketing Manager, Code Climate: Ale, can you introduce yourself and tell us a little bit about what you do and how you got to where you are now?

Alexandra Paredes, VP of Engineering, Code Climate: Sure. Thank you, everyone, for joining us today. I’m really excited to be here and discuss how I incorporate data into management. I am the VP of Engineering here at Code Climate. I joined Code Climate three years ago, and it has been really rewarding to build products for engineers and engineering leaders. I grew up in Venezuela where I studied computer science, and for most of my career I worked for startups remotely in the US and in Europe. Four years ago I moved to New York, and I joined Code Climate shortly after that.

My passion is distributed systems and reliability, and then leadership and engineering management, and growing the engineering team at Code Climate.

You’ve spoken a lot about the way you use data in your management. What’s interesting to me is the fact that you always emphasize that it’s not just about the data. That there’s a human side, too, and it’s about the qualitative and the quantitative information. Can you tell us a bit about your management philosophy and the role that data plays in that?

I start my management relationships with the foundation of trust, respect, and accountability. As a manager, my goal is to make sure that everyone on the team has the best environment to do their best job, so our team is able to deliver on business goals, and business vision. Quantitative data is a tool that I use to support my goals, but it’s not the goal in itself. I usually talk about feelings as much as I talk about metrics. When I am setting goals or I am finding opportunities for improvements, I read the data and I take into account the information that’s given to me, but then I use that as a starting point to gather more context — that could be through one-on-ones with my direct reports, or other meetings.

Where do you start when you’re working with data? How do you know what you’re looking at, if it’s good, if it’s bad? What’s your starting point?

Typically, either I or someone else has questions. Maybe I think we are not moving as quickly as we could, or we are not delivering at the rate that we should be delivering, or we are not achieving what we should be achieving. Usually I start with that first question and then get more information.

A year-and-a-half ago, I was in a position where I needed to understand why our team’s Cycle Time was over time. I started by creating a mental model of all of the steps our team was taking to deploy code into production.

Just creating the mental model is already a really good exercise to make sure that everyone is on the same page, and if there is any confusion, that’s a good moment to talk about it and make sure you make the changes or address the miscommunication. Then once you have that model of how your process flows, how code, for example, gets into production, you can start adding in data.

So the more general metric would be Cycle Time, but then there are multiple steps to how code gets into production, and you can break down that metric into more granular information to identify areas for improvement. For example, if the team is taking a lot of time to open pull requests, you would zoom into this area of the process and try to identify what may be impacting the team.

How do you figure out where you should zoom in, and what’s worth further attention?

I try to start with general metrics, so Cycle Time, for example. For us, the way we get code into production is, an engineer writes code, opens up a pull request, and then that code requires review. Either there is a request for changes, or it’s approved. If it’s approved, then the pull request is merged and deployed. So I kind of break down the process into its sub-processes.

Once I notice areas that have changed over time — like let’s say we used to open PRs very quickly six months ago, but right now it seems to be taking us much, much longer — that tells me maybe there is something that I need to zoom in on, even if there are opportunities to improve other areas of our processes. It seems like something that we used to be doing really well, but it has changed in a way that’s not desirable.

Once you are zooming in, bringing in context by slicing the data can be really helpful. That can help you, for example, understand, if there is a team within your organization that is doing really well, what are the behaviors, what are the practices that they are applying, so you can try to translate that into the entire organization.

Got it. You mentioned before that you work with this combination of quantitative data and feelings, qualitative information. So where does the qualitative part come in, and what do you do once you have that quantitative information?

So once I notice something that requires more of my attention, I will use one-on-ones to try to get more context, so I can understand either what someone is doing that could impact the way they are working or the way they are communicating, and then in our case we also use retros to talk about the way we work. Some teams use retros to talk about the issues they’re working on, but we talk about how we’re working with product and with design, and how we are communicating and managing our processes.

And once you’ve identified an opportunity for improvement, where do you go from there?

Usually we identify more than one area of improvement, but my philosophy is that it’s best to focus on improving one thing first and then move onto the next thing, rather than trying to improve too many things at once. I try to choose the most impactful area for improvement.

So, let’s go back to the idea of wanting to improve our ability to deploy code into production more quickly. If I know we take too long to open pull requests and also that our code review processes need help, opening pull requests happens first. Improving that bottleneck first is more impactful, even if we know we have other areas to improve. So I try to prioritize what would be the most impactful area of improvement, and then I discuss potential changes with senior managers on our team, as well as how we can set targets and measure progress.

You mentioned that when you share the mental model it’s a great opportunity to make sure everyone’s on the same page, and you’re talking now about discussing things with senior members of your team. How do you decide what portion of the process to share with each level? How looped in are the engineers into the fact that you’re looking at metrics, and which metrics you’re looking at?

I’m in a unique position where my team is building a product based on engineering metrics, so I am usually very transparent about the information I’m looking at. They are part of our company KPIs, so not only our team, but other teams have visibility into it. The way I usually share that information is I send a weekly update Monday morning with an update on what happened last week. It includes some quantitative data but it also includes some qualitative data, like what events could have impacted certain areas of our workflow. I send out that information, and if anyone has any questions I share more context.

I use our metrics on a team-level because I think the most important thing, at least for the size of our team right now, is to make sure that we are all improving together. My focus is on team-level metrics, and then in some cases if I need to, I may set individual target goals, but that’s not a practice that I do all the time.

You’ve made it clear that metrics are only relevant in context, and that you need to have conversations — there’s more that’s important than just the number. But how do you make sure that your team feels that way as well? That when you set a target people aren’t just singularly focused on that target? How do you create the right culture around metrics on your team?

The targets usually focus on improving processes and how we are working together, but we still have goals that are not related to a metric in itself, which I think is very important. We want to improve how we are working, and that means over time our metrics will improve in the direction that we’d like. But at the end of the day our focus as a team is to make sure we are delivering a product that is valuable for our customers, so a lot of our goals are based on what are the objectives we’re trying to achieve this month, this quarter, and that translates into business goals and customer goals.

How do you match those business goals with team level goals and team level metrics? What’s that process like?  

Business goals are more related to product deliverables, and then in terms of how I’m using metrics to measure progress, we use KPIs. We have sets of success KPIs, health KPIs. So an example of a health KPI for us is making sure that the system is healthy. So we track the amount of incidents that we’re having, and then if something seems to change on that front, we take action right away. We balance creating goals that are related to a product, while making sure we’re also paying attention to our KPIs help us calibrate how we use our time and which area we’re focusing on.

We’re in a period where there’s a lot of change, there’s a lot going on. How might metrics and the data that you’re looking at have helped you lead your team through those transitions?

To give you a little bit of context on what the team used to look like before the pandemic happened, we had an office in New York and most of our team was in New York, but we also have engineers in Brazil. So when we moved to remote, we thought we were somewhat prepared because we already had people who were working remotely. But we saw that was not the case, and having metrics was really helpful to understand trends. For example, our Rework, which is a metric that lets you see how often you are changing code that was recently changed, had increased. Our Cycle Time was increasing over time.  

So there were little things that told me, even when we were somewhat prepared to work remote, this was still a major shift. That helped me have conversations to understand how we were communicating about projects in general, how well we were helping less-experienced engineers to break down their work. How we were communicating not only in private channels, but also in public channels. So for example, right now we try to have most of our conversations in public channels with the rest of the team, so we can all be on the same page as much as possible.

And so you were able to look to certain metrics and sort of find areas where things weren’t maybe working as well remotely and then work through those. That’s very cool. How involved was your team in helping work through those pain points and figure out solutions?

Very involved, especially since the team grew right at the moment we transitioned into working remotely. I am not involved directly anymore with the engineers. I have two tech leads who are leading their respective teams, so they were very involved in relaying feedback on how the team was doing with working remotely, and sharing some of the communication challenges they were having, and then we worked together to find ways to improve.

As part of this year’s Engineering Leadership Summit: Virtual Edition, we spoke to Roger Deetz, VP of Technology at Springbuk. He discussed the role of a leader in fostering company culture, and his strategies for using engineering metrics to encourage healthy coding habits at scale. Below is an excerpt from the fireside chat portion of Roger’s session, edited for length and clarity.

Hillary Nussbaum, Content Marketing Manager, Code Climate: I’ll let you get started, Roger. Can you introduce yourself and tell us a little bit about your role at Springbuk?

Roger Deetz, VP of Technology at Springbuk: I lead the Technology Team at Springbuk. For us, that includes our Software Engineering teams, our Software QA teams, and our Infrastructure Group, which includes DevOps, Security, and our one-person IT organization, as well as our Data Quality team.

Springbuk does health analytics and health intelligence. We deal with tons of healthcare data. The quality of that data is critical to our customers, so we’ve got a group of folks, data analysts, who work on that. Those are all part of the Springbuk technology team.

Tell us a little bit about your particular role as VP of Technology, and how you got there.

In the day-to-day at Springbuk, we are building a platform, executing products and features. So I am partnering a lot with product, partnering with sales, partnering with our support and services team — who are ultimately servicing the product when a customer is using it — and balancing the needs of engineering execution with managing the costs of our infrastructure and the quality of our data.

I’m just representing the technology, as a whole. At Springbuk, we really think that we have an important mission, a mission that someday we can prevent disease with data. That requires good data, requires good technology. We’re all rallied around that cause.

In terms of my background and how I got there, I started as an engineer. I spent the beginning of my career writing code and bouncing a little bit back and forth between being an engineer and being more of a UX person. I moved to a few different companies. I had a chance to work on some more sophisticated, architecture-type projects, where I was designing a framework, and setting up the patterns that were going to be used across the product, across the enterprise.

That was when I started to get a little bit of team leadership skills. It was informally, and people weren’t reporting to me, but I was responsible for leading initiatives. I probably should say, I was a very reluctant manager. I viewed myself as the technical person, and I remember experiences with managers where I didn’t know what they did, and I didn’t know why we had managers.

I was reluctant to officially move into a management position. Then I realized that it was my path, and it was my calling, and there is a real craft to it. One of my first main management roles, I took on at a company called Angie’s List, which is based here in Indianapolis, and I was part of leading the engineering team there.

And then, I met some of the Springbuk folks at a community networking event. They were pitching the Springbuk product and platform, and they were getting ready to scale up their company and their engineering org, and I really enjoyed the challenge of being part of a growing team, and being part of a company that was looking to scale. I built a relationship with some of the folks there, and then I eventually joined, and I have been very excited and very proud to be part of that period of Springbuk’s growth.

Just to get us oriented a little bit, do you happen to remember how big the engineering team was when you started? And where are you at now?

When I joined, I was brought in as the leader of just the engineering group. I feel like it was about 10 folks at the time. We had two small delivery teams that were part of the engineering group. And then, as with any scale up, it was always lots of growth and change. We’re now at basically five delivery teams, and then some of those other technical functions, like infrastructure, came into my group over a period of time.

We’ve got a little more than 30 folks in tech right now. Springbuk, as a whole, has a little more than 100 people.

And what is it like to work at Springbuk? Can you tell us a little bit about the company culture?

Culture has been a really important part of Springbuk from the beginning. Our co-founders, Rod and Phil, were very intentional in knowing that culture would be important, and knowing that it needed to be rooted in the company from the beginning. They put a really big emphasis on that.

I don’t know if this is the dictionary definition or not, but I think about culture as the expression of the values of a group of people. Those values are the shared beliefs of those people.

At Springbuk, there are three core values that we talk about all the time, and they’re really important to us. They go by the short names of Win Together, Raise the Bar, and Never Settle. I think Win Together is pretty self-explanatory, because you want folks to cooperate on a common goal. Raise the Bar is all about leveling up and going to the next step, whether it’s as a company or as individuals. And then, Never Settle is really more focused on the individual. We want all individuals to pursue their own growth paths, and Springbuk wants to support them on their growth path.

We take those values pretty seriously. We fold them into our job descriptions, and our onboarding, and a lot of other materials. We talk about them all the time. You hear it in the language of the company, “Hey, this is a real Win Together moment,” or, “you really raised the bar there.” I think when you hear people just naturally say that as they’re talking, that’s a good indicator that the culture is woven in.

It does sound like it’s really persistent throughout the company. What do you think is your role as a leader, in terms of encouraging that, and cultivating and maintaining the company culture?

That’s a big question, I love it. I love a big question. I was thinking about this quite a bit earlier. If you think about the culture as the expression of values, the values are that social contract that everybody in a group has. We all believe in these same things, and we agree that we’re going to behave in this way. I know I’m going to behave this way, and I expect you to behave that way. That forms the social contract of a group.

Those expectations have to be there before any kind of trust can be built. And then, I think it’s fairly well-known that trust is then the foundation of everything else. Cooperation, and psychological safety, and mutual respect can only be there if there’s trust. Trust really comes from those shared values and those shared expectations.

That could apply to anything; a team, a company, a family, a whole nation. And so, as a leader, I really think the role is to foster an environment where that social contract is upheld and everyone knows that it’s going to be upheld. There’s a million things that then go into that — modeling your own behavior, coaching folks, all that sort of thing. Easier said than done, maybe.

As a leader, you’re accountable for what happens, even if you didn’t do it. That’s the essence of leadership. No individual can scale, so how can you get a group of folks to do something together, when you, yourself, can’t do it? The only way that happens is if people have the expectation that everybody else is going to uphold the contract. Right? They can’t be wasting any mental energy on, “What does this person believe?” or, “How are they going to behave in this situation?”

You have to set the tone that everyone just knows how it’s going to be, so then all your mental energy can be spent on doing the work together. As a leader, it all just comes back to making sure that the environment upholds the values of your organization.

It’s interesting that you said that Springbuk’s mission is to prevent disease with data, because I know you also have a data-driven approach to leadership, and a way to use metrics to help cultivate company culture. Can you tell us a little bit about that, and how that works?

Yeah. I also should probably confess that I’ve been leading engineering teams for a while, and have long been a skeptic when it comes to engineering metrics. I think many of us have had the trauma of a non-technical stakeholder or boss who doesn’t get it, and is looking for that one magic thing. We know that people in general and engineers specifically, we can really over-optimize for something. You say you’re looking for one thing, and they will make that one thing happen, even if it’s not aligned with the best outcome.

I’ve been part of high-performing teams and part of low-performing teams, and I spent a lot of time thinking — what are the habits of those teams and of those engineers that actually demonstrated that they were high-performing? We in the tech leadership team at Springbuk, we’ve thought about that. What are the habits we see, and then what are the data that might support those habits?

Making small changes, frequent commits, pushing regularly, small pull requests, prompt code reviews. All those things are good habits because they lead to small changes that are easy to review, easy to test, easy to get out the door.

And so then, we thought, that’s the kind of engineering behavior we want — lots of little small things going out quickly that we have high confidence in — what’s some of the data that would reveal that? We chose some metrics around Coding Days, which essentially is, “Are you pushing stuff every day? Are you making small commits?” Things around review time, and how quickly we’re getting to reviews, which is a function of how big the code change is. Things about throughput, like Number of PRs Merged, which is a lagging indicator — if all the changes are small, and if things are being quickly reviewed, then more code’s going to go through.

And then, of course, we have a whole other set of metrics around the actual performance of the app, like Page Load Time, and Uptime, and Data Quality, and all that other stuff. Those are more outcome-based. When it came to the engineering behavior, we started with those and used those to steer the ship.

How did the initial introduction of those metrics happen?  

We, from the very beginning, felt like we needed to be very transparent about it all. We didn’t want to have secret metrics that the managers knew about, that no one else knew we were watching. We did an intentional rollout with the whole team, to say “Hey, this is the stuff that we’re looking at, and this is why.”

We’ve adjusted that over time. I usually do a yearly kick-off with the team, and we look at some of the metrics from last year, and we talk about what we’re going to be looking at this year. We have that documented in some of our internal materials, and we cover that during onboarding. All of our engineers have access to the stuff that we use to look at those metrics.

One thing I think is really important is that we do take the metrics seriously, and we look at them, but we also realize that they’re just … They’re conversation starters. The outcomes are, are customers happy? And are we working towards our mission?

That makes total sense. We often talk about how it’s important to be transparent with your team and create that, as you mentioned before, that culture and foundation of trust. That’s really important.

Can you tell us a little bit about how you’ve changed which metrics you look at over time? You mentioned that you talk about that every year. Are there any that you’ve stopped tracking or started tracking, based on those conversations?

Yeah. We started very simple. We started with just Coding Days as our metric. That was, again, seeking to measure that healthy habit of, “Are you making small commits and pushing them frequently?” And then, over time, we expanded and we included Percent of Sprint Complete.

I’ve had this conversation a handful of times in my career, where people, particularly folks that aren’t familiar with agile practices, will be like, “How many story points can you do? Why is this team not doing 35 story points, when this other team is? What’s better?” You can’t compare, because story points are relative.

We did want to have some measurement around what I would consider to be the healthy habit of, “Can you accurately forecast what you can do, and then can you rally to complete that thing?”

We started to track our Percent of Sprint Complete, which is basically, when you planned the sprint, how many points did you and the team think that you could do, and then how many of those did you finish when the sprint was over?

We target 80%, because we know it’s not an exact science, and it’s not something we’re trying to be punitive about. It’s just that as professional, skilled engineers, you should have a pretty good idea of how much you can do in a sprint, and you should be able to accurately forecast that. And then when your team is working, you should rally around to finish what you said you would do.

So we introduced that one to keep an eye on how well we were forecasting and how well we were completing. And then, over time, we added things like Time to Review and PRs Merged. We’ve had a steady accumulation.

I love that you’re really iterating on this process, and that you have the awareness that the goal isn’t necessarily to get to 100%, because there are always exceptions. Setting realistic targets that are still aspirational, that’s great.

Let’s move a little bit from talking about things on the team level, to talking about things on the individual level. How do you use these metrics for professional development, and for helping your individual team members improve?

It’s part of our general cycle of continuous feedback. We don’t really do annual performance reviews. We do have quarterly check-ins, where we make sure that folks are progressing and exhibiting our core values, and some of those metrics will come up in one-on-one conversations, or in those quarterly reviews.

Again, it’s more of a conversation starter that leads us to ask more questions. There’s no right or wrong value for a particular number.

Can you share what that conversation might look like? If you’re in a one-on-one, and you have a particular metric or a particular data point, how do you introduce it, and how do you start the conversation?

We would lay out a lot of the things I touched on earlier, in the sense that we look at this stuff to encourage healthy habits. Whether it’s frequent commits, or getting to code reviews on time, or forecasting, we’ll start with what we think is a good quantitative expression of that habit, and then talk about what we see, and where you are.

That leads then into a conversation about your routine, or other behaviors, so we can figure out opportunities for coaching and continuous improvement.

“Meetings are expensive in person, and they’re even more expensive remotely. There’s a psychological overhead to bringing people together in a remote setting,” told us Noah Davis in his webinar on how engineering leaders can use data to run more effective remote meetings.

As teams go remote, the function of meetings changes. Team members lack context and use meetings to play catch-up rather than to discuss and make critical improvements. But when managers use remote meetings as status updates, they’re squandering the opportunity to help their team in the long-term.

In addition to advising hundreds of customers on data-driven best practices, Code Climate Co-Founder and VP of Customer Success, Noah Davis has nearly a decade of experience helping both developers and managers improve the speed and quality of engineering work.

"Assume best intentions of your engineers. Focus on the work, not the individual. Look for ways you can help."

In this webinar, Noah explained how to dig into engineering data from Velocity reports to get your team on the same page and keep meetings focused and productive. He shared actionable tips for spotting daily, weekly, and monthly trends in your data, so you can ensure that:

• Stand-ups are about unblocking engineers, not catching managers up
• Retros are about helping your team diagnose problems and process fault lines, not venting
• 1:1s are about constructive feedback and career development, not micromanaging

Noah pointed out that by walking into meetings already context-aware, you can replace check-ins with more informed conversations about how the team can stay on track.

Access the full webinar for free here.