The Customer Gap

In July, we hosted the first annual Code Climate Summit, a one-day conference for leaders of engineering organizations who want to better themselves, their processes, and their teams.

Today we’re sharing The Customer Gap, presented by Code Climate’s Director of Engineering, Gordon Diggs, and Code Climate’s Customer Support Lead, Abby Armada. In this talk, you will hear about how we’ve fostered a positive, productive relationship between customer support and engineering. By creating processes, and encouraging cross-team collaboration, we can combine our strengths to best focus on customers of current and future products.

Transcript of talk

Abby Armada: Before we get into this, I wanted to highlight this customer quote, and following exchange between our customer support and engineering teams. So: “Awesome dedication by the engineering and customer support team. Thanks, Jenna, Abby, and Ashley.” And there’s Jenna, who’s on customer support, saying, “You da best.” And Ashley saying, “No, you da best.” So, you can see this is the kind of vibe your team can have by closing the customer gap.

I’m Abby. I’m the customer support lead here at Code Climate. And including me our awesome team is three people, one being remote. I’ve been at Code Climate for over a year and in various customer-facing roles for about 13 years. I really like running and I really like eat tacos, but not at the same time.

Gordon Diggs: Hi, I’m Gordon. I’m a record collector and I like to cook lasagna. I do sometimes do those two things at the same time. I lead Code Climate’s engineering team and I’m responsible for growing the team in both size, in terms of hiring and, for lack of a better word, “mentality” in terms of our processes and the kinds of ways that we work on things.

Engineering at Code Climate takes up about 40% of the entire company. So, we’re a sizeable department. And Abby and I have actually been discussing and collaborating on the support and engineering relationship for over four years. So we’re really excited to sort of have this culmination of a lot of the stuff that we’ve talked about for so long and share some thoughts with you.

Many companies list customer focus as a core value, but few engineers spend time talking to their customers.

When you think of customer focused companies, you may think of companies like Amazon, where everyone trains in support center and answers calls, or Apple where the CEO regularly works on the support team. And so, many companies list customer focus as a core value. But few engineers spend time actually talking to their customers. So you have the big companies like Amazon and Apple, but many smaller companies that we found also list customer focus as a core value. A lot of people want to do this. They want to be a customer focused company, but it’s really hard and it’s time consuming.

The customer gap: The delta between a stated customer focus and the reality of how engineering spends its time.

And so we coined this term “the customer gap.” We define the customer gap as the delta between the stated customer focus and the reality of how engineering spends its time. Engineering tends to be very separated from our customers. But there are also gaps between your strictly customer-facing departments and your engineering team. If you can close the gap between engineering and those other departments, you can also work to close the customer gap.

Over the years we’ve talked extensively about the gap between customer support and engineering and we think that in order to close that gap, there are things engineering should do, things customer support can do, and things that we should do together.

As a little bit of a visual aid, I made this kind of cheesy chart. But you can sort of see the idea here. You have your customers on the left. You have a small gap between them and your strictly customer-facing departments. And then a wider gap between those departments and your engineering team. And so, one of the things that this shows is that your customer-facing departments are doing a better job of closing the customer gap than you are already, and that’s why they’re closer to the users. And then there’s this larger space between them and the engineering. So, if you can work on closing this gap on the right, then you can really work on closing the greater gap.

Abby Armada: Here’s a brief outline of what we’re going to cover in this talk. We’ll start by talking about ways to build customer empathy within your engineering team. And next, we’ll talk about some examples of processes customer support can implement to work more closely with engineering. And lastly, we’ll talk about closing gaps between engineering and other departments.

Building customer empathy within engineering

Gordon Diggs: Building customer empathy within engineering. This is a really important part of the formula because if your engineers don’t understand your users or their experience, or aren’t bought into this idea of customer empathy and focus, it’ll be a lot harder to close the gap. Ultimately, you really need to get them bought in.

And if I’m being honest, we at Code Climate have this kind of easy. Our customers are software engineers so it’s not a big gap for us, as engineers, to understand our users and their motivations. If we were some kind of printed product, a fashion company, or some kind of feminine care start up, we couldn’t be sure that 100% of our team has something in common with our users and understands them.

So, we can go to conferences, and we can go to meetups, and we can meet people who use our product or who want to use it or who we think should use it because it would really help their workflows. So, this is just a little bit easier for us. But it is also a little bit of a trap. Software engineers are a very diverse group of people and we need to make sure that we don’t project our own biases and assumptions when thinking about our users.

For a few examples: We at Code Climate are a team of Ruby engineers. We work at a small company. We have a good continuous deployment pipeline, and we generally have good coding practices as the nature of the work that we do. But people who buy and use Code Climate come from wildly different perspectives. A lot of them work on very, very large teams, and with tools that we don’t use and understand as well as our own. So while we can get pretty far working with our idea of software engineering, we need to be aware of where that ends and where our customer’s experience starts.

So, it’s really important that customer focus and empathy is a cultural driver. And you need to build that empathy into your engineering culture. It’s not enough to only think about focusing on your customers when things are broken or when your site is down. And so, what does that look like? What does it look like to build empathy into your engineering culture?

First, make your engineers talk to your customers on a regular basis. Include them in customer site visits. Put them on your sales calls. Make them work with support on debugging customer issues. And send them to conferences or meetups or if there’s some kind of trade show that’s relevant to your company, send them to that and put them at the table that you’re sponsoring.

Secondly, we put every mention of Code Climate on Twitter into a Slack channel. So, good and bad, engineers can see what people are saying about Code Climate online. If we ship something and it’s a poor user experience, or the CTA was a really weird color or something and people start complaining about that, we’re going to see it. And similarly, when an engineer ships something that really resonates with a group of users and they start talking about it online, there’s a nice boost there. So, you can kind of get both sides of the coin there.

We also put engineers on the frontline of responding to our community in both our community slack group and on our open source repositories. If you open an issue or a pull request on one of our repos, it doesn’t go to a member of our support team. It goes directly to an engineer who’s responsible for triaging that issue and finding the solution.

Our engineers also run our status page. If our site goes down or our service is in any way degraded, they’re the ones who are responsible for keeping our users up to date about what’s going on. Obviously, they collaborate with marketing and customer support on these updates to make sure that they are of the highest quality that they can be, but ultimately, the timing and the content is up to them.

And lastly, watch your users use the product. One issue that we have as engineers, particularly product engineers is that we know the code that goes into the site and we know that if a page is slow or a user experience is weird, it’s often because the code behind it is complicated. And we bring this bias into using our own product that, “Well, this page is slow because there’s all this nested logic in the templates and stuff.” But your users don’t understand that. They don’t have any of that. And watching them interact with your product, watching the way that they use it, and where they get frustrated, will really help eradicate some of that handwaviness.

So, as with most cultural efforts, building empathy is harder on bigger teams.

Panna actually touched on this a little bit earlier. So, if you can start when your team is small, that’s better. The bigger your team is, the more people you need to convince to buy in to this idea of customer focus and empathy. So, if you’re at a small company, that’s great. You’re in a really prime position to effect change across your team. If you’re at a larger company, all hope is not lost. You can start with a small subset of your team, find a pilot team, and have engineering managers of that team spread their success laterally throughout the organization.

And it’s really important, I really want to mention that it’s not enough to silo this empathy. You can do all you want to build customer focus within your engineering team, but it’s also important to work closely with your support team. And Abby’s going to talk a little bit about some of the ways to do that.

Helping each other

Abby Armada: So, you have to work together. There are a couple of solutions we created here at Code Climate to help both the engineering and support teams be successful at this. These solutions may seem obvious in some ways but the execution and upkeep is essential for it to work well and close the gap, which in turn closes the other gap that exists between engineering and your customers.

The first of these solutions on our side was addressing escalations, which is when a customer’s problem has gone past a scope of troubleshooting and knowledge of the support team and requires a solution from an engineer or someone else. I’m sure many of you here have had to deal with customer escalations in one way or another, and it might have been a thorn in your side. We did a lot of work to improve this process for both of our teams, which ultimately has made our customers way happier.

Our main channel of support is email, then it trickles through to Twitter, Slack, and sometimes GitHub Issues. This is where all of our escalations come from.

So, how did we address escalations before? Engineers worked on escalations on a weekly rotation. This wasn’t ideal for anyone. For the incoming engineer, there wasn’t a lot of context for existing open issues. Plus, it seemed like a giant chore that was an interruption to their regular work. On support’s side, it’s hard to ramp up someone for this work, and it’s especially difficult to form a good working relationship with your new engineer for only a one-week rotation.

We didn’t have a great way of tracking this work, or a consistent process for handing off open issues to the new engineer, or giving enough context for the problems. Prioritizing issues did not exist at all, and we kind of thought they would just figure it out. This led to everyone drowning in a sea of confusion and sadness.

To fix this we came up with a couple of solutions. Instead of a weekly rotation, we now have a dedicated support engineer for a quarter. This solved that problem of feeling like they’re interrupting other work and gives the engineer a chance to fully work and focus on escalations. It also builds a great rapport between engineering and support because we get to know each other for a quarter.

And we broke down our escalations by severity, which gives us an actual prioritization system that makes sense to both teams. There are now four levels of severity, one being the highest and four being the lowest. This isn’t a new concept by any means, but implementing something concrete was the most important part of the solution.

We also documented how to respond to each severity, both as a support person, and as an engineer, and holistically as an organization. This is written in our company handbook so everyone can see and have the knowledge. We also have concrete examples in the doc for easy reference. So, if someone is confused, they can look at the doc examples and know how to assign an issue a severity.

And lastly, we started using a GitHub repo and issues to track and update customer escalations. Everyone in the company already knows how to use GitHub, so implementing this part was the easiest.

This is an example of our severity documentation. You don’t need to read the whole thing – and the people in the back probably can’t anyway – but you can see the structure of how we define severities. In this case, this is a severity three, normal/minor impact. It’s something that’s like a moderate loss of application functionality that doesn’t really affect any of their other workflows. It matches the description. And then the examples live underneath. In this case, a customer reports a bug in an engine that’s kind of broken but doesn’t have any effect anything else that they’re doing. And then the response plan underneath details what both the customer support person and the engineers can do to sort of solve this issue.

Another change is that severity ones are treated differently than other severities within our organization. A severity one is a major issue. So, for example, if a customer’s instance of Code Climate Enterprise is down, we flag it as severity one and it requires all hands on deck to fix.

In past quarters, we found that the burden of solving these types of escalations is hard for a single escalations engineer. They often had to ask for extra help anyway. So we changed severity ones to be treated as production incidents. Support escalates directly to our engineering teams existing PagerDuty rotation. We have a fairly robust alerting pipeline for codeclimate.com, so most issues are caught by other alerts before they even get to support. It was pretty easy for us to integrate these other issues into that rotation. Distributing the work amongst those who are already on call helps solve severity ones more quickly and taps into existing expertise.

After adopting the aforementioned quarterly rotation, it helped highlight gaps in our own team’s knowledge about troubleshooting in other parts of the product. Thus, engineering has started leading proactive education workshops to teach concepts that will help us troubleshoot future issues. For instance, one of our support engineers noticed that we were pretty much escalating every single issue that had to do with our enterprise product. He set up a workshop to talk through troubleshooting concepts and how engineering looks at the same issues. This helped the support team work through more enterprise issues and lessen those types of escalations.

Engineers really benefit from this, too. Having them explain their work and their contributions to the product helps them grow and build empathy, and again, really strengthens that great rapport between your support and engineering teams. And informing the support team empowered us to troubleshoot and triage more effectively and even nipped future escalations in the bud.

Here’s our resolution time for escalations per month. So, after a slight rise due to those new processes, especially proactive education, we were able to have a fast mean time to resolution to customers than before and our customers, obviously, are really happy about that. And here are our escalations per month. You can see the tangible benefits of when we adopted these new processes and it’s because of that quarterly rotation, better processes around triaging, and proactive education. On average, our monthly escalations have gone down month to month, and that’s great.

Having worked on our escalation process, the next thing we did was take a look at the product engineering and support relationship. It’s a common problem that support doesn’t have the full picture of what’s being developed. I hear this all the time from other support professionals that I interact with every day, and this is a problem for everyone.

Even on a small team, communication is oversaturated. There are too many Slack posts and GitHub issues, and this is not sustainable at all. There’s just too much to keep up with. Last quarter, we established a weekly customer support and product meeting to sync up about work being done that week, as well as talk about things coming through the pipeline. Instead of trying to keep up with endless notifications, now we talk face-to-face and it’s much easier.

Personally, this is my favorite meeting I have every week because it’s really productive for my team and the product team, and it really gives my team great perspective on what’s happening in our organization. So, we talk about work in progress. What’s the progress on last week? What’s happening this week? We cover any upcoming releases. We get to answer the question of what exactly are we shipping and when? And how does it affect our customers? Then we talk about any customer-facing communications needed like updating docs, release notes, as well as inquiry response to incoming new customer questions. And lastly, we talk about feature requests. I’ll cover that in a little bit more detail in a bit.

This has helped our teams a lot. I share the most important knowledge back to my team since one of my people is remote, it’s good that we can be all on the same page about what’s happening that week. And it also helps us action any internal work that helps that product work. And it has also stopped confusion about what exactly is being released. We’re more confident and thus can help our customers better. And lastly, it fosters trust, again, between support and product. We know what’s going to be released and how to handle it.

Now that we have that open channel of communication with product and engineering, we decided to tackle the beast; feature requests. We all want to be the type of company that welcomes customer feedback, and feature requests, and actually actions on it. But I think this is the hardest gap to close but we’ve made a lot of steps here at Code Climate to do so.

Before, support had ways to catalog feature requests, but no way to surface them in a meaningful way for people to action them. So, we had a very bad GitHub repo, then a very bad Trello board, and – hey, Gordon, did you look at any of those ever?

Gordon Diggs: Ehhhhhhh…

Abby Armada: Yeah, me neither. There’s a limited process for both of the repos and the Trello board, which also housed bugs alongside feedback. Someone would make an issue, which would live on in perpetuity. And every time a request came up again, someone would comment on it and push it to the top. It sounds good in theory, but no one ever looked at them. Just having the tools isn’t enough. We thought that switching to Trello would solve the problem of stale, sad feature requests. But, in fact, you just needed a process.

The more you can get feedback into the eyeballs of your company, the better. You have to be loud to get this actioned.

We created a feedback repo on GitHub, again, which feeds into a Slack channel, then we talk about important feedback during that product meeting I mentioned before. The more you can get feedback into the eyeballs of your company, the better. You have to be loud to get this actioned.

This is what that GitHub repo looks like. You can see the different issues opened by the people on our team. And we use labels to easily identify the status of each feature request. You can see stuff that’s been actioned, what needs to be reviewed, and the different parts of the site that the feature request is about. And now these are pure feature requests. They are not bugs or anything pertaining to escalations. These are all nice to haves and legitimate product feedback from our customers.

Of note, this repo is completely internal. There’s no way for a customer to directly add feedback to this repo. Only people at Code Climate can do this. And then the feedback is curated by me. We have an issues template that asks a bunch of relevant questions. If the feedback doesn’t clearly answer why a customer is asking for a feature or isn’t detailed enough, it gets rejected. This curation process keeps the board fresh and keeps my finger on the pulse of what our customers want.

This is that Slack channel with that piped in feedback activity. It has all new issues and comments on old issues. And they all get fed into this channel that anyone in our company can join. So, in this screenshot, Jenna opened an issue, pinged Noah to get his thoughts, which he then shared to the issue itself. Loud is good.

As I mentioned, we also talk about the feedback issues in that weekly product and customer support meeting and see if they’re still relevant and see where they fit within our product roadmap. Sometimes feedback and feature requests can alter a product roadmap, giving us ideas for something we didn’t even consider before. And through all of this, we’ve seen an immense improvement in adoption of customer feedback within product development. In fact, after we adopted these processes, 25% of feature requests were actioned by our product team, which is a huge improvement from basically zero.

You’ve heard me talk a lot about what we did at Code Climate to close the gaps between engineering and support, but these solutions might not necessarily work for your specific teams or solve the types of problems you’re facing between your customers, support, and engineering team. As an engineering manager, it’s up to you to try and work in tandem with your support team to solve these problems. Collaboration, communication, and iterating on processes together are the key ways to do this and it leads to happier customers overall.

Closing the gap to other departments

Gordon Diggs: We’ve talked about how to build customer empathy and we’ve talked about some of the processes that support and engineering have worked on together. But what about the other customer-facing departments in our organizations? Your customers move through life cycles, and engineering should follow them.

Before your customers are even your customers, they interact with your marketing and with your marketing team’s efforts. At Code Climate, we dedicated an engineer for a whole quarter to help build automation for our marketing lead pipeline. He did this by supplementing leads with data from a variety of sources and piping it all into our CRM. He learned about what makes leads more qualified from a marketing perspective and our pipeline is fuller and faster than it’s ever been. He also learned a lot about our user personas and our segments. And this ties back to that idea of the diversity of our users from before. He came back from the marketing team with a better understanding of our users and where they come from. Dedicating a full-time engineer to this rather than hiring a contractor or farming out to buying some product, ensures that we built this in the right way and in a way that we can easily maintain and extend moving forward.

Sales is a really important part of most businesses, but particularly at a SaaS product we need to have a sales team. And sales people usually aren’t engineers and may not be able to articulate the same engineering concepts. But they are very good at learning and they can really learn from your engineers. And there are a few ways that engineers can get involved with your sales department.

The first and maybe most obvious of these is building features for customers who are in your sales pipeline. Putting engineers on your sales team means that you can more quickly action the high value projects that will sign customers right away. So, we identified three key features blocking sales and then we built them. And additionally, in just having the conversation about what features are the users that you’re talking to, what features do they really want? In just having that conversation, we clarified our product direction and have a clearer idea of where we need to go.

There’s a very, very important note and caveat to this. Some features will be too big for this setup and some will be in conflict with your existing product roadmap. I’m not suggesting that you give your sales people carte blanche to implement features in your product. But, working through the discussion of the features that they want to build and finding the compromises will really help grow your business. It’ll help clarify your roadmap, and it’ll sign customers, which is really good.

So, another sales engineering activity is reviewing leads technically. This is a little bit specific to Code Climate as a highly technical product, but we have an on-premise enterprise product and we want to be sure that we only deploy it to platforms and customers that will be successful. Doing this once a customer is about to sign a contract causes the sales process to lose momentum, but if you review the technical requirements – for us that’s things like virtualization platform, version control system, programming languages, that kind of stuff – if you review that earlier in the process, your leads will be more successful.

Pulling engineers in to do this on an ad hoc basis was something that we used to do like, “Oh, yeah, just go grab someone off of product to review one of these leads.” But it meant more prep, more interruption, and generally lower quality reviews. So having people on your sales team ready to do those reviews is a really big strength.

Similarly, installing and setting up the product with customers gives engineers a really good sense of what your onboarding experience is like. In many cases, it’s been years since one of your engineers signed up for your product and, most likely, they’ve never done that with money on the line. So, having them sit down and understand what the customer is going through helps them build it better in the future and it helps get the customers up to speed faster. This also ties back to that idea that I was talking about earlier of watching your users use the product. This will really help eradicate some of those unconscious biases that your engineers bring to your onboarding experience.

Lastly, putting customers in touch with your engineers directly build rapport. It’s rare that you can jump on a sales call with a company and there’s an engineer there to tell you about how they built the product, particular features that they really like, that kind of stuff. It’s also rare that you can be in the Slack group and DM an engineer directly to ask them a question about how do I configure this? How do I work with this? And so, having that engineer there really helps build this rapport.

Abby talked extensively about customer support and shared lots of really good thoughts, but I want to throw this department up here because escalations were where we started. Both Abby and I started talking about this customer support engineering thing, and it was the first place that we experimented with this quarterly rotation idea. I also want to mention that being on the support team doesn’t mean just answering escalations and providing technical knowledge to the support team. A successful setup of an engineer on a support team means also building the features that ensures that those customer issues don’t happen again.

So, in the same way that sales engineers build features for customers in the sales pipeline, your support engineers should collaborate with product and engineering to action the features that will really help prevent future issues. So, you can see that drop in escalations per month, and a lot of that is because we found the patterns and where users were running into trouble and where they were writing into support, and built better experiences for them.

What about once someone is a customer but their experience isn’t broken? So, they’re not writing into your support team. Customer success works with customers who are configuring the product, who are discovering how to use it, and who are using it in their workflows everyday. Your engineers can help train users on how to use your product. They can go do customer visits. They can sit down and eat lunch with your users and show off their work or talk about things that are coming up that they’re working on.

I really love training customers, and working with them, and seeing them get excited about a feature that someone on my team built, or seeing them understand a complex part of our system. And if your customers are engineers like ours are, that’s even better because they’re going to want to know what it’s like, what the special sauce behind Code Climate is.

So, in both customer success and sales, the engineers are both training you customers and your representatives about the product. Your representatives may not know about the newest features or what’s in development. And if they do, they may not know about all of the edge-cases or the best applications for these features.

And ultimately, working with customers and with other departments makes your engineers more well-rounded. They’re still going to be great product engineers. They’re not going to stop wanting to build features. It’s fine. But they’re also going to be better at sales. They’re going to be better at support. They’re going to be better at marketing. And this will set them up for success both in your company and in their careers at large. We’re lucky because our quarterly rotation lends itself well to putting engineers on other teams and working directly with other customers.

Engineers also remember their customer interactions. And it informs the way that they build things in the future. They remember seeing that customers missed a CTA or ran into an error because of a configuration problem. And this memory or this interaction lingers with them the next time they build a feature or talk to another customer. And it really informs the way that they go about their work in the future. So, we’ve seen that engineers who do a tour on the customer support team come back to product and have a new way of looking at building product because they’re thinking about customers and they’re focusing on them more. To restate that, more experience with customers leads to better features.

More experience with customers leads to better features.

Engineers don’t like to be interrupted. We’ve talked a lot about building processes to reduce this interruption. Getting into flow is a really important thing for engineers. All the engineers in the room are nodding their heads. And we, as managers, can’t pretend to change that. We’re not going to change flow. We’re not going to solve that problem. But understanding why your support team is bugging you to fix issues and understanding their motivations and that they’re advocating for your users will really help your engineers grow and will make your product more stable.

And I’ll also mention, we made these slides and we did a rehearsal of this talk. And one of my engineers is like, “I don’t like the word nagging on this slide because I think it’s a negative thing.” And it is. It’s only nagging if there’s a problem and if it’s something that they don’t really want to be doing. Talking to support more often, doing these weekly check-ins, having the people there for them to discuss escalations with, will prevent the unnecessary interruptions and it won’t be nagging. It’ll just be talking to you. Support will be talking to you for a reason.

Abby Armada: Support doesn’t like to be interrupted either. We’re doing work too. So having dedicated time to talk to engineers really helps everyone and your customers.

And here’s that visual we presented before except we’ve closed the gap significantly. And our engineers are close to our customers than ever before.

Let’s review some takeaways and what you can do.

Gordon Diggs: Build empathy within your engineering team to foster a focus on your customers. Know where the gaps are between your engineers experience and your customers, and then close them. And start small. If you have a small engineering team, that’s great. Get everyone onboard right away. But if your organization is larger, find a pilot team and then recruit your leaders within engineering to spread their success to others.

Abby Armada: Make sure your processes promote good collaboration between support and engineering. Don’t be afraid to experiment and try new things and talk to each other. You’ll learn a lot about the ways that you can work together and make your customers happier.

Gordon Diggs: Include engineers in every stage of your customers’ life cycle. Find a way. Go to those teams in the strictly customer-facing departments and say, “How can engineering help here?” From marketing to sales to support to customer success, find out where the engineers can help and then put them on the teams for a significant amount of time.

Abby Armada: At the end of the day, if you’re truly customer focused, then happy customers will mean a happy business.

Gordon Diggs: Thank you. Obviously, Abby and I have lots of thoughts on this and we’ve talked about this for a long time. If you have thoughts about any of this stuff, please come talk to us. If these sound like particularly interesting problems to you that you want to help us solve, Code Climate is hiring for both engineering and support roles. So, please come talk to us. And yeah, thank you for coming today.

Abby Armada: Thanks!

Gordon Diggs: Thank you for listening to us.

Gordon is the Director of Engineering at Code Climate. He spends his days managing and growing the engineering team. When he is not at work, he can usually be found at the nearest record store or at home cooking lasagna.

Abby is the Customer Support Lead at Code Climate, and is passionate about great customer experiences. She’s currently working on developing her team, scaling support infrastructure, and finding the perfect taco in New York City.