Productivity

(Updated: February 26, 2024)

Born from frustration at the silos between development and operations teams, the DevOps philosophy encourages trust, collaboration, and the creation of multidisciplinary teams. As DevOps rose in popularity, DevOps Research and Assessment (DORA) began with the goal of gaining a better understanding of the practices, processes, and capabilities that enable teams to achieve a high velocity and performance when it comes to software delivery. The startup identified four key metrics — the “DORA Metrics” — that engineering teams can use to measure their performance in four critical areas.

This empowers engineering leaders, enabling them to benchmark their teams against the rest of the industry, identify opportunities to improve, and make changes to address them.

What is DORA?

DevOps Research and Assessment (DORA) is a startup created by Gene Kim and Jez Humble with Dr. Nicole Forsgren at the helm. Gene Kim and Jez Humble are best known for their best-selling books, such as The DevOps Handbook. Dr. Nicole Forsgren also joined the pair to co-author Accelerate in 2018.

The company provided assessments and reports on organizations’ DevOps capabilities. They aimed to understand what makes a team successful at delivering high-quality software, quickly. The startup was acquired by Google in 2018, and continues to be the largest research program of its kind. Each year, they survey tens of thousands of professionals, gathering data on key drivers of engineering delivery and performance. Their annual reports include key benchmarks, industry trends, and learnings that can help teams improve.

What are DORA metrics?

DORA metrics are a set of four measurements identified by DORA as the metrics most strongly correlated with success — they’re measurements that DevOps teams can use to gauge their performance. The four metrics are: Deployment Frequency, Mean Lead Time for Changes, Mean Time to Recover, and Change Failure Rate. They were identified by analyzing survey responses from over 31,000 professionals worldwide over a period of six years.

The team at DORA also identified performance benchmarks for each metric, outlining characteristics of Elite, High-Performing, Medium, and Low-Performing teams.

Deployment Frequency

Deployment Frequency (DF) measures the frequency at which code is successfully deployed to a production environment. It is a measure of a team’s average throughput over a period of time, and can be used to benchmark how often an engineering team is shipping value to customers.

Engineering teams generally strive to deploy as quickly and frequently as possible, getting new features into the hands of users to improve customer retention and stay ahead of the competition. More successful DevOps teams deliver smaller deployments more frequently, rather than batching everything up into a larger release that is deployed during a fixed window. High-performing teams deploy at least once a week, while teams at the top of their game — peak performers — deploy multiple times per day.

Low performance on this metric can inform teams that they may need to improve their automated testing and validation of new code. Another area to focus on could be breaking changes down into smaller chunks, and creating smaller pull requests (PRs)‌, or improving overall Deploy Volume.

Mean Lead Time for Changes

Mean Lead Time for Changes (MLTC) helps engineering leaders understand the efficiency of their development process once coding has begun. This metric measures how long it takes for a change to make it to a production environment by looking at the average time between the first commit made in a branch and when that branch is successfully running in production. It quantifies how quickly work will be delivered to customers, with the best teams able to go from commit to production in less than a day. Average teams have an MLTC of around one week.

Deployments can be delayed for a variety of reasons, including batching up related features and ongoing incidents, and it’s important that engineering leaders have an accurate understanding of how long it takes their team to get changes into production.

When trying to improve on this metric, leaders can analyze metrics corresponding to the stages of their development pipeline, like Time to Open, Time to First Review, and Time to Merge, to identify bottlenecks in their processes.

Teams looking to improve on this metric might consider breaking work into smaller chunks to reduce the size of PRs, boosting the efficiency of their code review process, or investing in automated testing and deployment processes.

Change Failure Rate

Change Failure Rate (CFR) is a calculation of the percentage of deployments causing a failure in production, and is found by dividing the number of incidents by the total number of deployments. This gives leaders insight into the quality of code being shipped and by extension, the amount of time the team spends fixing failures. Most DevOps teams can achieve a change failure rate between 0% and 15%.

When changes are being frequently deployed to production environments, bugs are all but inevitable. Sometimes these bugs are minor, but in some cases these can lead to major failures. It’s important to bear in mind that these shouldn’t be used as an occasion to place blame on a single person or team; however, it’s also vital that engineering leaders monitor how often these incidents happen.

This metric is an important counterpoint to the DF and MLTC metrics. Your team may be moving quickly, but you also want to ensure they’re delivering quality code — both stability and throughput are important to successful, high-performing DevOps teams.

To improve in this area, teams can look at reducing the work-in-progress (WIP) in their iterations, boosting the efficacy of their code review processes, or investing in automated testing.

Mean Time to Recovery

Mean Time to Recovery (MTTR) measures the ‌time it takes to restore a system to its usual functionality. For elite teams, this looks like being able to recover in under an hour, whereas for many teams, this is more likely to be under a day.

Failures happen, but the ability to quickly recover from a failure in a production environment is key to the success of DevOps teams. Improving MTTR requires DevOps teams to improve their observability so that failures can be identified and resolved quickly.

Additional actions that can improve this metric are: having an action plan for responders to consult, ensuring the team understands the process for addressing failures, and improving MLTC.

Making the most of DORA metrics

The DORA metrics are a great starting point for understanding the current state of an engineering team, or for assessing changes over time. But DORA metrics only tell part of the story.

To gain deeper insight, it’s valuable to view them alongside non-DORA metrics, like PR Size or Cycle Time. Correlations between certain metrics will help teams identify questions to ask, as well as highlight areas for improvement.

And though the DORA metrics are perhaps the most well-known element of the annual DORA report, the research team does frequently branch out to discuss other factors that influence engineering performance. In the 2023 report, for example, they dive into two additional areas of consideration for top-performing teams: code review and team culture. Looking at these dimensions in tandem with DORA metrics can help leaders enhance their understanding of their team.

It’s also important to note that, as there are no standard calculations for the four DORA metrics, they’re frequently measured differently, even among teams in the same organization. In order to draw accurate conclusions about speed and stability across teams, leaders will need to ensure that definitions and calculations for each metric are standardized throughout their organization.

Why should engineering leaders think about DORA metrics?

Making meaningful improvements to anything requires two elements — goals to work towards and evidence to establish progress. By establishing progress, this evidence can motivate teams to continue to work towards the goals they’ve set. DORA benchmarks give engineering leaders concrete objectives, which then break down further into the metrics that can be used for key results.

DORA metrics also provide insight into team performance. Looking at Change Failure Rate and Mean Time to Recover, leaders can help ensure that their teams are building robust services that experience minimal downtime. Similarly, monitoring Deployment Frequency and Mean Lead Time for Changes gives engineering leaders peace of mind that the team is working quickly. Together, the metrics provide insight into the team’s balance of speed and quality.

Learn more about DORA metrics from the expert himself. Watch our on-demand webinar featuring Nathen Harvey, Developer Advocate at DORA and Google Cloud.

Engineering productivity is notoriously difficult to measure. Classic metrics tend to focus exclusively on outputs, but this approach is flawed — in software engineering, quantity of output isn’t necessarily the most productive goal. Quality is an important complement to quantity, and a team that’s measuring success in lines of code written may sacrifice conciseness for volume. This can lead to bloated, buggy code. A team focused on the number of features shipped may prioritize simple features that they can get out the door quickly, rather than spending time on more complex features that can move the needle for the business. The SPACE Framework aims to address this by replacing output-focused measurements with a more nuanced approach to understanding and optimizing developer productivity.

This research-based framework offers five key dimensions that, viewed in concert, offer a more comprehensive view of a team’s status. These dimensions guide leaders toward measuring and improving factors that impact outputs, rather than focusing solely on the outputs themselves.

Importantly, SPACE foregrounds the developer. It recognizes that leaders must empower their team members with the ability to take ownership of their work, grow their skills, and get things done while contributing to the success of the team as a whole.

What is the SPACE Framework?

The SPACE Framework is a systematic approach to measuring, understanding, and optimizing engineering productivity. Outlined by researchers from Github, Microsoft, and the University of Victoria, it encourages leaders to look at productivity holistically, placing metrics in conversation with each other and linking them to team goals. It breaks engineering productivity into five dimensions, Satisfaction and Well-being; Performance; Activity; Communication and Collaboration; and Efficiency and Flow.

Satisfaction and Well-being – Most often measured by employee surveys, this dimension asks whether team members are fulfilled, happy, and practicing healthy work habits. Satisfaction and well-being are strongly correlated with productivity, and they may even be important leading indicators; unhappy teams that are highly productive are likely headed toward burnout if nothing is done to improve their well-being.

Of all five SPACE dimensions, satisfaction and well-being is the hardest to quantify. Objective metrics can’t directly measure developer happiness, but they can point to circumstances that are likely to cause dissatisfaction or burnout. By viewing quantitative metrics in tandem with qualitative information and survey data, engineering leaders can better assess team satisfaction.

With Code Climate's Software Engineering Intelligence (SEI) solution, leaders can view engineering metrics like Coding Balance to get a sense of developer workload. Coding Balance displays the percentage of contributors responsible for 80% of your team’s most impactful work. If work is not evenly distributed, overloaded team members may be overwhelmed, while those doing less impactful work may not be challenged enough.

Another informative measure, Work in Progress PRs, offers a count of Pull Requests with activity over a 72-hour period. A high Work in Progress number could mean that developers are forced to context switch often, which can prevent developers from entering a state of flow. The ideal number of Work in Progress PRs is usually 1-2 PRs per developer.

To get a better understanding of overall satisfaction, engineering leaders should look at both short term changes and overall trends. For example, when an engineering manager at Code Climate noticed a spike in volume of PRs and reviews in a short amount of time, she took it as a sign to further investigate the workload of each developer. When she discovered that engineers were working overtime and on weekends in order to ship new features, she successfully advocated for extra time off to recharge and avoid potential burnout.

Performance – The originators of the SPACE Framework recommend assessing performance based on the outcome of a developer’s work. This could be a measure of code quality, or of the impact their work has on the product’s success.

Metrics that help evaluate performance include Defect Rate and Change Failure Rate. Change Failure Rate is a key performance metric, as it measures the percentage of deployments causing a failure in production, helping engineering leaders understand the quality of the code developed and shipped to customers. Every failure in production takes away time from developing new features and ultimately has a negative impact on customers.

When assessing team performance, PR Throughput is an important counterpoint to quality metrics. This metric counts PRs merged (a bug fix, new feature, or improvement) over time, and is correlated to output and progress. To get a full picture of performance, PR Throughput should be viewed alongside quality metrics; you’ll want to ensure that your team is delivering at volume while maintaining quality.

Activity – This dimension is most reminiscent of older measures of productivity as it refers to developer outputs like on-call participation, pull requests opened, volume of code reviewed, or documents written. Still, the framework reminds leaders that activity should not be viewed in isolation, and it should always be placed in context with qualitative information and other metrics.  

Useful Activity metrics to look at include coding metrics, such as Commits Per Day, the average number of times Active Contributors commit code per coding day, and Pushes Per Day, the average number of times Active Contributors are pushing per day.

These metrics offer an overview of developer activity so you can gauge whether or not work is balanced, if workload matches expectations, and whether delivery capacity is in line with strategic goals.

It can also be helpful to look at Deployment Frequency, which measures how often teams deliver new features or bug fixes to production.  

Communication and Collaboration – The most effective teams have a high degree of transparency and communication. This helps ensure developers are aligned on priorities, understand how their work fits into broader initiatives, and can learn from each other. Proxies for measuring communication and collaboration might include review coverage or documentation quality.

Code Climate supports Pair Programming, which encourages collaboration. Credit can be attributed to two developers who write code in a pairing session.

Metrics like Review Speed, Time to first Review, and Review Coverage, can help you assess the health of your collaborative code review process. These metrics can help identify whether your team is keeping collaboration top of mind, has enough time to review PRs, and is giving thoughtful feedback in comments.

Efficiency and Flow – In the SPACE framework, flow refers to a state of individual efficiency where work can be completed quickly, with limited interruption. Efficiency is similar, but it occurs at the team level. Both are important for minimizing developer frustration. (Though it’s worth noting that when an organization has too much efficiency and flow, it’s not always a good thing, as it may be at the expense of collaboration or review.) Perceived efficiency and flow can be important information gathered via survey, while speed metrics can capture a more objective measurement.

In Code Climate's SEI platform, Cycle Time is an important metric to view within the context of efficiency and flow, as well as Mean Lead Time for Changes. Cycle Time, aka your speedometer, is representative of your team’s time to market. A low Cycle Time often means a higher output of new features and bug fixes being delivered to your customers. Cycle Time is an objective way to evaluate how process changes are affecting your team’s output. Often, a high output with quality code can speak to a team’s stability.

Mean Lead Time for Changes, which refers to DevOps speed, measures how quickly a change in code makes it through to production successfully. This can be correlated to team health and ability to react quickly. Like Cycle Time, it can indicate whether your team’s culture and processes are effective in handling a large volume of requests.

How can you implement the SPACE Framework?

The framework’s authors caution that looking at all five dimensions at once is likely to be counterproductive. They recommend choosing three areas that align with team priorities and company goals. What a team chooses to measure will send a strong signal about what the team values, and it’s important to be deliberate about that decision. They also remind leaders to be mindful of invisible work and to check for biases in their evaluations.

At Code Climate, we’ve spent years helping software engineering leaders leverage engineering data to boost their teams’ productivity, efficiency, and alignment. With that experience in mind, we have a few additional recommendations:

• Be transparent. When introducing data in your organization or using data in a new way, it’s important to be open. Make sure your team members know what data you’re looking at, how you plan to use it, and what value you hope to gain from it.
• Place all metrics in context. Quantitative data can only tell you so much. It’s critical to supplement metrics with qualitative information, or you’ll miss key insights. For example, if you’re measuring activity metrics and see a decrease in pull requests opened, it’s important to talk to your team members about why that’s happening. You may find out that they’ve been spending more time in meetings, so they can’t find time to code. Or you may learn that they’re coding less because they’re working to make sense of the technical direction. Both these situations may impact activity, but will need to be addressed in very different ways.
• Work data into your existing processes. The most effective way to build a new habit is to layer it onto an existing one. If you’re already holding regular 1:1s, standups, and retros, you can use them as an opportunity to present data to your team, assess progress toward goals, and discuss opportunities for improvement. By bringing data to your meetings, you can help keep conversations on track and grounded in facts.

Why does the SPACE Framework matter?

The SPACE Framework helps engineering leaders think holistically about developer productivity. The five dimensions serve as critical reminders that developer productivity is about more than the work of one individual, but about the way a team comes together to achieve a goal. And perhaps somewhat counterintuitively, understanding productivity is about more than simply measuring outputs. In looking at multiple dimensions, teams can gain a fuller understanding of the factors influencing their productivity, and set themselves up for sustainable, long-term success.

Learn how you can use data to enhance engineering performance and software delivery by requesting a consultation.

Technology leaders looking to drive high performance can support the evolution of their teams by creating a culture of feedback in their organization. Read on to find out why honest and constructive feedback is crucial to engineering success.  

The Importance of Routine Feedback

Regular feedback on particular units of work, processes, and best practices can be an effective way to help ensure agile teams are aligned and that all requirements are understood and met.

Employees who receive regular feedback on their work feel more confident and secure in their positions because they understand what is expected of them and the role they play in the organizational structure. According to one report, only 43% of highly-engaged employees received feedback on a weekly basis, while an astonishing 98% said they fail to be engaged when managers give little to no feedback, showcasing how essential feedback is to the cohesiveness of teams.

Furthermore, a positive culture of feedback can enhance psychological safety on teams. When feedback is a fundamental part of a blameless team culture, team members understand that feedback is critical to growing as a team and achieving key goals, and will likely feel more secure in sharing ideas, acknowledging weaknesses, and asking for help.

How to Foster a Feedback-rich Culture

As an executive, it is important to lead by example, as your actions can serve as a benchmark for best practices to be followed throughout the organization.

Leverage the Power of 1:1’s

Regular check-ins can play an integral role in building a successful feedback culture. Providing dedicated time for managers and direct reports to discuss goals, share feedback, and more, 1:1’s are a great way to engage employees and build trust.

During your 1:1’s, you’ll have the opportunity to ask questions and make statements that:

1. 1. Build an interpersonal connection – “How are you and your family”
   2. Encourage collaboration – “What’s on your mind? Is there anything I can help you with that you may be stuck on?”
   3. Show recognition, appreciation, or provide coaching

Here at Code Climate, our executives and managers hold weekly 1:1’s with their direct reports to surface any process blockers, discuss ambitions, and generally catch up. We are committed to this practice as a way to foster safety, transparency, and cohesion within our organization. We aim to ensure that each team member understands how their work fits into larger company initiatives and feels secure enough to share ideas that enable us to continually innovate and improve.  

Enhance Feedback with Engineering Data

The most effective feedback is specific and actionable, so it can be helpful to use engineering data to ground conversations in particular units of work. For example, if you have a question on a specific Pull Request, it can be helpful to surface detailed information on that particular PR to help you place progress in context and check any assumptions you may have. Objective data can also help you identify areas where coaching may be beneficial or showcase areas where you might not have known an IC was excelling. Furthermore, data can help you set and track progress towards goals, adding value and effectiveness to your feedback.

Feedback Goes Both Ways

A culture of feedback is most effective when it’s holistic, so it’s important to make sure you’re giving your team members the opportunity to impart their feedback on processes, workflow, expectations, and more. To help achieve this, you can provide your team members with a forum to share viewpoints during 1:1’s, standups, and retrospectives. In doing so, you regularly afford yourself the opportunity to actively listen to and learn from your team.

Feedback is an essential tool that helps leaders drive excellence on their teams. Remember:

• A culture of positive feedback helps create the ideal conditions for innovation
• 1:1’s can be a powerful way to build trust and create alignment
• Engineering data can help you place observations in context
• A true culture of feedback requires that team members are also empowered to speak up and offer their feedback

With a thoughtful approach, you can cultivate a safe and welcoming work environment that motivates teams to strive for excellence. To find out how to employ data-driven feedback in your organization, request a consultation.

This is the second post in a two-part series. Read the first post here.

When you use an Engineering Intelligence Solution to answer the classic three standup questions, you free up your team’s valuable meeting time for an even more impactful set of conversations. You’ll be able to spend less time gathering information and more time leveraging your expertise as a leader — once you’re no longer using facetime to surface issues, you can actually start solving them.

To frame these conversations, I recommend a new set of questions to ask in standup meetings:

• How can I help remove distractions?
• How can the team help reduce risk?
• Are we working on the right things?

Here’s how to get the most out of each one.

Question 1 to ask in standup meetings: How can I help remove distractions?

To most effectively answer this question, you’ll need to come to standup prepared. Take a look through your Engineering Intelligence Solution and keep an eye out for patterns that might indicate your team’s attention is fragmented, such as engineers who are making commits on multiple different Issues, or team members who have stopped committing for a few days. Code Climate's Software Engineering Intelligence (SEI) solution offers visualizations of individual developers’ coding activity, which makes these patterns particularly easy to spot.  

If you notice an engineer bouncing around, committing to multiple pieces of work in a short time period, it’s possible that engineer is trying to do too much at once. Have a conversion with that developer, and find out exactly what’s causing them to work in this way. It’s possible that they’re trying to make progress on multiple fronts at the same time, or that they’re being pulled in different directions by external forces, bouncing from project to project whenever a stakeholder asks a question or requests a status update. As their manager, you’re in a position to help the engineer develop the tools they need to devote the necessary level of focus to each unit of work. Exactly how you help depends on the situation — a more junior engineer may benefit from some coaching in project management skills, while a more senior engineer may need assistance contextualizing and prioritizing their work.

Of course, every engineer is juggling multiple things at once, and it’s not uncommon for someone to jump to a new project while they’re waiting for a Pull Request (PR) to be reviewed. Still, there’s a point at which task switching becomes excessive, and the visualizations in an SEI platform make that really clear, revealing potentially problematic switching in a haphazard assortment of commits, or delays getting back to paused units of work.

Sometimes, the delay itself is noteworthy. If an engineer is committing to something every day, then stops for a significant period of time before picking it back up again, it’s likely they’ve been pulled away to work on something else. Given how much time engineering teams spend grooming and prioritizing their backlog, this sort of deviation from the plan can be problematic, and yet, well-meaning developers often say yes to side tasks and additional projects without thinking about the cost. As a leader, if you notice this occurring, you can help your team members determine whether unplanned work is worthy of urgent attention, and remind them to assess any new tasks with you before taking them on.

Question 2 to ask in standup meetings: How can the team help reduce risk?

This question gets to one of the key goals of standup meetings: spotting and resolving problems before they derail sprints. Work that is at-risk is work that has started to go off track but hasn’t gone completely off the rails yet, which means you have the opportunity to stay a step ahead of a real problem. An SEI platform will alert you whenever a unit of work meets your criteria for “at-risk,” whether it’s a PR that has stalled without review or one that has too many contributing engineers, giving you the opportunity to quickly get the work back on track.

While some at-risk work can be indicative of larger issues — outdated processes or recurring bottlenecks in your development pipeline — at-risk work is often the result of a miscommunication regarding who should be doing what. For example, it’s common to see work get stuck in work in Code Review because everyone thought it was being reviewed by someone else on the team. When you bring risky units of work to standup for discussion, your team can quickly implement any straightforward fixes, then set aside time to work through bigger issues.

Question 3 to ask in standup meetings: Are we working on the right things?

With an SEI solution, you can easily see what units of work are progressing, so you can ensure your team members are working on the right things. Let’s say you prioritized tickets coming into your sprint, but notice that a developer is committing against the lowest priority Issue — you can discuss this as a team and rebuild alignment in the moment, rather than pretending you’ll always get it right at the top of the sprint. Almost certainly there was a miscommunication, and the developer jumped on what they thought was an important unit of work, though it’s also possible that an engineer thought the low-priority item was easiest to tackle and wanted to get it out of the way. This is rarely a good reason not to work on the highest priority things, and you can help coach the developer back to more important units of work. In some cases, the low-priority item may be the only thing the developer understood, so they jumped into it despite its low importance. This is important to know and address because if one person on the team doesn’t understand something, chances are other members of the team are also confused.

Whatever caused the low-priority item to be started, the team now has the opportunity to decide what to do about the unit of work in question. It may be that the work is so far along it makes sense to finish it, or it may be something the team decides to pause until higher-priority work is completed. Without a SEI solution, there would be no choice to make, as you’d only find out that someone was working on the ‘wrong’ thing after the work was already complete.

Data is a Powerful Tool for Skilled Managers

Though some leaders hesitate to leverage data, fearing that it breeds micromanagement, it’s important to remember that you’re already gathering information about stalled work or stuck engineers. Standups are one of the many ways managers seek out that information, but they’re not the most efficient way, since meeting time is better used for solving problems than surfacing them.

True micromanagement is not found in the gathering of data, but in how that data is used; micromanagers wield punishments and prescribe solutions, while great managers ask questions, offer guidance, and help their teams collaborate on solutions. With a Software Engineering Intelligence (SEI) solution pointing you towards work that needs attention, you’ll spend less time looking for opportunities to help and more time actually managing your team.

Effective engineering leaders today deftly balance the needs of their organization with the needs of their developers. Tasked with making strategic decisions, coaching their teams, and driving process improvements to meet business objectives and key results, leaders are often distanced from the actual work of writing code. As such, leaders must empower their team members to excel, and engineering intelligence data can help in a number of ways.  

Find out how data can help you cultivate an engineering environment that drives success in our new ebook, The Engineering Leader’s Guide to Empowering Excellence with Data.

Focusing on four key areas—removing blockers, minimizing micromanagement, personalizing coaching, and fostering a culture of psychological safety—our ebook will help you gain actionable insights from data, rather than gut feelings, to achieve a developer-focused work environment.  

With the right data, you can determine when to step in to lend support, and when to step back and encourage autonomy, so you can empower your team members to go above and beyond. Used thoughtfully, data can help you build stronger, more successful teams and drive continuous improvement.

An empowered team is a successful team. Download our ebook (for free!) today.

The final principle in the Agile manifesto urges developers to reflect on the past and use that knowledge to improve future outcomes. Since we learn from the past, holding sprint retrospectives is key to improving the results of future iterations. Conducted well, sprint retrospectives can boost outputs and propel teams forward; conducted poorly, they may breed toxicity. The careful use of objective data can help you steer your retro in the right direction — read on to find out how to leverage data from the beginning to the end of the retrospective process, so you can maximize the value of this key opportunity for continuous improvement.

Preparing for Your Sprint Retrospective  

Practices vary by organization, but sprint retrospectives may be facilitated by anyone familiar with the sprint, from a developer on the team to a stakeholder from another department. If you find yourself in the facilitator role, it’s crucial that you build a strong foundation for your retro by performing an audit to collect data in advance.

Look back on the lifetime of the sprint, and ask yourself questions like:

1. What was finished?
2. Did we deliver all the items we intended to? If not, why?  
3. What specific units of work didn’t get shipped?
4. What bottlenecks or blockers arose, and are they part of a pattern?

The answers will help you identify patterns and problem areas and formulate meaningful conversation points to guide the retrospective.

For example, if your sprint finished with a lot of unshipped work, you’ll want to know that in advance, so you can dig into the reasons during the retrospective. Look for unassigned tickets, which may indicate that some units of work were not prioritized correctly or that tickets were lost or overlooked unintentionally — though you’ll need to bring these tickets up at the retro to know for sure.

You’ll also want to look at the Issues from the iteration that are still categorized as In Progress, and see how many days they’ve been open. You can dig deeper by looking at the Pull Requests (PRs) associated with that Issue, and taking a look at relevant activity and comments for each. This can help you formulate a hypothesis as to why a unit of work was unshipped. For example, Issues with many PRs may indicate that work was not batched efficiently, while PRs with high levels of Rework may signal that an engineer was struggling with a difficult area of the codebase, or unclear technical direction. You can further investigate that hypothesis during your retro by discussing particular units of work to gain additional context and information.

While you can piece together this information from your VCS and project management tools, gaining a holistic view can be tedious as this data is typically dispersed. A Software Engineering Intelligence solution, like Code Climate, can save time and add a layer of valuable insights by aggregating that data in a series of customizable dashboards and rich visualizations.

Prioritize Your Sprint Retrospective

Typically, retrospectives last approximately 30 minutes for each week of the sprint, so if your sprint was three weeks long, you may want to carve out an hour and a half for your retro. Keep this time frame in mind to help you prioritize speaking points and focus on conversation topics that will keep your team engaged and on task.

Once you have compiled a list of topics, see if you discover any common themes and group them together. It may be helpful to get the perspective of your team members when you reach this point. Here at Code Climate, our facilitators ask the team to vote on which items should be talked through first to ensure engagement and alignment.

Open the Floor to Collaboration

In order to have a productive retrospective — one that surfaces meaningful opportunities for improvement — the team must feel safe talking through any missteps. The purpose of a retrospective is to measure processes, not individuals, so it’s important to remind your team to focus on the work, and not on the people behind it. As you set the stage for your retro, keep in mind that the data you gathered during preparation is to be used purely as an empowerment tool. When used appropriately, data can keep the conversation grounded in facts and tackle negative biases, allowing you and your team to have genuine conversations about things that could have been done better without making developers feel singled out.

Discuss

Now to discuss. Based on the topics you prioritized, you can split your sprint retrospective discussion portion into easily digestible parts. Though format can vary based on team and personal preference, many teams focus on three categories using a “Start, Stop, Continue” exercise, which asks developers to provide feedback on the following:  

• Start: Actions we should start taking
• Stop: Actions we should stop or do away with
• Continue: Actions we should continue and codify

It can be helpful to use a visual aid to facilitate this exercise and keep the conversation on track. For in-person teams, that might mean distributing sticky notes that can be written on and affixed to a board; for remote teams, that might mean using a collaborative online platform like Trello. Take time to talk through each part, and…

Develop an Action Plan

By the end of the sprint retrospective, you and your team should have several actionable ideas to put into practice to help the next iteration go smoother. While this data is qualitative in nature, these new ideas can then be measured against the quantitative data (such as PR size) they are meant to improve during the next sprint, enabling you to enhance software development strategies as time goes on.  

Standardize Sprint Retrospectives and Keep Improving

Best practices are best utilized when reinforced. Each new retro you hold keeps you on the path of continuous improvement.  

While there is no golden rule as to how retros should be structured and held, some form of review is vital to achieving continuous improvement. By incorporating data into your retros, you can maximize the value of your discussions and help build a highly capable team that can successfully drive business goals.

Every day, engineering leaders ask their team members the same three standup questions to ensure the software development engine hasn’t come to a complete stop. It’s excruciatingly boring to be asked the same questions day in and day out, and it’s not a productive use of engineers’ time. These questions — the “are things moving?” questions — can be answered with the help of a Software Engineering Intelligence (SEI) solution, like Code Climate, freeing up valuable standup time for an even more impactful set of questions, the “how can we move faster?” questions. These are the questions that dig deeper into a team’s processes and practices, helping leaders identify opportunities for improvement and drive excellence on their teams.

In this two-post series, I’ll explain how data can help you answer the classic standup questions, then walk through the “next three questions,” that every engineering leader should be asking to level up their standups — and level up their team.

First, the classic questions:

1. What did you do yesterday?
2. What will you do today?
3. What (if anything) is blocking your progress?

Let’s dig in.

Standup questions one and two: What did you do yesterday? What will you do today?

These questions are meant to help you understand what progress your team has made so far, so you can assess your team’s output within the context of your current sprint or cycle. In comparing what’s been done so far to what’s planned, you can get a sense of the sprint’s status.

Rather than ask your developers for a rundown of yesterday’s completed tasks and today’s to-do list, let the data speak for itself. You could do this by running a Git diff for every branch, or you could let Code Climate's SEI platform do the work, drawing connections between Commits and Issues and providing an accurate view of what your team is working on. This can also help you check that the team is prioritizing appropriately, and working on the most impactful Issues. You can also review the Issues that have yet to be started to determine whether the team is saving the trickier work for last. This isn’t necessarily a problem, but it could indicate that a slowdown is to come, and may be worth discussing during standup.

Of course, data won’t tell you for sure whether your team will hit its deadline, but it will give you a big picture view of how your sprint is progressing and where the team might need your help.

Standup question three: What (if anything) is blocking your progress?

Once you have a high-level understanding of the progress of your current iteration, you’ll want to know if there’s anything that might throw it off track. Developers might be hesitant to call out blockers, preferring instead to solve problems themselves, or they may lack the context to foresee potential slowdowns. Data addresses both of those problems, though the relevant information has historically been harder to find, as it’s spread throughout your VCS and project management tools. This is where a Software Engineering Intelligence (SEI) solution is particularly valuable — Code Climate can even send you a customized alert when a particular unit of work is stuck or at risk.

Work that is stuck may have gone a few days without a Commit, or may be marked “In Progress” in Jira for an atypical length of time. Work that is at risk is work that is not moving through the software development pipeline as expected, and which may become a bottleneck if not addressed. Though the signs of risk vary from team to team, Pull Requests with high levels of Rework, or many comments or Review Cycles are worth further investigation, as are PRs that haven’t been picked up for review in a timely fashion.

Data can also help you identify possible blockers in developers’ collaboration patterns. Every engineer will have their own opinion of how the team is working, and though it’s important to understand how engineers are feeling about the current state of collaboration on the team, it’s critical to check any hidden biases with objective data.

Start by looking at the distribution of work across the team. Is there someone who is overloaded, or someone who isn’t getting work? Code Climate makes this easy with a Work in Progress metric that tells you how much each engineer is working on at a given time. Then, to get a sense of the team’s broad collaboration patterns, it can be helpful to determine how many engineers are working on the same Issue or Pull Request. This way, you can be aware of a possible “too many cooks” situation, or dependencies that may impact delivery.

So, what standup questions should you be asking instead?

When you enter standup with this information, you can skip past the classic three standup questions in favor of three more impactful questions:

• How can I help remove distractions?
• How can the team help resolve that risk?
• Are we working on the right things (see what people are working on, blend that with your context)?

The answers to these questions will help you move beyond a short-term focus on getting work done and help you get to the next level, where you’re focused on helping your team excel. Find out how in my next post.

This is the first post in a two-part series. Read the second post here.

Engineering metrics can be a powerful tool for tracking and communicating engineering progress, debugging processes, boosting team performance, and much more – but they must be wielded with care. When misused, metrics can backfire, creating confusion and resentment.

To help engineering leaders successfully leverage metrics, Code Climate has partnered with  LeadDev on a series of blog posts and webinars that explore the fundamentals of data-driven leadership. Drawing on the expertise of engineering leaders from a range of industries, we highlight real-world perspectives on the what, why, and how of measuring as an engineering leader. We touch on everything from selecting the best metrics to track for your organization, to introducing metrics successfully, and specific use cases, like using metrics to run more impactful standups.

Here are some key insights:

• “I’ve found that metrics are valuable in helping zero in on what might be getting in the team’s way. Instead of treating them as a way to judge performance, I believe metrics are most useful in bringing to light areas of opportunity.” Leslie Cohn-Wein, Engineering Manager, Netlify
• “First and foremost, I use metrics to help me identify what the biggest bottlenecks are across different teams. Without metrics, I’d be left to make this judgment based on hearsay instead of methodology. The second reason I use metrics is to understand changes over time, particularly as we undergo organizational changes or make investments in specific areas.” Abi Noda, Developer Experience Expert
• “Cycle Time is a metric that I consistently come back to — it’s a great proxy for engineering speed, and can be a useful high-level look at whether certain key decisions are having the desired impact. If we’re moving slower than we had been, I can then isolate parts of the engineering pipeline and investigate where exactly things are going off track.” James McGill, VP of Engineering, Code Climate
• “The outcomes of productive standups ripple throughout the engineering team. By discussing specific metrics in standups, you can tell if your team is moving forward. You can ensure that the risks are being addressed from an objective, quantifiable standpoint rather than opinion.” Khan Smith, VP of Product, Code Climate

To dig deeper into what engineering leaders are doing today with metrics, check out the full series.

Ready to get started with engineering metrics in your organization? Contact our product specialists.

A good leader can respond to issues and course correct in the moment, but a great leader is proactive about staying ahead of the curve. With the help of data provided by Engineering Intelligence tools, an Engineering Manager can gain visibility into the development pipeline and stay equipped with the knowledge needed to cut problems off at the pass. No matter where an EM is in their professional lifecycle, a proactive one can prioritize more successfully, strengthen coaching strategies, and boost team effectiveness in the short term, mid-term, and long term.  

Short Term Strategy: Spot Risk and Prevent Blockages

A lot of dedication goes into keeping sprints on track and software delivery on schedule. With so many moving parts in the development pipeline, an EM may find it tough to determine what needs their attention first, making it challenging to triage risks. However, by using a Software Engineering Intelligence platform — like Code Climate —  that conveys insights based on aggregated data, an EM can swiftly analyze coding progress and prioritize their task list to focus on what’s most important.  

For example, an EM can assess PR and Commit data to surface at-risk work — work that could benefit from their time and attention. If a Commit has had several days of inactivity and work on the associated PR has seemingly halted, it may indicate that the scope of the task is not clear to ICs, and that they require guidance. Or it may be a sign of task-switching, where too much Work In Progress pulls an IC’s focus and makes it difficult to complete work.

This is where data from a Software Engineering Intelligence platform is critical, as it can signal to a manager that a specific PR or Issue needs attention. Code Climate enables EMs to set Risk Alerts for PRs based on custom criteria, since risk thresholds vary from team to team. From there, the EM can use that information in standups, retros, and other conversations with ICs to help identify the root cause of the blocker and provide coaching where needed.

Mid-term Strategy: Improve Collaboration

As a proactive leader, an EM must understand the nuances of collaboration between all parties to ensure ICs and teams are working together effectively and prioritizing issues that are aligned with company goals. If teams fail to work cohesively, roadmaps may be thrown off course, deadlines may be missed, and knowledge silos may be created. Using their Engineering Intelligence tools, an EM can easily surface the quantitative data needed to gain visibility into team collaboration and interdepartmental alignment.  

When it comes to collaboration on their team, an EM might want to look at review metrics, like Review Count. Viewing the number of PRs reviewed by each contributor helps an EM understand how evenly reviews are distributed amongst their team. Using these insights, a manager can see which contributors are carrying out the most reviews, and redistribute if the burden for some is too high. Doing so will not only help keep work in balance, but the redistribution will expose ICs to different parts of the codebase and help prevent knowledge silos.

To look at collaboration between teams, an EM can rely on quantitative data to help surface signs of misalignment. Looking at coding activity in context with information from Jira can help an EM identify PRs that signal a lack of prioritization, such as untraceable or unplanned PRs. Since these PRs are not linked back to initial project plans, it may indicate possible misalignment.  

Long Term Strategy: Support Professional Growth and Improve Team Health

A proactive EM also needs to identify struggling IC’s, while simultaneously keeping high performers engaged and challenged to prevent boredom. This starts with understanding where each individual IC excels, where they want to go, and where they need to improve.

Using quantitative and qualitative data, an EM can gain a clearer understanding of what keeps each IC engaged, surface coaching opportunities, and improve collective team health. Qualitative data on each IC’s coding history — Commits, Pushes, Rework, Review Speed — can help signal where an IC performs well and surface areas where it might be useful for an EM to provide coaching. An EM can then use qualitative data from 1 on 1’s and retros to contextualize their observations, ask questions about particular units of work, or discuss recurring patterns.

For example, if an EM notes high levels of Rework, this signals an opportunity to open up a meaningful discussion with the IC to surface areas of confusion and help provide clarity. Or, an EM might see that an IC has infrequent code pushes and can coach the IC on good coding hygiene by helping them break work down into smaller, more manageable pieces that can be pushed more frequently.

Using a combination of both data sets, a manager can initiate valuable dialogue and create a professional development roadmap for each IC that will nurture engagement and minimize frustration.

Proactivity as an EM – The Long and Short of It

Proactivity is a skill that can be developed over time and enhanced with the help of data. Once empowered with the proper insights, EMs can more effectively monitor the health of their sprints, meet software delivery deadlines, keep engineers happy, and feel confident that they are well-informed and can make a marked, positive impact.