Software Engineering:

Why Has It Eluded

Data-Driven Management?
How Engineering has been operating in the dark
and what to do about it
OVERVIEW

Imagine yourself as a fly on the wall of an executive boardroom, listening to

the heads of each corporate division present an overview of the past year:

Marketing will share a range of metrics on lead conversion, cost per lead and
ROI on marketing spend; Sales will walk through a detailed, quantitative
breakdown of the sales funnel; Finance will present a broad set of Key
Performance Indicators; the Operations and Customer Retention teams will
similarly present a variety of critical metrics.

But, what about the software engineering organization? We can speak to

features delivered, story points completed and ticket velocity… but these are
all subjective measurements, not meaningful metrics based on hard data.

Even though much of a company’s value is directly tied to their investment in

software, most executives operate in the dark when it comes to understanding
the performance of their engineering team. Instinct and ‘gut feelings’ are the
best tools we’ve had to make decisions about budget items costing millions of
dollars.

Just as other disciplines have benefitted immensely from objective metrics and
actionable KPIs, forward-looking software organizations are experiencing the
same lift that has transformed the rest of the enterprise.

Today, gut feelings are being replaced with data.

© 2018 GITPRIME, INC. 2

HOW WE GOT HERE

Software engineering teams are made up of analytical, intelligent

individuals — so why aren’t they as metrics-driven as other
departments?

If you ask software engineering leaders why this is the case, you will
get a range of answers:

• Engineering is an art form—metrics cannot properly

reflect productivity
• Engineering data is not easily accessible
• We measure story points and ticket velocity

Of the answers above, the most legitimate is that engineering data is

not easily accessible. Until recently, data housed in git-based
repositories has been challenging to leverage, particularly if an
organization uses many repositories and if engineers use multiple
aliases to commit code.

However, this issue has been solved by a recent class of applications

that can measure and quantify data across git repositories.

With this hurdle cleared, the next question is how soon the software
engineering world will embrace data-driven management, the way
that every other organization in the executive boardroom has.

TURNING THE LIGHTS ON WITH DATA

Which brings us to the question of, if the software engineering

world is truly next in line to embrace a metrics-driven management
approach, what metrics will emerge as the industry standards?

The following is our take on the engineering metrics of the future,

based on our work with over 300 customers:

© 2018 GITPRIME, INC. 3

INSIGHT 01

Productivity and Output

The first question for a software engineering organization is whether its
total output and productivity are improving, compared to prior periods.
Software engineers are difficult to hire and often require ramp-up time, so
increasing the output of an existing team is the most immediate way to
move the needle on value delivered.

The most basic measurements of output, which are code volume (e.g.
Lines of Code) and commit volume, are deficient and not complete
indicators of the complexity or sophistication of work completed.

Therefore, we expect the industry to standardize around metrics that

better reflect the cognitive load of work completed, such as GitPrime’s
*More on Impact and
Impact metric. Impact attempts to answer the question: “Roughly how how it is calculated:
much cognitive load was carried when implementing these changes?”* http://gitpri.me/impact

Baselining past team output levels and measuring progress towards Over time, this engineering
systematic improvement is a clean way for engineering teams to generate team has steadily increased their
Impact to the codebase. Whether
and document productivity gains.
this is a result of hiring
additional engineers or
A team of 100 engineers may cost in excess of $10 Million a year in fully improving team performance,
loaded costs. So, documenting a 20% increase in output provides a path this graph paints a picture of
success that the rest of the
for engineering management to demonstrate multiple millions of dollars
organization can understand.
in value generated for their company.

© 2018 GITPRIME, INC. 4

INSIGHT 02

Commit and Pull Request Behavior

Receiving visibility to commits and pull requests as they occur, as well as
to their content, code profile and complexity, is a direct way for managers
to better understand the progress, and challenges, of their team members.

Software engineering managers rely heavily on daily stand-ups and 1:1 A manager’s time is zero-sum
and should be applied for
check-ins to understand the progress of their engineers. The portion of
maximum impact. When
this verbal interaction that is focused on diagnosing risk and getting reviewing code commits, data
status on progress can be reallocated to productive coding time if the cuts through the noise and
team manager instead uses a data-driven dashboard that shows commit signals which work carries an
elevated risk pro!le to be
progress and PR reviews as they happen.
prioritized for additional review.

While viewing commits in a data dashboard may not sound sexy, it is an

easy way to eliminate 1 to 3 hours of unnecessary meeting time each week
for each team member, which can result in roughly a 5% increase in
productive time.

© 2018 GITPRIME, INC. 5

INSIGHT 03

Code Churn
Code churn, or code rework, is not a bad thing. Testing and rework are
natural parts of the software development process. However, code churn
levels that deviate significantly higher or lower than expected norms can
represent smoke that is an indicator of a potential fire.

In benchmarking the coding behavior of over 85,000 software engineers*,

GitPrime found that code churn levels most frequently run between
13-30% of all code committed (i.e. 70-87% Efficiency), where a typical
team can expect to operate in the neighborhood of 75% Efficiency. *The Fundamentals Metrics, a
2017 GitPrime data science study
of 7M commits, across 1.8M
Since a baseline level of Code Churn is always expected, only when active days, from 87,000 authors
Efficiency moves materially above or below 75% should there be cause which established statistical
reason for concern — something is likely amiss. A churn level of less than evidence for industry
benchmarks and GitPrime’s
10% would indicate that an engineer is potentially sacrificing speed for
Fundamentals and its associated
precision; a churn level of over 25% would suggest that an engineer may Leadership Playbook. More at
be stuck, or is working on a project where they need assistance. http://gitpri.me/benchmarks

By baselining the ‘natural’ churn levels of typical types of projects, Over the past six months, this
engineering team set a target of
engineering managers can actively monitor churn by engineer, or by
75% e!ciency and is now
project, to identify areas where their team may be hung up and need working within industry norms.
assistance. A particularly high (or low) churn level could be an early
indicator that a project is not progressing as planned.

© 2018 GITPRIME, INC. 6

INSIGHT 04

Legacy Refactor vs New Work

A common question that is posed to both the CFO and CTO is ‘How
much of our software engineering investment is spent on new work,
versus supporting or refactoring legacy code?’

This is a metric that can be easily quantified by analyzing code at the time
of commit, creating hard data on how much of a team’s productivity is
dedicated to new projects versus to technical debt.

It is now possible to quantify the percentage of work delivered that At the beginning of the period,
this team focused on new and
relates to legacy refactoring, down to the line level. A 100-person
exploratory work as indicated by
engineering team that spends 30% of its time on legacy refactor is the New Work (green) and
spending over $3 Million a year working on older code. prominent Code Churn (red). In
mid-November, they
Incorporating this data into the standard KPI set of a software transitioned to paying down
technical debt with an emphasis
engineering organization should be a no-brainer, once the values are on Legacy Refactoring (orange).
accessed and aggregated from the source code repositories.

© 2018 GITPRIME, INC. 7

INSIGHT 05

Collaboration Trends in Code Reviews

Rating Pull Requests by their underlying code complexity, and
quantifying the number of reviewers and comments, is a simple way to
establish metrics for the code review process.

This data is accessible in the code repository and provides the engineering Data allows managers to identify
collaboration trends like how
manager with a clear input into the collaboration process that is taking
long Pull Request are staying
place around code reviews, providing another example of how simple open, if review protocols are
quantification of engineering behavior can make a manager more being followed, or when an
effective. unusual amount of PRs are
being rejected (closed).

When too much time is spent in code review, it can be an organizational

drain; if not enough review occurs on high complexity PRs, it can put the
broader code base at risk. By using data, managers can optimize the time
spent on code reviews while also decreasing downstream risk to the
codebase.

In addition to better managing pull requests that are in process, managers

can do systematic review of past PRs to identify healthy, and risky,
collaboration patterns.

© 2018 GITPRIME, INC. 8

WHAT’S NEXT

Will Software Engineering Embrace Metrics?

We are at an exciting threshold to the software engineering world entering the
realm of data-driven management. Some key points to understand about
adopting data-driven metrics within software engineering organizations:

Data does not replace management

Data is not a substitute for management - it is a tool that makes management
better. By using data, managers can better understand risk, identify
bottlenecks and replace low-value meetings with analysis of trends in the
codebase.

Data must be used for good

The cultural approach taken to embracing and utilizing metrics is critical in
determining the effectiveness of a data-driven management approach.
Everyone involved must agree that the purpose of using metrics is to make the
whole team better. The emphasis needs to be on team improvement, and on
learning so that everyone can get better and create a better work product.

If it is perceived that metrics will be used punitively, arbitrarily or without

context, then it is possible to create a defensive or destructive culture. From
the top-down, the emphasis should be on improvement, growth and increase
self-awareness so that the whole organization can evolve.

LEARN MORE FEEDBACK

By analyzing over 7 million commits from over We also welcome your feedback on how data-
85,000 professional software engineers, we have driven metrics will change the landscape of
learned that utilizing data can be extremely software engineering management over the
valuable to software engineering organizations. coming decade - please reach out.
Learn more about our thoughts on how metrics
can drive engineering productivity at: GitPrime, Inc.
https://www.gitprime.com
http://blog.gitprime.com/productivity research@gitprime.com
650.446.7979

© 2018 GITPRIME, INC. 9