Engineering Scorecard – Value Metrics

Reading time

6 min

A balanced set of metrics will allow us to publish KPIs to the organisation which are both robust and meaningful to the wider organisation. When introducing the Engineering Scorecard I proposed three categories for the Scorecard:

FlowMetrics focus on the flow of work through the engineering system, reducing waste and increasing efficiency.
QualityMetrics assess how well deliverables match their requirements and original intent.
Value Metrics focus on the wider value stream and looking end to end at how customer value is generated.

Why a Value Metric?

Having a single measure is a risk and a balanced scorecard is preferable. FlowMetrics indicate that work is being done and show that the backlog is being processed in an effective manner. When we add in QualityMetrics we ensure that we are delivering work without defects and rework. However, we still have a gap. We may be “doing the work right” but are we “doing the right work”?

For the third part of the Engineering Scorecard I want to look at the value we generate from the Engineering team. One of the first measures I am inevitably asked for in a new organization is a measure of productivity. Any organization would like to reliably measure how much value they are getting from their development team. Viewed positively this would allow the team to build improvements that increase this value.

Productivity measures

For the Flow and Quality metrics there was a set of well-established measures available and we had to choose which of the list to select. For a productivity measure, the situation is less straightforward.

Any productivity or efficiency measure is defined as a measure of output (value) divided by input (cost). We can assess cost fairly reliably. We work with stable teams, bringing the work to the team. We know the cost of maintaining the team in terms of salaries and associated overheads. It seems easy to build this into a productivity measure.

The challenge however lies in measuring the value of what the team creates. Absolute measures of value are rarely simple. Without a reliable measure of value generated, a true productivity measure is unlikely to be achievable

Volume of output

Most measures look at output rather than value and are covered by the FlowMetrics. We can make more and make it more smoothly, but can we demonstrate that we are making more value? Measures such as lines of code are flawed and encourage the teams to create volume. Agile principles are to focus on value not on quantity.

Simplicity–the art of maximizing the amount of work not done–is essential.
Principles behind the Agile Manifesto

Revenue based measures

Revenue-based measures seem promising. If we can assess how much revenue each Engineer generates, is this a valuable productivity measure?

The problem here is that in product development, an engineer doesn’t directly generate revenue. Even his/her team doesn’t. Revenue depends on many parts of the organisation – deployment, sales, product.

As a result, revenue tends to be very loosely linked to engineering activities and very strongly lagging. As a KPI direct revenue offers little value in steering the behaviours and capabilities in engineering teams.

Utilisation

A widely used measure for value is to look at the utilisation of teams. Utilisation is the amount of time that a team is busy. We are paying the cost of the team continually, so the argument is that utilisation gives a measure of how efficiently the team is used.

This measure does not work in an Agile development context. Being busy tells us little about being successful. Also we have seen that backlog works as fuel to ensure that the team always have planned work ready to be executed. Backlog ensures that teams always have full utilisation, which makes this an ineffective KPI.

So what option do we have for a good measure of efficiency which is available to a scaling organization? It seems that measuring productivity or efficiency directly doesn’t seem to be an option. Maybe there’s a different approach we can use.

Measuring waste

Rather than measure efficiency directly, we can take a Lean approach and consider waste instead. Remember that waste reduces value.

We can assess how much of the work is value, and how much is waste. By comparing the two we can get a measure of efficiency.

We can do this without using absolute measures of value. When we looked at planning and tracking we saw that relative estimates are effective when we are considering proportions.

We can use an Earned Value approach as a starting point to assess how much work is being done. Then we can measure that split of work and assess which items are considered value and which are waste.

And as we saw with Earned Value, we can do this with relatively low overhead to the teams.

Strategic Focus

Engineering teams work on roadmap features. However they also necessarily work on many other areas. Defect fixes, customer support, technical debt reduction and so on. We can assume that progressing the roadmap constitutes value, while other work constitute waste.

Remember “waste” is work which is not wanted by the customer. There is no suggestion this is unnecessary or should be removed. It is however true that the more efficient the organization, the less “waste” work it needs to do.

I generally use a metric I call “Strategic Focus”. This represents the fraction of the work which progresses the roadmap and represents the “value” work done by the team. This is relatively easy to communicate and understand.

Case study – Value Metric

For the Value measure we used Strategic Focus – how much of the work is categorised as “value”. To generate this metric you need to be able reliably to measure work and how this is categorised.

Initially, the data was not collected robustly enough for this to be effective. We needed the teams to be tracking work reliably. They also need to be putting estimates against the work to allow different work areas to be compared.

For the first quarter, these data were available for value (feature) work, but were far more patchy for waste (overheads). This biases the results to look like an unrealistic amount of work is value work.

From the second quarter, we were careful to improve logging of non-value work. This included defect fixes and technical debt reduction. This saw the metric decrease and stabilise at a more realistic figure. For this organization that was around 50%. It sounds a low number, but industry benchmarks suggest it is fairly typical.

For the rest of the first year that number was fairly constant. There was a slight increase, which was probably because we improved decision making, giving more clarity that some work was “value” when the value may have been unclear.

In the case study, significant improvement in the Strategic Focus number took more than a year to become visible. We can expect that value metrics will take some time to respond to change. There were several Plays driving the increase:

Reduction in Work in Progress meant the organization became more focussed. Decision making and prioritisation was improved.
The integration of Product Owners meant that there was more active optimisation of backlog. Previously, when individuals had unclear objectives, they would previously often respond with doing lower value work.
Improvement in quality and reduction in defect levels meant less rework was needed. And also less context switching was required to deal with emergencies allowing more focus.

Changes may look small, but a shift from 50% to 65% is significant. This reflects a significant shift in work from “waste” to “value” representing about a 30% increase in engineering productivity.