How to build metrics that work
The role of metrics in Agile development
How we measure and report is a key part of ensuring a consistent language and understanding around the organisation which is paramount to success. We can view the value stream as a set of behaviours or activities which we perform. These activities result in outcomes which represent value to the business.
The challenge is that the environment is complex. In a complex situation, the linkage between behaviours and outcomes is hard to assess. Introducing metrics as an intermediate layer allows more control. We can design experiments to change behaviour and use metrics to measure the results allowing us more rapidly understand the potential impact on organisational outcomes.
Capabilities-> Metrics -> Outcomes
Capabilities predict metrics, meaning that different activities or behaviours are the drivers for the numbers in the metrics. We cannot change the numbers directly; only by changing the behaviours can we influence the metrics. We could also say that the metrics are a lagging indicator for the capabilities. They tell us what has already changed in the capabilities.
In the same way metrics (if well chosen) predict outcomes, or are a leading indicator for outcomes. Since outcomes are what we really care about, this is why they are valuable to us. We aim to measure data which predicts valuable outcomes for the business. Increases in these metrics then predict increases in the outcomes (and thus value) for the business.
Designing successful KPIs
We can (and should) measure many different parameters to understand how the organisation functions and to watch for potential problems. Well known measures such as the DORA Software Metrics can add significant value. However, we choose some key measures which we communicate widely in the organisation. These we consider Key Performance Indicators (KPIs) and they become part of the organizational language. These are the measures that you choose to publish and track visibly.
Your key measures should be considered carefully. Your KPIs will be sending a message to the wider organization. And you do not want to change them too often. That would dilute the understanding and so the message in the reporting. Successful KPIs need to have four characteristics:
Relevant
Any KPI should be a leading indicator of business value. If the KPI increases, this should link to an improved outcome and an increase in value.
Generally the KPI does not directly measure value but for the metric to be relevant, there must be a strong correlation that an increase in the metric predicts an improved outcome. The strong linkage of the metric to the successful performance of the group is key to its relevance.
You should be clear about the outcome which we hope to see. If so, people will focus less on the mechanics of the measure and more on improving the underlying value.
Controllable
A KPI is a leading indicator of business value, and it must also be a lagging measure of specific activities or behaviours.
For the KPI to be controllable, there must be a set of behaviours or activities which can be modified which will modify the KPI.
The environment is complex, so the exact linkage between activities and metric will be unclear. However, experiments can be proposed which change behaviours in order to influence the metric.
Measurable
The metric must be measurable. That sounds obvious, but the way the metric is defined and the way the data is collected must be clear and public.
The collection of data should be as simple as possible. Too often each new metric adds a cumulative workload on the team.
You should be clear with the team what is being measured. Secret measures are likely to spread suspicion. You also need to be clear how it is measured so people understand what the numbers mean.
Robust
Like Heisenberg’s Uncertainty Principle, once we observe the data we affect it. The act of formalising and publishing a measure will have impact. In particular, teams will aim to increase the measure. Sometimes people refer to this as “gaming” measures. That seems unfair. They are typically just doing what they are asked.
The test of a “robust” measure is that when we seek to increase the measure, we are generally promoting a beneficial outcome. The robustness of measures can be improved by a Balanced Scorecard approach.
Robustness of Velocity
Many organizations successfully adopt Velocity as an effective prediction tool. By measuring past performance against estimates, we predict future performance against estimates.
However, some organisations then try and use Velocity as a measure of productivity. They reason that, because we already measure how fast we deliver work, we can use this to compare teams or ensure that we deliver more work simply by ensuring the Velocity metric increases.
A robust measure of productivity would indeed be very valuable. It is probably the top measure which I am requested to create at any of the companies where I have worked. Velocity seems very tempting as a measure. Velocity measures the amount of work delivered over time. It therefore sounds initially like a productivity measure.
But there is no absolute measure of sizing. To assess velocity, the team create an arbitrary sizing scale (for example using Story Points). For the purpose of tracking and predicting this is just what we need. We want to know how fast we will deliver based on how fast we have delivered. We are comparing like with like.
Let’s apply the test of robustness. Imagine we measure Velocity and encourage teams to increase the score. Does that lead to a good outcome? Does it necessarily mean they are becoming more productive?
For productivity however, an arbitrary scale isn’t useful. If a team is asked to “increase velocity” they can easily do so. Increasing velocity is as simple as halving the value of their measurement units, so doubling the number of units that they deliver. The more meaningful “increase productivity” is much harder to achieve.
Using metrics to improve
The effectiveness of metrics links closely to how the organisation uses those metrics and the resulting data. Measures should be used, often as part of retrospectives, to drive continuous improvement and learning.
Remember that we have a complex linkage between activities or behaviours and metrics, and a further linkage between metrics and value outcomes.
We collect data to identify areas where performance may not be as good as we hope. This is because these low values will likely lead to poor value outcomes at a later point.
We then develop experiments for which activities we might modify. We cannot be sure of the exact linkage but we can hypothesise that changing one behaviour would impact the metric.
We then run the experiment and observe any change in value. The point of the metrics is that this will be far faster than looking for a change in outcomes. And because the linkage is tighter, there will be less other factors coming into play.
If our experiment successfully improves the data, we adopt the new practices and we should eventually see the impact on value outcomes.
The ultimate purpose of taking data is to provide a basis for action or a recommendation for action
W.E.Deming
Metrics and management style
I was once discussing productivity measures with a senior manager. He wanted the team to implement velocity as a productivity measure. He then wanted to make sure that it continually increased. I pointed out that the team would naturally respond by changing the scaling. His immediate response was that we should punish the teams for “cheating”.
I have concerns that “punish the teams” should ever be the immediate reaction from any manager. This feels a Scientific Management approach, not an Agile one. Leaving that aside, it is not reasonable to see this as dishonesty on the part of the team.
If we ask a team to increase a metric and measure them on doing so, they will ensure it increases. A poorly designed metric will increase but not give the desired result. We have focussed only the metric and not the desired outcome. It is a failure of management, not of the team.
Tell me how you measure me and I will tell you how I will behave
Goldratt
Good Practices
As a leader you should constantly be collecting and studying data. Especially in a fast changing organisation you should be looking for what is going well and what isn’t working.
Design your metrics to be tightly linked to behaviours so you can see what activities would affect the metrics, and also linked to outcomes so you can see what value the metrics would affect.
Be constantly alert for the next area of attention. Look for the opportunity to work with the teams to run experiments on changed behaviours that might improve underperforming metrics.
When thinking about which of these measurements you will promote to Key Performance Indicators and make visible across the organisation, think carefully. You will want a small number of these, ideally not changing over time, and easily understood, collected into a balanced “Engineering Scorecard”
Leave a Reply