An Engineering Scorecard as common language
In another Play I look at Key Performance Indicators (KPI s) and the key factors to make these work in an Agile environment. A KPI is a metric we choose to make public, track and manage. For it to be successful we need it to have three characteristics.
- Relevant – it must be a leading indicator of value outcomes for the business, so an increase in the metric predicts benefit for the business.
- Controllable – it must be a lagging indicator of organisational capabilities, so we can build experiments for change and observe the impact on the metrics.
- Measurable – it must be clearly defined and able to be measured without undue cost
- Robust – it is difficult to improving the measure independenly of increasing value outcomes
Designing KPIs for robustness is a challenge. Any single metric is vulnerable to an excessive focus on the metric in isolation leads to negative results. For example, a well intentioned request to increase the number of defects fixed could “right shift” testing, finding defects later (when they count in the metric) rather than taking care to build in quality (which would not count).
Balanced metrics
Although we should take care to make KPIs as robust as possible, it is inevitable that any single measure, however well designed, can be increased without generating benefits. The environment is complex and the linkage with benefits is rarely simple.
We can address this by building and publishing a small set of complementary metrics. Each in isolation might be vulnerable to over-focus to some degree. However, over-focus on improving a single metric will have an effect on the other metrics. As a result the whole set of metrics is far more robust than any single one.
This approach of a balanced set of metrics was introduced at a business level in the 1990s by Kaplan and Norton as the “Balanced Scorecard”. The authors noted the importance of a wide ranging set of measures to avoid excessive focus on financial goals alone.
Their original research proposed four groups of measures at a business level – Financial, Customer, Internal Process, and Learning & Growth. This gave a wide spread to avoid excessive focus in any one area (especially, as they argued, on financial measures).
With a single measure there is no way to visualise the negative consequences. If instead we build a set of complementary measures, we can make this visible.
“Thirty years with the balanced scorecard” Kaplan & Norton, 1992
The idea of balance to increase robustness has continued to be key in metrics. The DORA software metrics, for example, explicitly balance software delivery with reliability to ensure that over-focus on fast flow does not impact quality.
Engineering Scorecard
For our purposes we want to build an “Engineering Scorecard” to give overall visibility to the business. The “Balanced Scorecard” of Kaplan and Norton is set at too wide a corporate focus for this purpose, and applies across the organisation. Conversely the DORA metrics are too low a level and require significant engineering understanding, and best used to drive continuous improvement.
There will be no single right answer for what is on your Engineering Scorecard for your organisation. As with all of the Plays, you will need to build your own Playbook for your specific needs.
The whole set of measures should give visibility of whether the group is functioning effectively. … like the dials in an airplane cockpit: it gives managers complex information at a glance
“Thirty years with the balanced scorecard” Kaplan & Norton, 1992
There are many potential categories for the Engineering Scorecard. I would suggest the three categories below. Initially choose one measure from each category – there is a Play to discuss each one. Start to collect data, ensuring the overhead of data collection is not a drain on the team. Publish the data across the organisation and use this to communicate what Engineering is doing.
Flow metrics
Flow metrics focus on the flow of work through the engineering system, reducing waste and increasing efficiency.
Including a metric based on flow will ensure that we are optimising the output from the engineering effort.
However, focussing on flow alone could lead to a very efficient organisation but one producing large amounts of low value output.
Quality metrics
Quality metrics assess how well deliverables match their requirements and original intent.
Including a metric based on quality will ensure that we are delivering what we are asked to deliver at good quality and without needing expensive rework.
Focussing on quality alone could lead to a very polished product, but at low speed and excessive cost.
Value metrics
Value metrics focus on the wider value stream and looking end to end at how customer value is generated.
Including a metric focussed on value will ensure that we are creating the right outcomes from the work.
However, focussing on value alone can lead to chasing customer requests causing parallelism and quality impact.
Good practices
Keep your Engineering Scorecard simple. The organisation is scaling, so your needs will change. Keep your KPIs focussed to ensure a consistent and meaningful organisational language.
Choose metrics which can be easily understood and are not too deeply technical.
Given the complex environment, combine the small number of public measures with a wide examination of data privately to look for insights that you can share with your team.
Leave a Reply