Complexity – are checklists an Agile tool?

In scaling quality in an organization my own Playbook always includes checklists as a key component.  I first experienced these used well twenty years ago and have found them effective ever since.

Checklists are not typically an area that we see associated with Agile development.  Checklists may raise a spectre of mechanistic, process-driven work which is at odds with the complex, Agile work environment. However, I would suggest many people have not seen checklists working effectively. 

It’s worth stepping back and looking at what I mean by checklists and why they can be very powerful tools.

Where do checklists come from?

In the 1930s Boeing bid for a contract to build a long-range bombers for the US Army Air Corps.  Their design, the Model 299, could fly further and faster than previous bombers, and carry far more bombs than had been originally specified.

The final test flight was held on October 30^th 1935.  It was little more than a formality before the contract was confirmed.  In the test, the aircraft made a normal take-off but then stalled and fell, bursting into flames. Boeing lost the contract. 

The investigation found “pilot error” as the cause of the crash. The test pilot had been the U.S. Army Air Corps’ Chief of Flight Testing.  But the aircraft was complex.  The newspapers summed up the Model 299 as

too much airplane for one man to fly

Rather than rely on pilots remembering the key activities, Boeing introduced a checklist to ensure that everything was done correctly in the right sequence.  

With checklists and rigorous training, the test aircraft flew 1.8 million miles without a serious accident.  The U.S. Army eventually ordered over 12 thousand of the aircraft which became a key bomber of the Second World War.

What do checklists do?

There’s a key message in the example above.  The issues with the Model 299 were not about skills and training.  A checklist will not increase a team member’s skill and allow him or her to create higher performing outputs.  The checklist didn’t make the plane fly further or faster.  A checklist will not manage teams, communicate with stakeholders or make decisions.  These are all critical areas that individuals must perform.

How often have you been distracted and missed something?  Maybe when it was pointed out it was something so obvious that you couldn’t believe you had missed it.  Whenever you focus intently on a high skilled task, it’s easy to make mistakes, especially in simple work.  Like the aircraft, these may be trivial in nature but high in impact.

Development is a great example of this sort of high skilled work.  As a complex activity using significant expert skill, developers can easily make relatively mistakes.  A checklist is a great supporting tool for sustainable pace working.

Checklists are a tool to prevent stressed experts from making basic errors

A checklist is a set of minimum required steps for successful completion of an activity.  One example of a Checklist is covered in the Play on “Definition of Done”.  The Definition of Done (DoD) is a formal definition of the items which need to be completed to reach complete or “Done” quality.  Like any good checklist it doesn’t give a detailed list of specific activities.  The DoD doesn’t for example include the actual tests for a specific task.  What it does is ensure that no part of what is needed (such as the testing) is forgotten.

How is a checklist constructed?

Each checklist contains a set of questions.  These should be pre-defined for the specific situation.  The value of the checklist is to ensure that everything for a busy or crisis situation has been considered.  This can best be ensured if the questions are written at a quiet time, not in the middle of the crisis.  The team then work through the questions.  I use a model of teams giving one of three answers – Yes, No or N/A (not applicable).

The primary function of a checklist is as a review point to ensure that nothing has been omitted.  When the team reaches the relevant point (for example they intend to mark something as “Done”) they go through the questions.  A typical Definition of Done question might be as below:

• Has all internal documentation been updated?
  • “Yes” – Document updates are completed (the status is as we would normally expect )
  • “No” – Document updates are not yet done (the item is incomplete when we would expect this to be done at this point )
  • “N/A” – No associated documentation needed (the work is not required).

The team assesses the answer.  We design each question so the expected answer question is “Yes”.  In some cases the answer proves to be “N/A” because the checklist doesn’t exactly match the circumstances. Here, some work may not require documentation updates.  However, if it is “No” the team have work to complete in this area.  In many cases this is a simple omission.  They will typically complete the missing work, return to the checklist and be able to mark it as “Yes”. In other cases it may indicate a blocking item preventing progress.

Why have a checklist?

Do/Review checklists

The more obvious usage is looking backwards. The checklist functions as a review point for checking the expected outcomes. 

This is known as a “Do/Review” approach.  The team does the work and then looks at the questions to review what has been done.  The questions tell them whether they have covered everything they should have done.  The checklist structures a review. 

This approach is effective but has a limitation.  If something has been missed, it may involve some rework or at least an unexpected delay.  The way teams work may not match the checklist.  We then have a constant resetting when the review is done.  This leads to repeated delay and inefficiency.

Check/Do checklists

The second use of a checklist is looking forwards – a “Check/Do” approach. 

Here the team review the checklist in advance of doing the work.  This allows them to make sure they are clear on everything which they need to do.  They then perform the task. 

Finally they do the review, again checking the questions.  But this final review should be trivial as they are already familiar with the questions.

There is an important learning from anyone who has worked with checklists.  You need to use both approaches.  This is similar to testing. Tests are designed based on understanding the requirements. They are then implemented to verify the code.

The checklist is an important review point to be used after the work to ensure that nothing is missed.  However it is also an important source of guidance.  The team need to be aware of the questions to shape their plans before they start. 

Knowledge Base

In many ways, the checklist functions as a knowledge base.  It is both advising the team what to be considering (“Check/Do”) and ensuring they have done so (“Do/Review”).  Treating a checklist as a knowledge base adds even more value to this effective approach.

A key learning from teams using checklists well is that the checklist questions should not be static.  Every organization grows and changes.  As with any process, there needs to be a mechanism for learning. 

If something goes wrong in a development, this will be looked at in a retrospective.  The retrospective should identify what the root cause of the issue was.  We then need a mechanism to prevent this recurring. 

Checklists work well for this function.  You can add in a new question to the checklist to ensure the team consider this problem area.  By having the team check beforehand and review after, the problem is unlikely to recur.

For example, imagine that a team is rushing and closing work items as “done” before the Continuous Integration pipeline has reported all tests passing. Adding in a question to the “Definition of Done” to say “has the pipeline completed and been marked as green” could help with this.

A valuable learning from experience is that you want to avoid the number of questions growing. This specific one could probably be retired from the checklist when everyone is doing this check automatically. In general it is good to avoid having many more than ten questions in a checklist. Too many questions and people will click through them without really thinking “have I done this” and the quality value will be lost.

Checklists can be a great knowledge base

Good practices

As an Agile leader, you should consider how to introduce a checklist concept. A good starting point is often “Definition of Done”. Expect some level of resistance and misunderstanding. You may find you need to coach teams on the value. Atul Gawande’s book “The Checklist Manifesto” is a good starting point. I once bought it for all of my team and it was a good investment.

You will need to think about how to implement checklists. There is little “off the shelf” tooling available for checklists. You should be looking for a tool which will allow you to maintain the questions as a template, and then instantiate specific lists for the team to check off, based on that template. In the past I have used Confluence, Jira, Excel and custom tools to do this.

Once checklists have started to be used, you want to build a learning mechanism. How can retrospectives flag up issues which are considered for inclusion in checklists? Who owns the question list for each checklist? What is the threshold for addition of a new question? Also importantly, what is the threshold for removal of a question if it is never answered “No”?