The blameless postmortem - writing an RCA that produces change

Everything in this part of the course produces material: a decomposition, a timeline, a causal tree, a set of systemic findings. The document is where that material either becomes work or becomes a file nobody opens again.

Most post-incident reviews are written to be filed. They are long, defensive, chronological, and end with a list of vague intentions. A review written to be used looks different, and the differences are specific enough to be checked.

The structure

Nine sections. Order matters, because it puts the answers before the narrative.

1. Summary: three or four sentences a director can read - what broke, who was affected, for how long, and what is being done
2. Impact: measured, from first user-visible symptom to recovery, in users, requests, transactions or revenue as appropriate
3. Timeline: system and response columns, every line sourced, in UTC
4. Trigger: the single event that started it, with a timestamp
5. Contributing factors: the conditions that turned the trigger into an outage, each labelled supported or plausible
6. Root cause: one or two, stated as a gap or decision rather than as an event
7. What went well: the controls that worked, so they do not get removed later by someone economising
8. Action items: owner, due date, concrete change, verification
9. Constraints and open questions: what you could not fix and what you could not determine

Two sections earn their place against the instinct to cut them.

What went well is not morale management. It records which controls limited the damage - the circuit breaker that contained the failure, the alert that did fire, the runbook that was correct. Those are the things a future cost-cutting exercise will propose removing, and this is the only place their value is written down.

Open questions is what stops a review being fiction. If you never established why the retry storm started, say so. An unanswered question, written down, is worth more than a confident guess, and it is what lets a later reviewer pick the thread up rather than inheriting a story that quietly closed over it.

Impact must be measured

“Some users were affected for a while” is not an impact statement. The impact section is what determines how much attention the incident gets and how much time is allocated to the actions, so vagueness here quietly caps everything that follows.

IMPACT
  Duration (user-visible):  09:14:02 - 09:47:52 UTC, 33 m 50 s
  Requests failed:          412,000 of 1,190,000 in the window (34.6%)
  Customers affected:       all API consumers; 214 distinct accounts
                            observed at least one 5xx
  Data loss:                none; writes were rejected, not lost
  Revenue impact:           not estimated (see open questions)
  SLO burn:                 consumed 71% of the monthly error budget

Measure from the first user-visible symptom, not from the first alert. Those differed by seventeen minutes in this incident, and measuring from the alert would have understated the outage by half - which is precisely the direction the measurement drifts if nobody insists.

Where a number is unknown, write “not estimated” and put it in open questions. Do not omit the line; an omitted line reads as zero.

Blameless in practice

“Blameless” is widely agreed to and inconsistently practised, because the hard part is not intent - it is the writing.

The rule that works is mechanical: describe actions and system responses, not judgements of the actor.

Every right-hand entry is more specific, more useful, and happens to be blameless. That is not a coincidence: blame is a low-information summary, and replacing it with the actual mechanism is what produces an actionable finding.

Three further habits matter as much as the wording.

Use roles, not names, in the document. “The operator”, “the on-call”. Names belong in the meeting, where they are useful for asking questions, not in a document that will be read by people who do not know the context.

Do not include disciplinary implications, and do not let the document feed a performance process. The moment a review can affect somebody’s rating, reviews become defensive and the evidence stops arriving. This is a management commitment, and without it the rest is theatre.

Let people describe their own reasoning. “The operator believed the timer would be re-enabled by the configuration management run” is a finding about a reasonable belief the system failed to correct. It is also information nobody else could have supplied, and it only arrives if giving it is safe.

The meeting is not the document

They do different jobs and running only one of them fails differently.

The meeting surfaces information the document cannot: what people were thinking, what they tried and discarded, what they found confusing, which parts of the timeline are wrong. Hold it within a few days, while recall is still usable, and have the draft timeline in front of everyone so corrections are anchored to specific lines.

The document is the durable artefact. It is read by people who were not there, months later, usually because something similar has happened.

A meeting without a document produces shared understanding that evaporates with the next reorganisation. A document without a meeting misses everything that was never logged, which is most of the response.

Action items decide everything

Everything before this section is analysis. This is the part that changes the system, and its quality is measurable.

Each action needs an individual owner, a due date, a concrete named change, and a verification - the four-part test from the preventing-recurrence lesson. Beyond that, three properties of the list matter.

Keep it short. Five well-chosen actions get done. Twenty produce a burndown chart and three completions. Prioritise by the causal tree: the actions attacking general, removable factors first.

Track them where other work is tracked. Actions living in the review document are actions nobody will see again. They go into the same backlog, with the same visibility and the same prioritisation contest, or they do not exist.

Measure completion. Report the fraction of actions completed by their due date, per quarter. This single number tells you whether the review process is producing change or producing documents - and it is the number that turns “we keep having the same incidents” into a diagnosable problem rather than a complaint.

Knowledge check