Incident review — what makes a useful postmortem

An incident that is not reviewed is an incident that will happen again. An incident that is reviewed poorly wastes everyone’s time and produces no improvement.

The single most tedious part of writing one is the timeline, and it is also the part most likely to be wrong, because it is usually reconstructed from memory and Slack scrollback hours after the fact. On a Docker host, most of it can be generated.

Blameless is the point

The reviewer looks at the system, not the person. The question is “what conditions allowed this failure?”, not “who clicked the wrong button”. People operating under time pressure make rational choices given the information they had; the system either surfaced the right information or it didn’t.

A blameless review produces durable improvements:

• Documentation that was wrong or missing.
• Alerting that fired too late or too noisily.
• Runbooks that were out of date.
• Code that had no tests, or tests that didn’t catch the bug.
• Approvals that required heroic effort.
• Configurations that no one understood.

Generate the timeline

#!/usr/bin/env bash
set -uo pipefail
FROM='2026-08-12 02:00:00'
TO='2026-08-12 03:30:00'
OUT=/var/tmp/timeline.txt

{
# Container lifecycle transitions, with exit codes in the attributes
docker events --since "$FROM" --until "$TO" \
  --format '{{.Time}} DOCKER {{.Type}}/{{.Action}} {{.Actor.Attributes.name}} exit={{.Actor.Attributes.exitCode}}'

# Daemon and containerd
journalctl -u docker.service --since "$FROM" --until "$TO" \
  -o short-unix --no-pager | sed 's/^/DOCKERD /'
journalctl -u containerd.service --since "$FROM" --until "$TO" \
  -o short-unix --no-pager | sed 's/^/CONTAINERD /'

# Kernel: OOM kills, filesystem errors, link state
journalctl -k --since "$FROM" --until "$TO" \
  -o short-unix --no-pager | sed 's/^/KERNEL /'
} | sort -n > "$OUT"

wc -l "$OUT"
head -40 "$OUT"

docker events gives Unix timestamps by default with {{.Time}}, and journalctl -o short-unix prints the same format, which is what makes a plain sort -n merge them correctly. Fighting timestamp formats is otherwise most of the work.

Illustrative merged output:

1755050047 DOCKER container/health_status checkout exit=
1755050049 KERNEL Memory cgroup out of memory: Killed process 14822 (node)
1755050049 DOCKER container/oom checkout exit=
1755050049 DOCKER container/die checkout exit=137
1755050050 DOCKER container/start checkout exit=
1755050112 DOCKER container/die checkout exit=137
1755050113 DOCKER container/start checkout exit=

That sequence is a complete narrative and nobody had to remember any of it: the healthcheck went unhealthy two seconds before the kernel’s OOM killer fired, the container died with 137, the restart policy brought it back, and it repeated on a sixty-second cycle. The oom event immediately before the die is the evidence that closes the question of whether it was memory.

Structure of a useful postmortem

1. Summary. Two sentences: what broke and what the user impact was.
2. Timeline. UTC timestamps with: detection, escalation, mitigation, resolution, all-hands-clear. Use one-minute granularity. Generate it rather than recalling it.
3. Impact. Who was affected, for how long, with what symptoms. Be specific.
4. Root cause. The technical reason the failure manifested. Avoid "human error"; name the system gap.
5. Contributing factors. What made the failure worse or harder to detect. These are usually the most valuable list.
6. What went well. Detection that worked, runbooks that helped, escalations that were timely. Reinforce what is working.
7. Action items. With owners, dates, and severity. Without these the postmortem is just narrative.
8. Lessons. One paragraph: what is the durable takeaway that should outlive the specific incident?

Root cause, at the right depth

The failure mode of the root-cause section is stopping too early or going too far. Both produce a review that generates no useful action.

Take the OOM example above and ask “why” repeatedly:

Depths 3 and 4 are where the useful action items live. Depth 4 is the one that prevents the class rather than the instance, and it is the one most reviews never reach because depth 2 feels like an answer.

Docker-specific contributing factors worth checking every time

Most Docker incidents have one or more of the same handful of amplifiers. Running this checklist during the review surfaces them without anyone having to be clever.

#!/usr/bin/env bash
set -uo pipefail

echo '--- 1. Was log rotation configured? Unbounded logs fill disks ---'
docker info --format 'daemon log-driver: {{.LoggingDriver}}'
docker ps -q | while read -r c; do
printf '%s max-size=%s\n' \
  "$(docker inspect "$c" --format '{{.Name}}')" \
  "$(docker inspect "$c" --format '{{index .HostConfig.LogConfig.Config "max-size"}}')"
done

echo '--- 2. Would a daemon restart have been survivable? ---'
docker info --format 'live-restore: {{.LiveRestoreEnabled}}'

echo '--- 3. Do containers have memory limits, or can one take the host? ---'
docker ps -q | xargs -r docker inspect \
--format '{{.Name}} mem={{.HostConfig.Memory}} cpus={{.HostConfig.NanoCpus}}'

echo '--- 4. Do they have healthchecks, and is anything acting on them? ---'
docker ps -q | xargs -r docker inspect \
--format '{{.Name}} health={{if .State.Health}}{{.State.Health.Status}}{{else}}NONE{{end}}'

echo '--- 5. Restart policies: will a crash loop be visible or silent? ---'
docker ps -aq | xargs -r docker inspect \
--format '{{.Name}} policy={{.HostConfig.RestartPolicy.Name}} restarts={{.RestartCount}}'

echo '--- 6. Was the previous release image still local for a rollback? ---'
docker image ls --format 'table {{.Repository}}\t{{.Tag}}\t{{.CreatedSince}}' | head -20

echo '--- 7. Headroom on both axes ---'
df -h --output=pcent,target | sort -rn | head -3
df -i --output=ipcent,target | sort -rn | head -3

Item 5 deserves particular attention in a review. A container with restart: always and a RestartCount in the hundreds has been failing continuously, possibly for weeks, and nobody noticed because the restart policy kept papering over it. The restart policy converted a loud failure into a silent one. That is a contributing factor and often a genuine root cause of why detection was slow, distinct from the root cause of the failure itself.

Common mistakes

• Naming an individual in the root cause. Replace “Alice ran the bad command” with “the runbook did not warn against this command under these conditions”.
• Action items with no owner. “We should improve monitoring” is not an action item. “Bob to add an alert on X by 2026-09-15” is.
• Skipping the “what went well” section. It is the cheapest way to reinforce good behaviour and shows the team what is working.
• Filing the postmortem and never re-reading it. The action items are the artefact; track them.
• Stopping at depth 2. “The limit was too low” invites “raise the limit” and prevents nothing. Ask why the limit was what it was, and why nothing detected the drift.
• Recording only the correct hypothesis. The wrong ones, with the commands that disproved them, are what save the next responder twenty minutes.

Knowledge check