The incident record - writing it while it is happening

Three different documents come out of an incident and they are routinely confused, which is why all three end up mediocre:

This lesson is about the first one, because it is the only one that cannot be produced later. The review depends on it, the status updates are drawn from it, and both degrade badly when it was not kept.

Incident roles and command are covered in linux-incident-roles-and-communication; causal analysis is covered in the root cause analysis part. Here the subject is the artefact.

Why it cannot be reconstructed

Human recall of a stressful two-hour period is poor within hours and unreliable within a day. Specifically, it degrades in ways that matter:

• Ordering collapses. People remember that two things happened and not which came first, which is exactly the information a causal analysis needs.
• Durations compress. “We restarted it a few times” was four restarts over 40 minutes.
• Negative results vanish entirely. Nobody remembers the three things they checked that were fine, so the review re-derives them and the next responder re-checks them.
• Reasoning is replaced by outcome. Once you know the cause, you remember having suspected it, and the record of what you actually believed at 02:40 - which is what would tell you why the diagnosis took ninety minutes - is gone.

The last one is the most damaging, and it is invisible. A review built from memory produces a tidy narrative in which everyone behaved sensibly and the cause was found as fast as possible. That narrative has no improvements in it.

The timeline

One line per event, absolute UTC timestamps, and a marker for the kind of entry:

02:14Z OBS  Alert: myapp health check failing on app02, app03. app01 healthy.
02:16Z OBS  app02: systemctl is-active myapp -> failed. Journal shows
            "cannot allocate memory" repeated from 02:11Z.
02:18Z OBS  free -h on app02: 15Gi total, 210Mi available. No swap.
02:19Z DEC  Not restarting yet - capturing state first, because a restart
            clears the process and we lose the allocation picture.
02:21Z ACT  Captured ps auxf, ss -tanp, /proc/meminfo to /var/tmp/INC-0811-01.
02:24Z OBS  Checked storage: df -h and df -i both fine on all three. NOT the cause.
02:27Z OBS  app01 differs: it was rebooted on 09 Aug, the others on 21 Jul.
02:31Z DEC  Hypothesis: memory leak in myapp, ~3 weeks to exhaustion.
            app01 was reset by its reboot. Testing by restarting app02 only.
02:33Z ACT  systemctl restart myapp on app02.
02:35Z OBS  app02 healthy. Memory 12Gi available. Consistent with hypothesis.
02:36Z ACT  Restarted app03. Service restored, both nodes healthy.
02:38Z ACT  Status update sent: service restored 02:36Z, monitoring.

Three markers, and the distinction between them is what makes the record usable:

• OBS - something observed. A fact, with the command or the alert that produced it.
• ACT - something changed. Every action taken on the system, so the review can correlate a change with what happened next.
• DEC - a decision, with the reasoning at the time.

UTC, always. An incident spanning a daylight-saving change, or involving responders in two time zones, produces a timeline nobody can order otherwise. Note the Z suffix explicitly so a reader does not have to infer it.

Capture evidence before you recover

Recovery destroys state. Restarting the process, rebooting the host, failing over, clearing the disk - each is the right thing to do for the outage and each removes the evidence of what caused it.

The conflict is real and it resolves in one direction: restore service first when the impact is severe. But a capture takes under a minute, and knowing that lets you do both.

INC=INC-2026-0811-01
sudo install -d -m 0700 "/var/tmp/$INC"
cd "/var/tmp/$INC"

ps auxf                       | sudo tee processes.txt   >/dev/null
sudo ss -tanp                 | sudo tee sockets.txt     >/dev/null
{ free -h; cat /proc/meminfo; } | sudo tee meminfo.txt   >/dev/null
{ df -h; df -i; }             | sudo tee filesystems.txt >/dev/null
sudo dmesg -T                 | sudo tee dmesg.txt       >/dev/null
sudo journalctl -u myapp --since '-2h' --no-pager | sudo tee myapp.log >/dev/null

dmesg requires root on most modern distributions, because kernel.dmesg_restrict defaults to 1; check with sysctl kernel.dmesg_restrict if it fails for an unprivileged user.

For a host you may need vendor support on, collect a full diagnostic bundle while the system is still in the failed state:

# sos report --batch --case-id INC-2026-0811-01

sosreport (version 4.7.2)
Setting up archive ...
Setting up plugins ...
Running plugins. Please wait ...
Finished running plugins
Creating compressed archive...
Your sosreport has been generated and saved in:
/var/tmp/sosreport-app02-INC-2026-0811-01-2026-08-11-xkqvbnt.tar.xz

--batch stops it prompting, which matters when you are running it under time pressure, and --case-id stamps the archive so it can be matched to the incident later.

Detection and impact, recorded honestly

Two fields decide most of the improvement work that comes out of an incident, and both are easy to write in a way that is technically true and useless.

Detection. How did you find out? The options, roughly in descending order of how well the system is working:

1. An alert fired, and it named the right thing.
2. An alert fired, but on a symptom two layers away.
3. An engineer noticed something while doing other work.
4. A customer told you.
5. It was found during an unrelated investigation days later.

Record which one it was, plainly. “Detected by monitoring” when the truth is that a customer opened a ticket 40 minutes before the alert fired removes the single most actionable finding the incident produced. Also record the gap: the time between the first user impact and the first human being aware, which is usually much longer than anyone assumes.

Impact. In user terms, with numbers and a bounded period:

IMPACT
  From    : 02:11Z (first failed request)
  To      : 02:36Z (last failed request)
  Who     : all users of the checkout API, ~4,100 requests
  What    : HTTP 503; retries also failed; no data was lost
  Not hit : the reporting API and the batch pipeline

The “not hit” line prevents the review from over-scoping, and it is a genuine finding in its own right: it tells you the blast radius the architecture actually produced, which may be narrower or wider than the design intended.

Follow-up actions

An incident record ends with a list of follow-ups. Every one needs an owning team and a date, and every one needs to be in the same tracker as ordinary work.

Actions living only in the incident document do not happen. There is no mechanism by which they would: nobody has that document open, it is not in anyone’s queue, and the incident is closed. Two months later the same incident recurs and the review discovers its own recommendations from last time, not implemented, in a document nobody reopened.

Record the tracker reference in the incident record and the incident reference in the tracker item, so either one leads to the other.

Language: systems, not people

Write what the system permitted, not who typed the command.

• Not: “Sam deployed the config without running the validator.”
• Instead: “The change was deployed without validation. The deployment pipeline does not run the validator, and the validator is a separate manual step documented in the build procedure.”

The second version is longer and contains the improvement. The first version contains a person and ends the investigation, because “someone made a mistake” is a conclusion rather than a finding - and it teaches everyone watching that the safe move during the next incident is to share less.

Use roles - “the on-call engineer”, “the change author” - in the record. The names are in the chat log and the ticket history if an investigation genuinely needs them.

Knowledge check