Preventing recurrence - closing the failure gap and the detection gap

Step 10 is the step that pays for the other nine. Everything up to step 8 restored one outage. Step 10 is what stops it being an outage you have every quarter.

The measure of a good follow-up is not how thoughtful it sounds in the review. It is whether, if the exact same trigger fired tomorrow, something different would happen.

Two gaps, not one

Every incident has two independent failures, and teams routinely fix only the first.

• Failure gap: why the system broke. The connection pool was sized for the old upstream configuration and nothing tied the two together.
• Detection gap: why nobody knew until a user complained. There was no alert on pool saturation, so the first signal was a support ticket forty minutes in.

Close only the failure gap and the next, different incident is still discovered by a customer. Close only the detection gap and you find out faster about a problem you still have. Both need an action.

Choose the strongest control available

Rank candidate actions by how much they depend on a human doing the right thing. Prefer the top of the list.

1. Eliminate: make the failure impossible - remove the option, derive the value instead of duplicating it
2. Constrain: make the failure hard - validation, resource limits, quotas, safe defaults
3. Limit blast radius: canary one node, stage the rollout, cap the resource, isolate the tenant
4. Detect: alert on the earliest real signal, and alert on the guard rail itself
5. Recover faster: a runbook, an automated rollback, a tested restore
6. Inform: documentation, training, a note in the checklist

Something at the bottom of that list is a fine addition to something further up. On its own it is a way of writing down that nothing was done.

$ systemctl show myapp.service -p MemoryMax -p MemoryHigh -p TasksMax -p Restart -p RestartSec

MemoryHigh=infinity
MemoryMax=infinity
TasksMax=9830
Restart=no
RestartSec=100ms

$ systemd-analyze verify /etc/systemd/system/myapp.service.d/10-limits.conf; systemctl daemon-reload; systemctl show myapp.service -p MemoryMax

MemoryMax=2147483648

Writing a follow-up that will actually happen

An action passes only if it has all four of these.

• A named owner. A team is not an owner. A person is.
• A due date. Without one it will lose every priority contest it ever enters.
• A concrete change. Name the file, the alert, the unit, or the runbook.
• A verification. State how you will know it works, and prefer a test you can actually run.

Good: “Ada adds a MemoryMax=2G drop-in for myapp.service in the base role by 24 August, verified by systemctl show -p MemoryMax on all nodes and by a load test that hits the ceiling without taking the host down.”

Bad: “The team will monitor memory usage more closely.”

Verify the prevention

A prevention action that has never been exercised is a belief, not a control. Test it the same way you test a restore.

1. Re-create the trigger on a disposable host or in staging, never in production
2. Confirm the guard rail engages: the limit holds, the validation rejects, the canary fails first
3. Confirm the alert fires, reaches the on-call rota, and names the affected component
4. Confirm the runbook step you added is correct by having somebody else follow it verbatim
5. Record the verification date on the action, then close it

Add the same check to the relevant production checklist, so the control is re-verified on a cadence rather than once.

Feed it back into the loop

Prevention closes step 10 by improving earlier steps for the next incident:

• The evidence you wished you had becomes a standard item in the step 4 capture.
• The change you failed to spot becomes an entry in the step 3 change sources.
• The mitigation you improvised at step 8 becomes a runbook.

That is why it is drawn as a loop. Each pass should make the next pass shorter.

Knowledge check