Systemic causes - the conditions that made the incident possible

A systemic cause is one that was true before the incident, is still true after it, and will be true for the next incident too

• possibly in a completely different component.

You can recognise them by a test: if this incident had never happened, would this cause still be worth fixing? If yes, it is systemic. If it only makes sense as part of the story of this particular outage, it is a contributing factor for this incident and nothing more.

The previous lesson gave you a tree with labelled nodes. This one is about reading the tree for the causes that outlive it.

Normalised deviance

The most common systemic cause, and the hardest to see from inside, is a deviation from the intended state that has been present so long it has stopped registering as a deviation.

The pattern is always the same. Something is not right. Nothing bad happens. The absence of consequences is taken as evidence that it was fine. The new state becomes the baseline, and the next deviation starts from there.

Concrete forms, all of them ordinary:

• A monitoring alert that has fired daily for months and is routinely closed unread
• A cluster that has been running one node short since a failure last quarter
• A backup job that has been failing on one of six hosts and is described as "the flaky one"
• A test suite with eleven known-failing tests that everyone skips past
• A certificate renewed by hand each year because the automation broke and nobody had time
• A "temporary" firewall rule from a 2024 incident that is still in place

Each is a control that has been quietly switched off. The incident happens when the thing that control existed to catch finally arrives - and the review, looking only at the incident window, sees a fault rather than months of erosion.

$ systemctl list-units --state=failed --no-legend; systemctl list-timers --all --no-pager | awk '$1 == "-"'

● backup-offsite.service loaded failed failed Off-site backup sync
-      -   Mon 2026-05-03 02:00:12 UTC  100 days ago  certbot-renew.timer  certbot-renew.service

The question that turns an observation into a systemic finding is how long has this been true? A failed unit is a ticket. A failed unit that has been failing for a hundred days is a statement about the feedback loop.

Missing feedback loops

Most systemic causes reduce to the same shape: something broke, and nothing told anybody.

The generalised question is not “was this component broken” but “what would have told us?” Ask it about every control the incident touched, and the answers are usually uncomfortable.

The last row generalises to a rule worth stating on its own: a control and the check on that control must not share a mechanism. If certificate expiry is monitored by the same system that renews certificates, one failure disables both, and you will discover it at expiry.

$ systemctl show backup-offsite.service -p ExecMainStartTimestamp -p Result -p NRestarts; journalctl -u backup-offsite --since '90 days ago' -p err --no-pager | wc -l

Result=exit-code
NRestarts=0
ExecMainStartTimestamp=Mon 2026-05-03 02:00:03 UTC
612

Coupling and blast radius

Some incidents are large not because the fault was large but because the system had no boundaries. That is a design property and it is systemic by definition.

Look for these in the tree:

• Shared fate. Independent services sharing a host, a filesystem, a database, a network path, or a credential. One fault takes all of them.
• No bulkhead. No resource limits, so a leak in one unit exhausts the host. MemoryMax and TasksMax in a drop-in are the cheapest bulkhead in the Linux toolbox.
• Synchronous dependency on something optional. A request path that blocks on a service which is not required for the request to succeed.
• Retry amplification. Clients retrying aggressively against a struggling service, converting a slowdown into an outage.
• Circular dependency. The recovery procedure depends on something the outage took down: DNS, the identity provider, the CA, the monitoring stack, the runbook wiki itself.

The last one deserves testing rather than assuming. Ask directly what recovery requires, and whether every one of those things survives the failure you are recovering from.

Look across incidents, not only within one

A single review cannot see a pattern by definition. Some systemic causes are only visible when several reviews are read together, and they are usually the highest-value findings an operations team ever produces.

1. Tag every review with the component, the failure class, the detection method, and the time to detect
2. Once a quarter, read the last quarter of reviews as a set
3. Look for a component appearing repeatedly - that is a design problem, not a run of bad luck
4. Look for a repeated detection method, especially "a customer told us"
5. Look for repeated action types that keep being written and keep not working
6. Look for actions from previous reviews that were never completed, and count them

That last item is the one that changes behaviour. A quarter in which forty per cent of follow-up actions were never completed is a finding about capacity and prioritisation, and it explains recurrences far better than any technical cause in any individual review. It is also invisible to every one of those reviews taken alone.

Knowledge check