Trigger, contributing factors, root cause - taking an incident apart

Root cause analysis begins where incident response ends. The troubleshooting loop takes you to step 8, service restored, and step 9 says “find the root cause after the pressure is off”. This part of the course is step 9 done properly.

The boundary matters. During the incident you need a cause good enough to act on, quickly, and being wrong is recoverable because you can try something else. After the incident you need a cause good enough to change the system with, and being wrong is expensive, because the work gets done and the incident happens again anyway.

Why “the root cause” is the wrong question

Ask a team what caused an outage and you will get one sentence. Ask five people and you will get five different sentences, all true:

• “The certificate expired.”
• “Nobody renewed it.”
• “The renewal cron job had been failing silently since March.”
• “There was no alert on certificate expiry.”
• “We have no inventory of certificates, so nobody could have known.”

These are not competing answers. They are different layers of the same causal structure, and the argument about which one is “the” root cause is the argument that stops reviews producing useful work.

Decompose instead.

The three parts

Trigger. The specific event, at a specific time, that started this incident. It is usually unremarkable and it is usually a change: a deploy, a config push, a package upgrade, a traffic spike, a disk finally filling, a clock reaching a particular value.

Contributing factors. The conditions that were already true, and that turned the trigger into an outage rather than a non-event. Missing redundancy, a missing alert, a wrong runbook, a tight coupling, an undocumented dependency, a saturated resource.

Root cause. The decision, gap or absence that put those conditions in place and kept them there. Not “somebody made a mistake” but “the system had no mechanism by which this would have been noticed or prevented”.

Applied to the certificate example:

TRIGGER
  The TLS certificate for api.example.com reached its notAfter
  time at 09:14 UTC on 2026-08-11.

CONTRIBUTING FACTORS
  1. The renewal timer had been failing since 2026-03-02; its
     failure was logged and not alerted on.
  2. No monitoring check existed for certificate expiry.
  3. The service loads the certificate at start-up only, so the
     failure surfaced as a total outage rather than a warning.
  4. No inventory listed which certificates existed, on which
     hosts, or who owned them.

ROOT CAUSE
  Certificate lifecycle was treated as a one-off installation
  task rather than as a managed asset with an owner, an expiry
  date, and an alert. Nothing in the system would have surfaced
  the gap before it caused an outage.

The trigger is a date arriving. You cannot prevent a date from arriving. Everything you can act on is in the other two sections, and that is the point of separating them.

Which contributing factors matter

Most incidents have between three and eight contributing factors, and you will not fix all of them. Sort them by two questions.

1. Is it general or specific? A factor that applies to one service is worth less than one that applies to every service on the platform
2. Is it removable, or is it inherent to a design you are not going to change this year?

The best actions come from factors that are general and removable. In the example above, factor 2 - no expiry monitoring

• is both: one check applied to every certificate in the estate, and the whole class of incident becomes a ticket instead of an outage.

Factor 3 - the service reads its certificate only at start-up - is general but not cheaply removable; it is how the software is written. Record it as a known constraint rather than pretending it is an action item, because an action nobody can complete sits open forever and devalues every other item on the list.

Counterfactual, not chronological

The reliable test for whether something is a contributing factor is counterfactual: if this had not been true, would the incident have happened, or been shorter, or been smaller?

It is easy to list everything that was happening at the time. Most of it is background. The counterfactual separates the two:

• “The deploy happened at 13:55.” Remove it and the incident does not occur. Trigger.
• “The pool-saturation metric existed but had no alert.” Remove that gap and the incident is detected in two minutes instead of forty. Contributing factor.
• “It was a Tuesday.” Removing it changes nothing. Background.
• “The on-call engineer was new to the service.” Remove it and diagnosis is faster. Contributing factor - and note that the action is a runbook, not a different engineer.

That last one is where reviews slide toward blame. The counterfactual identifies unfamiliarity as a real factor, and the honest response is that the system required knowledge it did not supply. The lesson on avoiding human error as a cause takes this further; the point here is that the counterfactual test finds the factor without implying the fault.

Root cause is usually organisational

Contributing factors are technical. The root cause underneath them is usually not.

• Nobody owned the component, so nothing about it was maintained.
• The team knew about the gap and had no time allocated to close it.
• The alert existed and was routed to a channel nobody reads.
• The check was removed during an unrelated migration and nothing verified it came back.
• The design assumed a dependency that was never contractual and quietly changed.

These are uncomfortable to write, which is exactly why blameless review matters as a working condition rather than as a slogan. A review that cannot say “no time was allocated to this” will produce a technical action item that nobody has time for either, and the loop closes on itself.

Reading it out of the evidence

The decomposition is built from the timeline, not from recollection. The trigger in particular has to be pinned to a timestamp you can point at.

$ journalctl --since '2026-08-11 08:55' --until '2026-08-11 09:25' -o short-precise --no-pager -u myapp -u certbot-renew

09:14:02.771 app01 myapp[2214]: ERROR upstream https://api.example.com: x509: certificate has expired or is not yet valid
09:14:02.902 app01 myapp[2214]: WARN circuit breaker open for upstream api
09:14:41.004 app01 systemd[1]: certbot-renew.service: Main process exited, code=exited, status=1/FAILURE

The next lesson covers timeline reconstruction properly. The point here is that the trigger is an observation with a timestamp, the contributing factors are conditions you can evidence as having been true beforehand, and the root cause is the argument you make from both. Each is a different kind of claim, and a review that blurs them cannot be checked by anyone who was not there.

Knowledge check