Mitigation vs Root Cause

A P1 pages at 03:14. The primary on-call engineer recognises the symptom from the previous quarter. They open the incident-channel. The mitigation (a service restart that has worked before) is initiated within four minutes. The parallel RCA work continues with a second engineer. The mitigation lands at 03:19. SLO burn stops at 03:19. The RCA work continues to 03:48, by which point the actual root cause is identified: a connection-pool misconfiguration, distinct from the symptom the restart masked. Both are written to the runbook. The mitigation timeline is five minutes; the RCA timeline is thirty-four minutes. The two clocks overlap.

This is the most common shape of mitigation-vs-RCA for P1: parallel work in two lanes, with the mitigation landing first and the RCA closing later. The shape exists because the SLO burn is the cost-driver on a P1 and the root cause is the cost-driver on the next P1. The methodology assigns the two lanes deliberately.

What mitigation and RCA are

Two distinct objectives, each with its own loop and its own budget.

   Mitigation                              RCA
   +-----------+                           +-----------+
   | Detect    |                           | Locate    |
   +-----+-----+                           +-----+-----+
         |                                       |
         v                                       v
   +-----------+                           +-----------+
   | Choose    |                           | Identify  |
   | tactic    |                           | the layer |
   +-----+-----+                           +-----+-----+
         |                                       |
         v                                       v
   +-----------+                           +-----------+
   | Apply     |                           | Test      |
   | change    |                           | against   |
   +-----+-----+                           | evidence  |
         |                                +-----+-----+
         v                                       |
   +-----------+                                 v
   | Verify    |                           +-----------+
   | symptom   |                           | Document  |
   | stops     |                           |           |
   +-----+-----+                           +-----------+
         |
         v
   +-----------+
   | Pass to   |   <-- handoff: mitigation is on the RCA track
   | RCA       |
   +-----------+

Mitigation is symptom-oriented: it stops the user-visible failure without claiming a root cause. Restart, revert, drain, failover, rate-limit, capacity bump are all mitigations. RCA is cause-oriented: it identifies the upstream failure that produced the symptom, with evidence that survives the mitigation. Mitigation is reversible in minutes; RCA is durable in the runbook log.

The two objectives overlap heavily in the inputs (the same telemetry, the same change log) but produce different artefacts.

The two lanes are run by different people (primary on-call plus a domain specialist, or two engineers in parallel) or by the same engineer at different phases of the same call.

Per-tier ordering

The platform usually defines four tiers. The order between mitigation and RCA depends on the tier; getting the order wrong is the most expensive mistake in this lesson.

Three points the table makes:

1. At P0, mitigation is overriding. The SLO burn is the single driver. The hypothesis is written after the mitigation, not before.
2. At P1, the lanes are parallel. Mitigation is on the primary; RCA is on a secondary. The handover between lanes is the runbook log entry.
3. At P3, RCA is the driver. Mitigation is taken only when the cost of the ongoing degradation is lower than the cost of the mitigation work. Some P3 incidents never get mitigated; they get RCA’d to completion at release.

The most common shape: the runaway parallel

The most common shape the methodology catches is a runaway parallel at P1: the mitigation is in flight, the RCA continues alongside, and the engineer who is doing both keeps finding more evidence. The mitigation never lands because every new finding pulls the engineer into a new hypothesis iteration. The SLO burn continues. The engineer finally applies the original mitigation thirty minutes late, having generated a stack of new evidence that does not change the mitigation.

   wallclock
   0            15 min           30 min           45 min
   |-------------|----------------|----------------|
   | m1 (start) | m1 (in flight) | m1 (revised)   | m1 (lands)
   |            | r1             | r2             | r3 (RCA done)
   |            |                | r2 (re-find)   |
   |                                          burn stops here

The shape has three signatures:

1. Mitigation has been “in flight” for more than ten minutes without landing. A restart, a revert, a drain, a failover should each land within ten minutes; if it has not, the engineer is running two loops and finishing neither.
2. The RCA is producing evidence faster than the mitigation can verify. Each new piece of evidence refines the hypothesis; the engineer wants to wait for the refined evidence before applying the mitigation.
3. The engineer is the only person on the lane. Two lanes, one engineer; the budget is exceeded on both.

The fix is the handoff: at the first sign of runaway, the primary on-call engineer hands the RCA lane to a secondary (or to a domain specialist) and lands the mitigation. The handoff is one Slack message and a five-line summary of the hypothesis so far.

How to configure it

The mitigation-vs-RCA ordering is operational discipline, but the platform can encode the handoff trigger as an alert:

# /etc/prometheus/rules/mitigation.rules.yml
groups:
- name: mitigation-handoff
  rules:
  - alert: MitigationInFlight
    expr: |
      max_over_time(
        alert_active_minutes{alertname=~"CheckoutHighErrorRateP1"}
        [10m]
      ) > 10
    for: 0m
    labels:
      severity: ticket
      team: payments
    annotations:
      summary: 'P1 mitigation has been in flight > 10 min without landing'
      description: |
        The mitigation for {{ $labels.alertname }} has been active
        for more than 10 minutes. Action: confirm the mitigation
        engineer is the only person on the lane; hand off RCA to
        a secondary if not; land the mitigation.

The alert posts a ticket the moment the mitigation clock passes ten minutes without landing. The on-call manager reads the ticket, sees the runway, and intervenes before the RCA pulls the engineer further off course.

A runbook log entry template that records the mitigation-vs-RCA decision:

# Runbook: {{ alertname }}, {{ date }}

## Mitigation
- Tactic: {{ restart | revert | drain | failover | rate-limit | capacity-bump }}
- Started at: {{ timestamp }}
- Landed at: {{ timestamp }}
- Verification: symptom metric returned to baseline at {{ timestamp }}

## RCA
- Hypothesis (with falsifier): {{ ... }}
- Evidence (3 pieces): {{ ... }}
- Test verdict: confirmed / refuted / unde-verified
- Root cause: {{ ... }}
- Follow-up actions: {{ ... }}

## Handoff
- Mitigation handed off to RCA at: {{ timestamp }} (if applicable)
- Engineer assigned: {{ who }}

The “Handoff” section is the audit trail for the lane switch. Without it, the next engineer reading the runbook cannot tell which lane was open for how long.

How to validate it

Validate the mitigation-vs-RCA discipline by auditing the runbook log entries against the per-tier ordering rule and the handoff trigger.

# SEVERITY: READ-ONLY
# 1. Confirm the mitigation alert rule is wired.
curl -s 'http://prometheus:9090/api/v1/query?query=ALERTS%7Balertname%3D%22MitigationInFlight%22%7D' \
  | jq '.data.result | length'

Expected output:

3

Three alerts in the firing window is consistent with the P1 mitigation profile: each one fired during an incident that recovered within 30 minutes.

# SEVERITY: READ-ONLY
# 2. Count P1 runbook entries with both Mitigation and RCA sections
# populated.
grep -l '^## Mitigation' /srv/runbooks/checkout/p1-*.md | wc -l
grep -l '^## RCA'       /srv/runbooks/checkout/p1-*.md | wc -l

Expected output:

12
12

Twelve P1 entries, twelve with both sections filled. The two counts should match: a P1 with no Mitigation section is a P1 that did not mitigate; a P1 with no RCA section is a P1 that did not investigate.

# SEVERITY: READ-ONLY
# 3. Confirm at least one handoff appears in the last 90 days of P1 entries.
grep -l '^## Handoff' /srv/runbooks/checkout/p1-*.md | wc -l

Expected output:

2

Two handoffs in twelve P1 entries is a 16% rate; consistent with the per-tier budget (one in six P1s reaches the handoff trigger without recovering). A team with no handoffs on record is either running below the trigger rate (suspicious) or not recording the handoff (worse).

How it can fail

Six failure shapes occur when the mitigation-vs-RCA discipline is missing or reversed.

1. Mitigation-first on a P3. A P3 with no SLO pressure has the engineer applying a fix before the hypothesis is written. The fix is correct (or wrong); the evidence to confirm which is missing. Symptom: P3 incidents resolve without runbook entries; rate of recurrence is unchanged across quarters.
2. RCA-first on a P0. A P0 with active SLO burn has the engineer insisting on full RCA before applying any mitigation. The SLO burn continues; the engineer has spent the budget on phase 3-4 and the symptom has not been addressed. Symptom: the runbook entry has a 90-minute phase 3-5 with no Mitigation section; the SLO budget is exhausted.
3. Runaway parallel. The mitigation has not landed at minute 10; the engineer keeps refining the hypothesis. The two lanes compete for the same engineer. Symptom: the in-flight alert fires; the runbook entry records no handoff; the mitigation lands at minute 30 with the refined hypothesis, but the SLO budget is exhausted.
4. Same engineer on both lanes. A solo on-call engineer tries to mitigate and RCA at the same time. Neither lane finishes. Symptom: the runbook entry has gaps in both Mitigation and RCA sections; the engineer recorded the budget exhaustion in phase 6.
5. Mitigation hides the RCA. The engineer applies the mitigation, the symptom resolves, and the engineer closes the runbook with “mitigated by restart”. The RCA section is empty or one line. Symptom: the next P1 of the same class restarts at phase 1; the runbook log does not explain why the restart worked the previous time.
6. Mitigation that makes the symptom worse. The engineer applies a mitigation that is for a different root cause class (e.g., raises CPU limits for a connection-pool misconfiguration). The symptom persists; the engineer has spent budget and changed state. Symptom: the runbook entry records the mitigation, records that the symptom did not improve, and notes the engineer rolled back the change at minute 25.

How to troubleshoot a mitigation-vs-RCA that is not being run correctly

When the runbook review shows missed mitigation landings or RCA-first behaviour on a P0, the diagnostic order is:

1. Audit the in-flight alert history first. A team that has the in-flight alert firing but no handoffs on record is the runaway-parallel shape.
2. Audit the Mitigation section completeness. A P1 with no Mitigation section was supposed to mitigate; did not record why it did not.
3. Audit the RCA section completeness. A P1 with Mitigation but no RCA is the mitigation-hides-the-RCA shape.
4. Audit the tier labels. A P0 page that ran with P1 budgets (mitigation-first turned into mitigation-then- think) is the tier misapplication shape from lesson 03.
5. Audit the handoff contacts. A handoff that is recorded without an “Engineer assigned” line is the dropped-handoff shape.

Security implications

The mitigation-vs-RCA discipline is largely orthogonal to security, with two exceptions.

• A mitigation that resets credentials or rotates keys during an incident is a security event. The rotation logs need to be added to the runbook entry at phase 6; they are evidence the security team needs to find.
• An RCA that pulls PII from logs to confirm a hypothesis is a PII access event. The access logs need to be linked to the runbook entry; the on-call role assumption is the audit trail.

Performance implications

The mitigation-vs-RCA discipline has two platform-level performance implications:

• The mitigation lane produces bounded query load. A ten-minute mitigation window against a known prior incident is a bounded phase 4 cost.
• The RCA lane produces unbounded query load. A 14-day RCA window against a novel failure mode can produce sustained query load on the platform; the load is offset by recording rules and structured log indices, but a runaway RCA can still pin a Loki querier. The handoff is also a load control; the RCA lane assigned to a secondary has a budget that the audit captures.

Production guidance

• Make the lane assignment explicit in the runbook template. A runbook with no Mitigation or RCA section is incomplete.
• Make the handoff trigger a platform alert. A ten- minute clock without a landing is the cheapest operational signal the platform can produce.
• Bound the RCA lane with the same per-tier budget. A P1 RCA wallclock of 14 days is not optional; the runbook log has the slot to record the closure.
• Review handoff rates per tier monthly. A 0% handoff rate on P1 means the trigger is not firing or the handoffs are not being recorded.

Verification

You should now be able to answer:

• What is the per-tier ordering between mitigation and RCA for a P0 versus a P1 versus a P3?
• What is the most common shape of the discipline (the parallel investigation), and what are its three signatures?
• What is the handoff trigger, and how is it encoded in the platform?
• Which of the six failure shapes (mitigation-first on P3, RCA-first on P0, runaway parallel, same engineer on both lanes, mitigation hides RCA, mitigation makes the symptom worse) maps to a P1 runbook entry whose Mitigation section says “landed at 03:48” and whose in-flight alert fired at minute 10?
• Why does the handoff record belong in the runbook template instead of in a Slack thread?

Quiz