Hypothesis and Evidence Discipline

A primary on-call engineer opens a P1: cart-svc returns 503 for 18% of attempts. The dashboards are open. The engineer sees a deploy at 02:47 that changed cart-svc’s circuit breaker threshold. The engineer also sees an elevated CPU on the database host at 03:00. Both pieces of evidence correlate with the symptom window. The engineer reverts the deploy. The 503 rate does not improve. Then the engineer raises the database host CPU limits. The 503 rate does not improve.

The engineer has spent thirty minutes, two refuted hypotheses, and four changes on a system. The actual cause, discovered forty minutes later, was a stale DNS entry caused by a third-party registry pushing an old SOA record during the same window. Three pieces of evidence, all visible on the dashboards, had pointed at three different causes. The engineer followed two of them to mitigation.

This lesson is the discipline that catches the third shape.

What the discipline is

The hypothesis-and-evidence discipline is the part of the methodology that protects the on-call engineer from the four red herrings that account for the majority of long investigations. It is a four-step protocol.

  +-------------------------+
  |  1. List 3 candidate    |
  |     causes              |
  +-----------+-------------+
              |
              v
  +-------------------------+
  |  2. Pick the most       |
  |     likely + write      |
  |     a falsifier         |
  +-----------+-------------+
              |
              v
  +-------------------------+
  |  3. Gather 3 pieces of  |
  |     independent         |
  |     evidence            |
  +-----------+-------------+
              |
              v
  +-------------------------+
  |  4. Test: confirm /     |
  |     refute / replace    |
  +-------------------------+

The discipline has four properties the runbook log enforces:

1. The hypothesis sentence is on paper before any query runs. Phase 4 is a test of the hypothesis, not a search for a hypothesis.
2. The falsifier is on paper at the same time as the hypothesis. A hypothesis without a falsifier is not a hypothesis; it is a hope.
3. The three pieces of evidence are independent. Two pieces of evidence that both measure the same metric label are one piece of evidence.
4. Three refuted hypotheses means stop and escalate. The engineer is in evidence-shopping territory; another engineer’s eyes are the next input.

Why a sysadmin cares

Four operational pains disappear when the discipline is applied.

1. The two-evidence confirmation. The engineer collects evidence that supports the hypothesis, applies the mitigation, and the symptom resolves by coincidence. The root cause continues to evolve; the next incident of the same class surfaces the gap. Three independent pieces of evidence catch coincidence.
2. The recoiling mitigation. The engineer finds a piece of elevated CPU, raises the limit, sees the symptom resolves, and considers the incident closed. Without a falsifier, the engineer has no way to distinguish “CPU was the cause” from “CPU was correlated with the cause”. The discipline requires a falsifier that explicitly addresses the correlation.
3. The hypothesis anchor. The engineer forms a hypothesis, finds an early piece of evidence that supports it, and stops searching. The hypothesis is correct as far as it goes, but the actual cause is upstream. Without independent evidence from three angles (metric, log, trace, dependency, change log), the engineer cannot tell.
4. The abandon-after-three rule. The engineer has spent two refuted hypotheses and is considering a fourth. The fourth is almost always shopping. The discipline forces escalation rather than continuing; the next pair of eyes may see what the first did not.

How to configure it

The discipline is operational, but the platform can encode the three-evidence requirement and the one-falsifier requirement in the runbook log template. A trimmed runbook template:

# Runbook: {{ alertname }}, {{ date }}

## Phase 1 - symptom
- One-sentence symptom statement, with timestamp.

## Phase 2 - impact
- Number + time window: e.g., 3,200 failed requests in
  18 min; SLO burn 14x.

## Phase 3 - hypothesis
- Hypothesis sentence:
  "The {{ component }} is the causal bottleneck; symptom
  pattern matches {{ class }}; most recent change at
  {{ timestamp }} aligns with symptom onset at
  {{ timestamp }}."
- Falsifier sentence:
  "Falsified if the trace of one failing request shows
  {{ metric }} outside the hypothesis window, OR the
  {{ metric }} is not elevated in the symptom window, OR
  a {{ metric }} change at {{ timestamp }} aligns with
  a different cause."

## Phase 4 - evidence (3 independent pieces)
1. {{ evidence_1 }}
2. {{ evidence_2 }}
3. {{ evidence_3 }}

## Phase 5 - test
- Confirmed: hypothesis confirmed by 3 pieces; proceed
  to mitigation.
- Refuted: hypothesis refuted by 1 piece; record which
  piece and why; refine or replace hypothesis.
- Unverified: evidence ambiguous; record why and the
  next query.

## Phase 6 - root cause + mitigation
- Root cause: ...
- Mitigation: ...
- Verification: symptom metric returned to baseline at
  {{ timestamp }}.
- Follow-up actions: ...

The template enforces the discipline. An engineer who fills in the template without writing a falsifier at phase 3 is visibly off-protocol; the next engineer reading the runbook can ask for the falsifier before continuing.

A second encoding is the alert rule annotation, which can carry the falsifier alongside the symptom:

annotations:
  symptom: 'Checkout returns HTTP 5xx for ~18% of attempts since 03:14'
  falsifier: 'Refuted if the trace of one failing request shows the slow span on cart-svc rather than payment-svc, OR the recent payment-svc deploy does not align with symptom onset'

The falsifier in the annotation is the engineer’s first read at page time. Pasting it into the runbook log entry at phase 3 is one minute of work.

How to validate it

Validate the discipline by reading the runbook log and checking the falsifier column.

# SEVERITY: READ-ONLY
# 1. Confirm the falsifier slot exists in the runbook template.
grep -c 'Falsifier sentence' /srv/runbooks/templates/incident.md

Expected output:

1

# SEVERITY: READ-ONLY
# 2. Count runbook entries with the falsifier slot filled.
grep -l '^Falsifier sentence: .\{20,\}' /srv/runbooks/checkout/p?-*.md \
  | wc -l

Expected output:

9

# SEVERITY: READ-ONLY
# 3. Count runbook entries with the falsifier slot empty.
grep -l '^Falsifier sentence: *$\|^Falsifier sentence: *$' \
  /srv/runbooks/checkout/p?-*.md | wc -l

Expected output:

1

Ten entries total, nine with falsifier, one without. A team with 10% falsifier-empty rate is at the boundary; a team with 0% is doing well. A team with 50% is running the methodology without the discipline.

# SEVERITY: READ-ONLY
# 4. Confirm phase 4 lists three pieces of evidence per entry.
grep -E '^[0-9]\. ' /srv/runbooks/checkout/p?-*.md | wc -l

Expected output:

30

Thirty numbered entries across ten runbooks: three pieces of evidence per entry. The four checks together confirm that the discipline is wired.

How it can fail

Six failure shapes occur when the discipline is missing or half-applied.

1. Hypothesis without falsifier. The engineer forms a hypothesis sentence but does not write a falsifier. Phase 4 becomes confirmation bias: every piece of evidence is read as supporting the hypothesis. Symptom: the runbook entry has a phase 3 sentence; the falsifier slot is empty.
2. Three pieces of dependent evidence. The engineer collects three pieces of evidence that all measure the same metric label. They count as one piece, not three. The mitigation looks well-supported but is actually anchored on a single measurement. Symptom: phase 4 has three line items; all three cite the same Prometheus metric with different PromQL functions.
3. Anchor on the most salient piece of evidence. The engineer sees the recent deploy and treats it as primary, ignoring the elevated CPU and the database problem. Symptom: phase 4 evidence list contains only change-log citations; metric and trace evidence is absent.
4. Recycling a refuted hypothesis. The engineer has refuted hypothesis A. The engineer forms hypothesis A again in slightly different wording and tests it again. Symptom: phase 5 records “refuted” for one iteration and “refined” for the next, where the refinement is identical.
5. Confirmation by coincidence. Two pieces of evidence support the hypothesis; the mitigation resolves the symptom; the root cause continues. The engineer did not gather a third piece because “two is enough”. Symptom: the runbook entry has two numbered items; the mitigation was applied; the next incident of the same class surfaces within 30 days.
6. Abandon-after-three not enforced. Three hypotheses have been refuted; the engineer forms a fourth. The fourth is shopping; the engineer does not escalate. Symptom: the runbook entry records four iterations; the phase 6 mitigation is on hypothesis four; the runbook log lists no escalation contact.

How to troubleshoot a discipline that is not being followed

When the runbook review shows missing falsifiers or single-metric evidence, the diagnostic order is:

1. Audit the falsifier column first. A missing falsifier is the most informative failure shape; it correlates with missing iterations and escalations.
2. Audit the evidence independence. Two line items that quote the same metric with different PromQL functions are dependent evidence; the discipline requires three independent angles (metric, log, trace, change).
3. Audit the iteration count. Four iterations in phase 3-4 is the abandon-after-three rule violated.
4. Check the abandon-after-three alert history. A team that has refuted-three alerts firing but no escalations on record has a discipline-without-escalation shape.

Security implications

The hypothesis-and-evidence discipline is largely orthogonal to security. Two interactions matter.

• The change log is an evidence source. Phase 4 may consult the change log for evidence; if the change log contains secrets in annotations (database credentials, API keys), the evidence surface becomes a credential surface. Strip secrets at the change-log layer.
• Three independent angles means three access paths. Reading metrics, logs, traces, and the change log from one network zone is operationally efficient but expands the access boundary. The shortest fix is to grant the on-call engineer a single role that includes the four read paths and to audit at the role boundary.

Performance implications

The discipline is performance-neutral for the platform but has a subtle effect on the on-call rotation.

• Evidence independence limits redundant queries. A discipline that requires three independent angles prevents the engineer from running three PromQL queries against the same metric family. The platform saves query load.
• Falsifier-on-paper shortens phase 4. An engineer with a falsifier on paper knows which query to run next; an engineer without one runs queries until something looks plausible. Phase 4 elapsed time drops by roughly half when the falsifier is on paper.

Production guidance

• Make the falsifier a mandatory slot in the runbook template. The template is the cheapest enforcement mechanism.
• Review the falsifier column monthly. A 10% falsifier empty rate is a sign the discipline is being short-circuited; a 50% rate is an incident waiting to happen on the next mitigation.
• Use the abandon-after-three alert as a routing signal. When a third hypothesis is refuted, the alert manager routes to the on-call manager’s rotation; the manager joins the channel and is the second pair of eyes.
• Treat the falsifier as operational documentation. A falsifier that names a trace attribute is a reusable falsifier; the next engineer can re-use it for the next investigation of the same class.

Verification

You should now be able to answer:

• What is the four-step protocol of the hypothesis-and- evidence discipline?
• Why must the falsifier be written alongside the hypothesis, not after the evidence?
• What counts as “independent” evidence, and why do two PromQL queries against the same metric not count as two pieces?
• What is the abandon-after-three rule, and what does it mean for the runbook entry?
• Which of the six failure shapes (hypothesis without falsifier, three pieces of dependent evidence, anchor on most salient, recycling refuted hypothesis, confirmation by coincidence, abandon-after-three not enforced) maps to a runbook entry whose phase 5 records “refined” twice with identical wording?
• Why does writing the falsifier on paper shorten phase 4 query time?

Quiz