Investigation Anatomy

At 03:14 the on-call rotation is paged: checkout 5xx rate above 5% in us-east-1. Two engineers respond in the channel. Engineer A opens Grafana, scrolls the dashboards, opens the dependency panels, opens the change log, finds a deploy at 02:47 that changed the payment-service connection pool, correlates the deploy window with the 5xx start, pulls the trace of one failed checkout, confirms a 4.2 s wait on payment-svc, and writes a mitigation within twelve minutes. Engineer B opens the same Grafana, sees the wall of green panels, and starts reading the Prometheus configuration file. Three quarters of an hour later Engineer B is still reading.

The two engineers had access to the same telemetry. Engineer A worked an investigation. Engineer B was looking at dashboards. The difference is structure. This lesson is the structure.

What it is

An investigation is a six-phase loop that converts a user-visible failure into a verified root cause and a documented mitigation. The phases are not optional, and the loop does not run once: a phase that fails to produce evidence sends the engineer back to the hypothesis, not forward.

  +---------------------+
  |  1. Define symptom  |
  +----------+----------+
             |
             v
  +---------------------+
  | 2. Quantify impact  |
  +----------+----------+
             |
             v
  +---------------------+        +---------------------+
  | 3. Form hypothesis  |<------>| 4. Find evidence    |
  +----------+----------+        +----------+----------+
             |                              |
             +------------------------------+
                            |
                            v
              +--------------------------+
              | 5. Test the hypothesis  |
              +------------+-------------+
                           |
                           v
              +--------------------------+
              | 6. Locate root cause and |
              |    document mitigation   |
              +--------------------------+

Phases 3 and 4 iterate: hypothesis, evidence, refine, evidence. Phases 5 and 6 are linear. The investigation is not complete until the sixth phase has produced a written record.

Why a sysadmin cares

Mean time to mitigate (MTTM) for the same incident, on the same platform, varies by an order of magnitude depending on whether the on-call engineer follows the loop. The variation is not a talent difference; it is a discipline difference. Engineer A above did not know the answer at 03:14. Engineer A knew which dashboard to open and which log query to run. The loop is what shortened the path.

Without a loop, the on-call rotation learns the platform by accident. One engineer learns the checkout service; another learns the payments service; a third knows the database. The team appears competent because each individual is. The team is fragile because none of them can stand in for another. The loop is what converts individual knowledge into shared operational practice.

How it works

The mental model is a loop with a written artefact at every exit. Each phase answers one question and produces one artefact that the next phase consumes.

The loop is not strictly linear. Phases 3 and 4 iterate. A hypothesis that is refuted by the evidence returns to phase 3 with a new hypothesis; a hypothesis that is confirmed but does not explain the symptom returns to phase 3 with a refined hypothesis. The loop only exits at phase 6.

The six phases in detail

Phase 1. Define the symptom. “Checkout is failing” is not a symptom. “Checkout returns HTTP 503 for roughly 30% of attempts in us-east-1 since 02:50” is a symptom. The definition must be specific enough to drive a query and falsifiable enough that a later engineer can confirm whether the symptom still applies.

Phase 2. Quantify impact. Translate the symptom into business units: percentage of users affected, requests per minute failing, error budget burned per hour. The number matters because it sets the urgency of the next phase.

Phase 3. Form a hypothesis. Pick the most likely cause before you collect more data. Naming the hypothesis narrows the next query; without it, the engineer collects everything and explains nothing.

Phase 4. Find evidence. Run one or two queries. The hypothesis tells you which dashboard to open; the evidence tells you whether the dashboard confirms or refutes the hypothesis. The most expensive mistake in this phase is collecting more evidence than the hypothesis needs.

Phase 5. Test the hypothesis. Does the evidence confirm or refute? A refuted hypothesis is not a failure; it is the next input to phase 3. A confirmed hypothesis that does not explain the symptom returns to phase 3 with a refined hypothesis.

Phase 6. Locate root cause and document. The root cause is the layer at which the failure began. The mitigation is the change that restores service. Both are written to a runbook log so the next investigation starts from evidence, not from zero.

How to configure it

The investigation loop is operational discipline, not configuration. The closest analogue in Prometheus configuration is the alert rule annotation set: every firing alert should carry a dashboard_url, a runbook_url, and a summary annotation that names the symptom in plain language. The alert rule is the trigger; the runbook log is the record.

An alert rule that respects the loop:

groups:
- name: checkout.rules
  rules:
  - alert: CheckoutHighErrorRate
    expr: |
      sum by (region) (
        rate(http_requests_total{job="checkout",code=~"5.."}[5m])
      )
      / sum by (region) (
        rate(http_requests_total{job="checkout"}[5m])
      ) > 0.05
    for: 5m
    labels:
      severity: page
      team: payments
      service: checkout
    annotations:
      summary: 'Checkout 5xx rate above 5% in {{ $labels.region }}'
      description: |
        Region {{ $labels.region }} checkout 5xx rate is
        {{ $value | printf "%.2f" }} for 5 minutes. Phase 1
        symptom is "checkout failing"; phase 2 impact is the
        rate above; phase 3 hypothesis is the most recent
        deploy to the region.
      dashboard_url: 'https://grafana.example.com/d/checkout'
      runbook_url: 'https://runbooks.example.com/checkout/high-error-rate'

The annotation names the symptom, the dashboard URL is the phase 4 evidence surface, and the runbook URL is the phase 6 record. The labels route the alert to the right team; the annotations give the team the loop.

A runbook log entry, recorded by the on-call engineer:

# Runbook: CheckoutHighErrorRate

## Symptom (Phase 1)
Checkout returns HTTP 503 for ~30% of attempts in us-east-1
since 02:50 UTC on 2026-08-13.

## Impact (Phase 2)
3,200 failed requests over 18 minutes; ~12% of error budget
for the 30-day window consumed in 18 minutes.

## Hypothesis (Phase 3)
The deploy at 02:47 changed payment-service connection pool
from 20 to 200; the database max_connections is 100. The pool
saturates the database; checkout waits on a connection; the
pool exhaustion surfaces as 503.

## Evidence (Phase 4)
Database active connections: 100/100 since 02:49.
payment-svc pool_size gauge: 200 (was 20).
Trace of failed checkout (id 0a3f...): 4.2 s wait on
payment-svc /charge, no upstream timeouts.

## Test (Phase 5)
Confirmed. Connection pool saturation aligns with symptom
onset, matches the deploy, and explains the 503 rate.

## Root cause + mitigation (Phase 6)
Connection pool size 200 exceeds database max_connections 100.
Mitigation: rollback deploy. Long-term: cap pool size at 80%
of max_connections; add a pre-deploy check.

## Follow-up
- Add pre-deploy check on payment-svc pool vs DB
  max_connections.
- Add recording rule on payment-svc pool saturation.
- Add alert at 80% pool utilisation (severity: ticket).

The entry is the artefact. The next engineer who pages on CheckoutHighErrorRate reads the entry first; the previous engineer has already done phases 1-5.

How to validate it

Validate the loop by replaying the last investigation against the runbook log:

# SEVERITY: READ-ONLY
# 1. Confirm the alert rule syntax is valid.
promtool check rules /etc/prometheus/rules/checkout.rules.yml

Expected output:

SUCCESS: rule files validated; 12 rules found, 0 errors

# SEVERITY: READ-ONLY
# 2. Confirm the rule fired in the alert history window.
curl -s 'http://prometheus:9090/api/v1/query?query=ALERTS{alertname="CheckoutHighErrorRate"}' \
  | jq '.data.result[] | .metric'

Expected output:

{
  "alertname": "CheckoutHighErrorRate",
  "severity": "page",
  "team": "payments",
  "service": "checkout",
  "region": "us-east-1"
}

# SEVERITY: READ-ONLY
# 3. Confirm the runbook URL resolves and the entry exists.
curl -sI 'https://runbooks.example.com/checkout/high-error-rate' \
  | head -n 1

Expected output:

HTTP/2 200

# SEVERITY: READ-ONLY
# 4. Confirm the dashboard URL resolves.
curl -sI 'https://grafana.example.com/d/checkout' \
  | head -n 1

Expected output:

HTTP/2 200

The four checks above confirm that the artefacts are wired. The fifth check confirms the loop is being followed: a post-incident review (PIR) or a random audit of the runbook log finds a phase 1-6 entry for each alert that paged in the prior 30 days. The audit is the only validation that catches phases 3 and 4 (hypothesis and evidence), because those phases leave no Prometheus-side artefact.

How it can fail

Six specific failure shapes recur when teams run the loop without the discipline:

1. Premature mitigation. The on-call engineer reads the symptom, opens the dependency dashboard, sees elevated CPU on the payment host, and resizes the host. The CPU was downstream of the actual cause (a connection pool misconfiguration). The host resize takes 20 minutes; the 5xx rate stays elevated; the engineer has lost the loop. Symptom: the runbook log entry for the incident does not contain a hypothesis, only a “raised CPU limits” action.

2. Evidence shopping. The on-call engineer forms no hypothesis at phase 3 and instead opens every dashboard in sequence, hoping to find a smoking gun. Each panel either confirms the engineer’s last guess or is set aside; the loop never reaches phase 5 because no hypothesis was tested. Symptom: the runbook log entry has 12 panel screenshots and no falsifier.

3. Loop abandonment at phase 5. The engineer finds evidence that confirms the hypothesis and treats the investigation as closed. The mitigation is applied without phase 6 documentation. Symptom: the next incident of the same class starts from phase 1 because the last investigation did not produce a record.

4. Symptom misdefinition. The page says “checkout errors”; the engineer treats the symptom as “any 5xx”. The actual symptom is “checkout 503s on payment-svc dependency timeout”. The misdefinition sends the engineer to the wrong service. Symptom: the investigation timeline shows the engineer opening panels for cart, inventory, and shipping before reaching payment.

5. Single-signal tunnel vision. The engineer follows one signal (usually the metric the page cited) and ignores the others. A latency investigation that opens only the metric panels and never opens a trace will miss the dependency that is the actual cause. Symptom: the runbook log entry lists only metric queries; no logs or traces are referenced.

6. Loop skipped because the answer “looked obvious”. A senior engineer who has seen the symptom before skips phases 3-5 and applies a known mitigation. The mitigation works for the wrong reason (or for a different cause class), and the actual root cause continues to evolve. Symptom: the runbook log entry is one line; the next incident of the same class has a different root cause.

How to troubleshoot it

When an investigation is taking longer than expected, the diagnostic order is:

1. Confirm the symptom is still observable. Phase 1 may have drifted: the user may have stopped reporting, but the metric may still be elevated. Or the metric may have recovered and the user reports may be stale.
2. Re-confirm the impact. Phase 2 may have been over-estimated; the SLO burn rate may be lower than the first engineer computed. A lower impact allows more time for phases 3-5.
3. Walk the loop. If phase 3 has no hypothesis, the investigation is data collection. Pick the most likely cause and write it down.
4. Bound the evidence search. If phase 4 has been open for more than ten minutes, the hypothesis is too broad. Refine the hypothesis or pick a different one.
5. Stop and re-read the change log. The most common cause of long investigations is a recent change. The change log is the fastest path back to phase 5.

Security implications

The investigation loop touches all four telemetry surfaces. The permissions needed to walk the loop are the permissions needed to read everything: metrics, logs, traces, and dependency metrics. Audit access accordingly.

• /api/v1/query and /api/v1/query_range on Prometheus expose metric labels, which can carry user identifiers for the metrics that include them.
• /loki/api/v1/query on Loki exposes log lines, which can carry PII if the application is not structured carefully.
• Tempo query by trace ID exposes the trace contents, which may include database queries, secrets passed as headers, or user identifiers in span attributes.

Limit investigation access to the on-call rotation and to operators with a recorded purpose. The investigation loop is the strongest argument for short-lived credentials on the observability stack: the on-call engineer needs access during the page and not the rest of the week.

Performance implications

The loop adds latency to the resolution path, but it removes latency from the learning path. Teams that follow the loop resolve the first incident in roughly the same time as a structured guess, but resolve the second incident of the same class in a fraction of the time because the runbook log already has phases 1-5.

The largest performance cost is at phase 4: the on-call engineer runs queries against a live Prometheus, Loki, and Tempo while the system is in an incident state. The queries are bounded by the time window of the investigation (typically 30 minutes) and by the cardinality of the suspect metric family. On a well-instrumented platform this is negligible; on a high-cardinality platform the same queries can dominate the rule evaluation budget if run during a high-traffic window. Phase 4 should reuse recording rules where they exist; ad-hoc queries against a hot TSDB during a page are a risk.

Production guidance

• Every page surfaces the loop. The alert annotation set must include the symptom statement (phase 1), the impact (phase 2), and the runbook URL (phase 6). A page without these is a page that does not invite the loop.
• The runbook log entry is the loop’s artefact. Every alert closure writes one entry. The entry is in version control, not in the chat thread.
• Phases 3 and 4 iterate, not the dashboard collection. A runbook entry that names the hypothesis is the cheapest possible onboarding artefact for the next on-call.
• The loop is reviewed, not policed. A monthly review of the last 30 days of runbook log entries is the right validation. The review finds missing hypotheses, missing evidence, and missing phase 6 records.
• The loop has an escape hatch. A P0 incident with customer-visible impact may skip phase 3 (hypothesis) for the first ten minutes while the team stabilises the system. The hypothesis is written after the mitigation, as the first line of the post-incident review. Skipping the loop is a one-time exception, not a habit.

Verification

You should now be able to answer:

• What artefact is produced by each of the six phases of the investigation loop, and which phase consumes which?
• Why must phase 3 (hypothesis) precede phase 4 (evidence)?
• What is the role of the runbook log entry in the loop, and what does it allow the next investigation to skip?
• Which of the six failure shapes maps to a runbook log entry that is missing a hypothesis?
• Why is the loop faster on the second incident of the same class than on the first?

Quiz