Methodology Summary

A primary on-call engineer opens a Sev1 ticket: checkout 5xx rate at 7% in us-east-1. They have one stack of telemetry and one SLO budget. They have 15 minutes before the on-call manager asks for an update. A second engineer, three hours into a Sev3 ticket about a slow report exporter, opens the same Grafana at a different time, with no SLO pressure and a 5-day target. The two engineers ran the same loop but applied it with different weights. That difference is the methodology.

This lesson is the methodology as a whole. The next five lessons in Part XCVIII drill into each phase. The intent here is the map: what the loop looks like, what artefact each phase leaves behind, and how the loop is weighted when the incident is a P0 versus a P3.

What the methodology is

The methodology is a six-phase loop that converts an observed failure into a verified root cause and a documented mitigation. The loop is the same regardless of the signal (metrics, logs, traces) and regardless of the on-call schedule. The weighting of each phase, however, varies by severity tier.

   +---------------------+
   |  1. Define symptom  |   <-- phase 1: observed failure in plain language
   +----------+----------+
              |
              v
   +---------------------+
   | 2. Quantify impact  |   <-- phase 2: numbers; how many, how bad
   +----------+----------+
              |
              v
   +---------------------+         +---------------------+
   | 3. Form hypothesis  |<------->| 4. Find evidence    |  <-- phase 3 / 4 iterate
   +----------+----------+         +----------+----------+
              |                                |
              +--------------------------------+
                            |
                            v
              +--------------------------+
              | 5. Test the hypothesis  |   <-- confirm or refute
              +------------+-------------+
                           |
                           v
              +--------------------------+
              | 6. Locate root cause and |   <-- artefact: runbook log entry
              |    document mitigation   |
              +--------------------------+

Every phase produces an artefact that the next phase consumes. The artefact trail is what makes the loop repeatable across engineers and across shifts.

Phases 3 and 4 iterate. Phases 1, 2, 5, and 6 are linear. The loop does not exit until phase 6 has produced a written entry.

Why a sysadmin cares

The methodology is the difference between a platform that pages you and a platform that helps you. Two teams with identical Prometheus, identical Loki, identical Tempo, and identical alerts will have an order-of-magnitude difference in MTTM (mean time to mitigate) if one team follows the loop and the other does not. The loop is operational discipline; the tooling is the surface.

Without a shared methodology, the on-call rotation learns the stack by accident. One engineer knows the checkout service; another knows the payments service; a third knows the database. The team appears competent because each individual is. The team is fragile because no one can stand in for another during a page. The loop is what converts individual knowledge into shared practice.

How it works: severity tiers weight the loop

The methodology is the same shape, but each severity tier compresses, expands, or skips some phases. The platform usually defines four tiers; the weights below are the implementation that has survived contact with three production SRE rotations.

Three points the table makes:

1. The hypothesis (phase 3) is the most-compressed phase. At P0 the hypothesis can be written after the mitigation, but it is still written. At P1 and below the hypothesis must exist before any change is applied.
2. Mitigation precedes full RCA only when SLO pressure forces it. A P0 with a burning error budget has mitigation as the primary obligation; a P3 with no SLO impact has the opposite ordering and may skip mitigation entirely if the cost of mitigation is higher than the cost of the ongoing degradation.
3. The PIR (phase 6+) has a deadline that depends on tier, not on the on-call engineer. A P0 PIR is opened within 24 hours and reviewed within 5 days; a P3 retrospective is attached to the release that closed the issue.

How to configure the methodology in code

The methodology is operational discipline, not configuration. The platform artefacts that enforce the loop are alert annotations, on-call routing, and the runbook log. A trimmed example:

# /etc/prometheus/rules/checkout.rules.yml
groups:
- name: checkout.rules
  rules:
  - alert: CheckoutHighErrorRateP0
    expr: |
      sum by (region) (
        rate(http_requests_total{job="checkout",code=~"5.."}[2m])
      )
      / sum by (region) (
        rate(http_requests_total{job="checkout"}[2m])
      ) > 0.10
    for: 2m
    labels:
      severity: page
      tier: P0
      team: payments
      service: checkout
    annotations:
      summary: 'Checkout 5xx above 10% in {{ $labels.region }}'
      symptom: 'Checkout returns HTTP 5xx for the majority of attempts in {{ $labels.region }} since {{ $labels.alert_started_at }}'
      impact: 'Customer-visible checkout failure; SLO burn at 14x baseline'
      dashboard_url: 'https://grafana.example.com/d/checkout'
      runbook_url: 'https://runbooks.example.com/checkout/p0-high-error-rate'

  - alert: CheckoutHighErrorRateP1
    expr: |
      sum by (region) (
        rate(http_requests_total{job="checkout",code=~"5.."}[5m])
      )
      / sum by (region) (
        rate(http_requests_total{job="checkout"}[5m])
      ) > 0.03
    for: 10m
    labels:
      severity: page
      tier: P1
      team: payments
      service: checkout
    annotations:
      summary: 'Checkout 5xx above 3% in {{ $labels.region }}'
      symptom: 'Checkout returns HTTP 5xx for more than 3% of attempts in {{ $labels.region }}'
      impact: 'Customer-visible checkout failure; SLO burn at 4x baseline'
      dashboard_url: 'https://grafana.example.com/d/checkout'
      runbook_url: 'https://runbooks.example.com/checkout/p1-high-error-rate'

Two rules, two tiers, two annotation sets. The tier label is the platform signal that drives the per-tier weighting of the methodology. Alertmanager routes to a heavier rotation (P0 fan-out) and the runbook URL points to a different entry.

How to validate it

Validate the methodology by replaying the last 30 days of pages against the runbook log:

# SEVERITY: READ-ONLY
# 1. Confirm the alert rule syntax is valid and the tier label exists.
promtool check rules /etc/prometheus/rules/checkout.rules.yml

Expected output:

SUCCESS: rule files validated; 4 rules found, 0 errors

# SEVERITY: READ-ONLY
# 2. Confirm tier labels are present on fired alerts in the last 30 days.
curl -s 'http://prometheus:9090/api/v1/query?query=count%20by%20(tier)%20(ALERTS%7Balertname%3D%22CheckoutHighErrorRateP0%22%7D)' \
  | jq '.data.result'

Expected output:

[
  {
    "metric": {"tier": "P0"},
    "value": [1723641600, "2"]
  }
]

# SEVERITY: READ-ONLY
# 3. Confirm the runbook URL for the P0 alert returns 200 within 5 seconds.
curl -sI --max-time 5 'https://runbooks.example.com/checkout/p0-high-error-rate' \
  | head -n 1

Expected output:

HTTP/2 200

# SEVERITY: READ-ONLY
# 4. Confirm the runbook log has a phase 1-6 entry per P0 page in the last 30 days.
git -C /srv/runbooks log --oneline --since='30 days ago' -- checkout/p0-high-error-rate.md

Expected output:

a1b2c3d 2026-08-13 03:18 checkout p0: pool-size-vs-max-connections incident
e4f5g6h 2026-08-04 11:02 checkout p0: payment-svc 503 incident

The four checks confirm that the artefacts are wired. The fifth check confirms the loop is being followed: every P0 page in the last 30 days has a corresponding runbook entry. The review is the only check that catches phases 3 and 4, because those phases leave no Prometheus-side artefact.

How it can fail

The methodology fails when the team treats it as a checklist rather than a discipline. Six failure shapes recur:

1. Severity-shopping. The on-call engineer reads the alert, decides the tier is “really” P2 not P1, and runs the loop with P2 budgets. The SLO burns at 4x and the response cost matches P2 (60 min phase 3-4) when it should have matched P1 (15 min). Symptom: tier labels on resolved alerts differ from tier labels at page time.
2. Phase 3 skipped because it looked obvious. A senior engineer recognises the symptom and goes straight to mitigation. The mitigation works for the wrong cause; the next incident of the same class surfaces the gap. Symptom: the runbook entry has one line (“restarted X”); no phase 3 sentence.
3. Phase 4 evidence-shopping. The on-call engineer opens every dashboard in sequence hoping to find a smoking gun. The loop never reaches phase 5 because no hypothesis was tested. Symptom: runbook entry has 12 panel screenshots and no falsifier.
4. Loop abandonment at phase 5. Evidence confirms the hypothesis; the engineer treats the incident as closed and applies the mitigation without phase 6 documentation. The next incident of the same class restarts at phase 1. Symptom: alert resolves; runbook log has no entry for the page.
5. Mitigation-before-evidence on a P3. A P3 with no SLO pressure has the engineer applying a fix before the hypothesis is written. The fix is correct (or wrong); the evidence to confirm which is missing. Symptom: P3 incidents resolve without runbook entries; rate of recurrence is unchanged across quarters.
6. Tier static across the incident. A P1 that escalates to P0 partway through is still run with the original tier’s budget. Mitigation cost grows; SLO burns faster than the methodology can follow. Symptom: post-incident timeline shows a tier change at minute 18; the runbook entry still says “P1”.

How to troubleshoot a methodology that is not being followed

When the runbook review at the end of the quarter shows missing entries, missing hypotheses, or panel-shopping, the diagnostic order is:

1. Check the tier label first. A missing tier label produces all three failure modes above because it disables the per-tier weighting.
2. Audit the last 30 days of P0 entries by hand. The presence or absence of a hypothesis sentence is the single most informative signal.
3. Compare mitigation time per tier. A P1 with a 40-minute MTTM suggests the methodology is being run with P2 budgets; the tier labels are right but the budget is wrong.
4. Count phase 6 entries. If phase 6 entries per page fall below 80%, the runbook log is not being treated as operational. Re-read the alert annotations; the link is probably broken or the runbook URL is a 404.

Security implications

The methodology walks every telemetry surface. The permissions needed are the permissions to read everything: metrics, logs, traces, dependency maps, change logs.

• Prometheus query endpoints expose label values that may include user identifiers for metrics instrumented at request scope (cart totals, user IDs in error metrics).
• Loki query endpoints expose log lines, which carry PII if the application is not structured carefully.
• Tempo query by trace ID exposes span attributes, which may include database statements with PII or secrets passed as HTTP headers.

Limit methodology access to the on-call rotation and to operators with a recorded purpose. The on-call engineer needs access during a page and not the rest of the week. Short-lived credentials issued at page time and revoked at incident closure are the production answer.

Performance implications

The methodology adds latency to a single resolution but removes latency from every subsequent resolution. Teams that run the loop resolve the first incident of a class in roughly the same time as a structured guess. They resolve the second incident of the same class in a fraction of the time because the runbook log already has phases 1 to 5.

The largest performance cost is phase 4: queries against a live Prometheus, Loki, and Tempo during an incident. The queries are bounded by the time window of the investigation (typically 30 minutes) and the cardinality of the suspect metric family. Phase 4 should reuse recording rules where they exist; ad-hoc queries against a hot TSDB during a P0 are a risk.

Production guidance

• Tier the alert at definition time. Severity labels live in the alert rule, not in the engineer’s head.
• Annotate the alert with phase 1 and phase 2. Symptom statement and impact number belong on the alert; the page message is the phase 1 of the loop.
• Run the runbook log as operational infrastructure. Treat it like a database; back it up; have it on the platform’s uptime budget; alert on a 404 from runbook_url.
• Review the loop monthly. A 30-minute review of the last 30 days of P0 entries catches missing hypotheses, missing evidence, and missing phase 6 records.
• Bound the loop by tier, not by engineer. An engineer on a P0 has a different budget than the same engineer on a P3. The tier is the contract; the engineer does not negotiate it.

Verification

You should now be able to answer:

• What artefact does each of the six phases of the methodology produce, and which phase consumes which?
• How does the per-tier weighting of the methodology differ between a P0 and a P3?
• Which phase is most compressible, and at which tier is it allowed to be written after the mitigation?
• Which of the four common failure shapes (severity-shopping, evidence-shopping, loop abandonment, mitigation-before-evidence on a P3) maps to a runbook entry that has no hypothesis?
• Why is the loop faster on the second incident of the same class than on the first?

Quiz