What Failed?

A page arrives at 03:00. The on-call engineer opens the runbook. The first line says “the system is broken.” They read it twice. It does not tell them whether they are looking at a database, a load balancer, or a payment provider. They scroll down. The next section says “investigate the problem.” They go back to Grafana.

A what-failed section is the load-bearing first paragraph of the runbook. Its job is small and unambiguous: tell the on-call exactly what has stopped working, in language that a telemetry-derived signal can confirm or refute. The rest of the runbook flows from that statement. A runbook whose first sentence is vague produces a vague investigation.

What it is

A what-failed section is a one-or-two-sentence statement at the top of a runbook that names:

1. The component that has failed. The service, dependency, host, or external integration. The name must match a label on the alert and a label in the platform (a service label, a host label, a deployment identifier).
2. The failure mode. Degraded, unavailable, slow, erroring, saturated, drifting, or leaked. The verb must be one the telemetry can confirm; “broken” is not.
3. The scope. The region, the customer segment, the dependency chain, the time window. Scope is what turns a component-level failure into an incident-shaped one.

A runbook that does not name all three has not yet said anything the on-call did not already know from the alert payload.

Why a sysadmin cares

The first thirty seconds of an incident set the trajectory of the next thirty minutes. The on-call needs to confirm the alert is the alert they are looking at, scope the failure to a region or dependency, and decide whether to escalate. A runbook whose first sentence fails to anchor the failure costs the on-call those thirty seconds and biases the rest of the investigation toward a wider, vaguer search.

The what-failed section also has a second role: it is the contract the alert payload is checked against. If the alert fires for orders-api in eu-west-1, the runbook must say orders-api in eu-west-1. If the alert fires for an empty label set, the runbook must say so explicitly and the team must have decided the empty-label case is acceptable. The what-failed statement is the meeting point between the rule and the doc.

How it works

The what-failed statement is the alert summary, expanded once. The rule already has the alertname, the labels, and the summary annotation. The what-failed section uses the same labels in prose and adds the scope:

  Alert payload (machine-readable)
  --------------------------------
  alertname: OrdersApiHighErrorRate
  service:   orders-api
  region:    eu-west-1
  severity:  critical
  summary:   orders-api 5xx ratio above 5% in eu-west-1
              |
              v
  What-failed section (human-readable, expanded)
  ---------------------------------------------
  The orders-api service in eu-west-1 is returning HTTP 5xx
  responses at a ratio above 5% over a rolling 5-minute window.

The mapping is mechanical. The alertname decomposes into service and failure mode. The labels carry the scope. The summary is the what-failed sentence in compressed form. A runbook whose what-failed section disagrees with the alert summary has a bug in either the rule or the doc.

How to configure it

The alert rule, with its summary annotation expanded into a what-failed sentence that the runbook can echo:

groups:
  - name: orders-api.slo
    rules:
      - alert: OrdersApiHighErrorRate
        expr: |
          sum by (service, region) (
            rate(http_requests_total{service="orders-api", status=~"5.."}[5m])
          )
          /
          sum by (service, region) (
            rate(http_requests_total{service="orders-api"}[5m])
          )
          > 0.05
        for: 5m
        labels:
          severity: critical
          team: checkout
          service: orders-api
        annotations:
          summary: 'orders-api 5xx ratio above 5% in {{ $labels.region }}'
          description: |
            The orders-api service in region {{ $labels.region }}
            has returned a 5xx ratio above 5% over the last
            5 minutes. Current ratio: {{ $value | humanizePercentage }}.
          runbook_url: 'https://runbooks.example.com/checkout/orders-api-5xx.html'
          dashboard_url: 'https://grafana.example.com/d/orders-api/orders-api-overview?var-region={{ $labels.region }}&from=now-1h&to=now'

The matching runbook opening:

# orders-api 5xx ratio above 5%

## What failed

The orders-api service in {{ $labels.region }} is returning HTTP
5xx responses at a ratio above 5% over a rolling 5-minute window.
The condition has held for at least 5 minutes, which is why the
alert has fired.

Component: orders-api (Kubernetes deployment in the checkout
namespace).
Failure mode: HTTP 5xx ratio above threshold.
Scope: a single region (the alert payload carries the region).

Three elements present:

• Component. “orders-api” matches the service label on the alert. The deployment reference disambiguates for engineers unfamiliar with the label.
• Failure mode. “5xx ratio above threshold” is the verb the telemetry can confirm. The sentence is testable: a PromQL query against the alert’s expression either returns a value above 0.05 or it does not.
• Scope. “a single region” is the constraint that distinguishes this incident from a global failure of orders-api.

A what-failed statement that does not have all three is not a what-failed statement; it is a heading.

How to validate it

Three checks, in order. The first two are read-only against the runbook; the third is read-only against the alert and the platform.

# 1. Does the runbook opening name a component, a failure mode, and a scope?
#    A simple lint that flags runbooks without a "What failed" section.
grep -L '^## What failed' runbooks/checkout/*.md

Expected output: empty. A file listed here is a runbook missing its opening section. Open it and add the section before merging.

# 2. Does the component named in the runbook match a label on the alert?
#    This is a sanity check that the rule and the doc agree.
rule='OrdersApiHighErrorRate'
doc_component='orders-api'
curl -s "http://alertmanager:9093/api/v2/alerts?filter=alertname%3D%22${rule}%22" \
  | jq -r '.[].labels.service' \
  | sort -u \
  | grep -F "$doc_component"

Expected output:

orders-api

An empty result means the alert does not carry a service label, or the doc names the wrong component. Reconcile before merging.

# 3. Does the PromQL excerpt in the runbook return data for the affected region?
curl -sG http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=sum by (region) (rate(http_requests_total{service="orders-api", status=~"5.."}[5m])) / sum by (region) (rate(http_requests_total{service="orders-api"}[5m]))' \
  | jq '.data.result[] | select(.value[1] | tonumber > 0.05) | .metric.region'

Expected output during the incident: a single line with the affected region. An empty result means the telemetry does not support the what-failed statement. The doc is wrong, not the platform.

How it can fail

Six failure modes, each observable:

1. The opening names the symptom, not the component. Symptom: the first line says “5xx ratio is high” without naming the service. Cause: the runbook was written from the alert summary alone, which is a paraphrase, not a statement. Confirm by reading the first sentence and checking whether service appears.

2. The opening names the component but not the region. Symptom: the first line says “orders-api is failing” when the failure is scoped to eu-west-1. Cause: the rule’s region label was dropped from the doc. Confirm by comparing the alert payload to the runbook opening.

3. The opening uses a verb the telemetry cannot confirm. Symptom: the first line says “the system is broken.” Cause: the doc was written in prose, not from the alert expr. No PromQL or LogQL excerpt can confirm “broken.” The sentence has no testable claim.

4. The opening disagrees with the alert summary. Symptom: the alert summary says “5xx ratio above 5% in eu-west-1” but the runbook opening says “orders-api is slow.” Cause: the rule was edited without editing the doc. Confirm by inspecting ALERTS{alertname=...} and comparing the rendered summary to the runbook.

5. The opening is too long. Symptom: the first paragraph fills a screen. Cause: the author tried to combine the what-failed, impact, and mitigations into a single block. Confirm by counting sentences; a what-failed section has two to four sentences at most.

6. The opening references a region that is not a label on the alert. Symptom: the doc says “the failure is in the European region” but the alert label is eu-west-1. Cause: the doc uses prose where the rule uses labels. The label name must match.

How to troubleshoot it

In order:

1. Is there a “What failed” section at all? grep -L '^## What failed'. A runbook without the section has no opening contract.
2. Does the section name a component, a failure mode, and a scope? Read the first sentence. If any of the three is missing, rewrite the opening before the next incident.
3. Does the component name match a label on the alert? Compare the service label on the firing alert to the noun in the opening sentence. Disagreement is the bug.
4. Does the failure-mode verb match a PromQL or LogQL expression? If the verb is vague (“broken,” “weird,” “unstable”), the runbook is not yet anchored to telemetry.
5. Is the runbook opening the same in every alert region? A runbook whose opening says “in eu-west-1” but fires for us-east-1 is wrong. The opening must use the alert labels.

Security implications

The what-failed section is not security-sensitive by itself. It becomes sensitive when it includes internal hostnames, credentials, or customer identifiers. The discipline is to reference the alert’s labels rather than hard-coded values; the labels are non-sensitive by construction, and templating them through the alert payload avoids leaking customer identifiers into a doc that is shared more broadly than the alert.

A runbook whose opening includes an internal hostname or a production database connection string is a doc that should be moved behind the same access controls as the production system it describes. The simpler path is to template the value from the alert label.

Performance implications

The what-failed section is read once per incident. Performance implications are about the time-to-confirmation, not the doc size. A precise opening compresses the time-to-confirmation because the on-call can match the doc sentence to the alert payload in a single read. A vague opening extends it because the on-call has to reconcile the doc to the alert before proceeding.

The cost of writing a precise opening is one extra minute of authoring. The cost of a vague opening is measured in incident-seconds every time the alert fires. The trade-off strongly favours precision.

Production guidance

• Write the what-failed section after the alert rule exists. The rule is the source of truth for the alertname, labels, and summary. The doc derives from the rule.
• Name the component the same way the alert label names it. A mismatch between the doc and the label is a bug.
• Keep the section to two to four sentences. A precise paragraph compresses well; a vague paragraph balloons.
• Use the alert labels as the source of scope. “Single region” or “all regions” is a label-driven statement; “some users” is not.
• Run the opening lint in CI. A runbook without a ## What failed heading fails the check.

Verification

• What three elements must a what-failed section name?
• Why is “the system is broken” an unacceptable opening for a runbook?
• How does the what-failed section tie back to the alert rule’s labels?
• What is the symptom in Grafana when the doc opening names the symptom but not the component?

Quiz