Documentation from Telemetry

A 03:00 page arrives: OrdersApiHighErrorRate. The on-call opens the runbook, scrolls to “first steps,” and sees a sentence that reads “open the orders-api overview dashboard.” They open Grafana. There are four dashboards whose titles contain “orders-api.” None of them is the one the runbook meant. The runbook was written six months ago by an engineer who has since left; the dashboard was renamed twice in the interim. The on-call is now searching Grafana by trial and error at 03:00.

This is what a runbook that was not written from telemetry looks like under load. The failure shape is consistent: the runbook claims something exists, the platform has moved on, and the on-call pays for the drift.

The discipline that prevents the shape is documentation from telemetry. The doc describes what the platform actually exposes: the exact PromQL, the exact dashboard UID, the exact LogQL selector, the exact runbook URL. Every claim is anchored to a verifiable reference. The cost is a slightly less authoritative voice. The benefit is a doc that survives refactors and is useful at 03:00.

What it is

A runbook written from telemetry has three rules:

1. Every claim in the doc must be verifiable against a live reference. Not “approximately this query” but the actual PromQL, LogQL, or trace selector.
2. Every URL in the doc must resolve. Not “the dashboard, somewhere in Grafana” but a stable URL with the alert labels templated into the query string.
3. Every panel, query, or log stream the doc references must be owned. If the on-call opens it and finds wallpaper, the runbook has failed before the incident started.

The discipline is not glamorous. It produces docs that are shorter and less authoritative in tone. The trade-off is that they are correct, and they stay correct.

Why a sysadmin cares

Runbook docs that drift from the platform are a tax on every incident. The on-call opens the runbook, follows the first instruction, hits a dead end, and falls back to ad-hoc investigation. The runbook that was supposed to compress mean-time-to-resolve has extended it.

The cost is silent. Nobody notices when a runbook is correct; the on-call finds the panel, the query, the log line, and moves on. The cost appears the first time the runbook is wrong: the on-call opens the dashboard, sees nothing useful, and the investigation collapses to first principles in the worst possible hour.

The same discipline pays compound interest. Once the runbook references live queries and URLs, the doc becomes a side-effect of the platform. A change to the alert, the dashboard, or the metric set forces a change to the runbook. Drift is impossible when the doc is generated from, or cross-referenced with, the actual configuration files.

How it works

The platform already has a hierarchy of evidence. The doc mirrors it:

       Runbook section
              |
              v
       Telemetry claim
              |
              v
       Concrete reference (panel / query / log stream)
              |
              v
       Live system (Prometheus / Loki / Tempo / Grafana)

When the runbook says “CPU is above 80%,” it points at a specific PromQL expression (node_cpu_seconds_total{mode="idle"}), at the panel that shows it, and at the dashboard that hosts the panel. When it says “search for error logs,” it points at a LogQL selector that returns something on staging. When it says “trace a failed request,” it points at a Tempo trace query that resolves to a real trace ID.

The shape is the same whether the runbook is one page or twenty. Each claim is anchored. The most common shape in a production runbook is a header, a one-line symptom statement, two or three sections that each reference a panel or query, and a short list of mitigations. The structure compresses to roughly one screen.

How to configure it

The minimum viable shape is an alert rule whose annotations reference the runbook URL, and a runbook whose first paragraph is a stable URL back into the platform.

The alert rule:

groups:
  - name: orders-api.slo
    rules:
      - alert: OrdersApiHighErrorRate
        expr: |
          sum by (service, region) (
            rate(http_requests_total{service="orders-api", status=~"5.."}[5m])
          )
          /
          sum by (service, region) (
            rate(http_requests_total{service="orders-api"}[5m])
          )
          > 0.05
        for: 5m
        labels:
          severity: critical
          team: checkout
          service: orders-api
        annotations:
          summary: 'orders-api 5xx ratio above 5% in {{ $labels.region }}'
          runbook_url: 'https://runbooks.example.com/checkout/orders-api-5xx.html'
          dashboard_url: 'https://grafana.example.com/d/orders-api/orders-api-overview?var-region={{ $labels.region }}&from=now-1h&to=now'

Each annotation earns its place:

• summary is the one-line alert name rendered in Alertmanager and PagerDuty. It must fit on one line.
• runbook_url points at the runbook for this alert. The URL is static because the runbook covers the entire rule; alert labels are not needed in the path.
• dashboard_url is Go-templated on region so the dashboard opens already filtered to the affected region. The from and to query parameters set the time range to the last hour, which covers the typical for: window of 5 to 10 minutes.

The matching runbook excerpt, written from telemetry:

# orders-api 5xx ratio above 5%

## What failed

The orders-api service is returning HTTP 5xx responses at a ratio
above 5% over a rolling 5-minute window, in a single region.

## Telemetry to read first

Open the orders-api overview dashboard filtered to the region
from the alert payload. The "5xx ratio (5m)" panel shows the
underlying value. Confirm:

- The ratio is above 5% over the last 5 minutes.
- The rise coincides with a recent deploy, a dependency latency
  rise, or a database host resource pressure event.

The PromQL the alert is based on:

  sum by (service, region) (
    rate(http_requests_total{service="orders-api", status=~"5.."}[5m])
  )
  /
  sum by (service, region) (
    rate(http_requests_total{service="orders-api"}[5m])
  )

## Where to look next

- Structured logs filtered to service="orders-api" and level="error"
  in Loki.
- Traces filtered to status="error" and service="orders-api" in
  Tempo, then drill into the payment-svc and orders-db spans.

The doc references the same query the alert uses, the same dashboard, the same labels. A reader who pastes the PromQL into Grafana Explore sees the same series the alert fired on. A reader who opens the dashboard sees the same panels the alert’s author saw when the rule was written.

How to validate it

Three checks. The first is a static check against the doc; the second and third are read-only against the live platform.

# 1. Does every URL in the runbook resolve?
#    This is a CI step on the runbook repository.
grep -oE 'https?://[^ )]+' runbooks/checkout/orders-api-5xx.md \
  | sort -u \
  | xargs -I{} curl -sSI -o /dev/null -w '%{http_code} {}\n' {}

Expected output:

200 https://runbooks.example.com/checkout/orders-api-5xx.html
200 https://grafana.example.com/d/orders-api/orders-api-overview

A non-200 status means the URL in the doc does not match a live resource. Fix the doc, or fix the platform, before merging.

# 2. Does the PromQL excerpt return the same series the alert does?
curl -sG http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=sum by (service, region) (rate(http_requests_total{service="orders-api", status=~"5.."}[5m])) / sum by (service, region) (rate(http_requests_total{service="orders-api"}[5m]))' \
  | jq '.data.result | length'

A non-zero length confirms the query returns series. An empty result confirms the query is broken, the selector is wrong, or no traffic is hitting the selector at the moment; check against staging where traffic is reproducible.

# 3. Does the dashboard exist in Grafana?
curl -sI http://grafana:3000/api/dashboards/uid/orders-api \
  -H "Authorization: Bearer $GRAFANA_TOKEN" \
  | head -1

Expected output:

HTTP/1.1 200 OK

A 404 means the dashboard was renamed, deleted, or never imported. The runbook cannot fix a missing dashboard. The doc must be updated to point at the current UID.

How it can fail

Six failure modes, each observable in the platform:

1. The runbook references a panel that was renamed. Symptom: the on-call opens the dashboard, finds the renamed panel under a new title, and has to guess which panel the runbook meant. Cause: a dashboard refactor changed panel titles without updating the runbook. Confirm by diffing the dashboard JSON against the panel names in the doc.

2. The runbook references a metric that no longer exists. Symptom: the PromQL excerpt returns an empty result in Explore. Cause: the service was upgraded and the metric was renamed or removed. Confirm by running the query against Prometheus and inspecting the __name__ label.

3. The runbook URL is a 404. Symptom: the on-call clicks the runbook link in the alert payload and gets a 404. Cause: the runbook was moved, renamed, or deleted. Confirm by running the URL check above in CI.

4. The dashboard link lacks the alert labels. Symptom: the on-call opens the dashboard and sees every region rather than the affected one. Cause: the dashboard_url annotation was a static link rather than a Go-templated one. Confirm by inspecting the alert payload in Alertmanager and comparing the URL to the label set.

5. The runbook references a label that does not exist on the alert. Symptom: the Go template renders empty. Cause: the rule was edited and a label was dropped. Confirm by inspecting the alert payload and confirming the label set matches the runbook’s template variables.

6. The runbook exists but no alert links to it. Symptom: every alert has a dashboard_url annotation but no runbook_url. Cause: the discipline was not enforced on the rule. Confirm by grep-ing rule files for runbook_url.

How to troubleshoot it

In order:

1. Does the URL resolve? curl -sSI <url>. A non-2xx status means the resource is gone; the platform or the doc must be fixed.
2. Does the query return data? Run the PromQL excerpt in Grafana Explore against the same Prometheus the alert is based on. An empty result means the metric is gone or the selector is wrong.
3. Does the dashboard exist? Open it directly in Grafana. A 404 means the dashboard JSON is not in the platform; the runbook cannot fix this.
4. Are the labels consistent? Compare the alert’s labels to the runbook’s references. A label that does not exist on the alert cannot be used in the runbook.
5. Was the runbook changed recently? Diff against the previous version. A change to the rule file or the dashboard JSON is the most likely cause of drift.

Security implications

Runbook docs are not security-sensitive by themselves. They become sensitive when they include credentials, internal hostnames, or private URLs. The discipline is the same as for any other doc: do not paste credentials, do not hard-code internal endpoints that should be templated, and review the doc under the same access controls as the source code it describes.

URLs that embed credentials (a Grafana URL with ?token=... in the query string, for example) leak the credential into chat transcripts, alertmanager history, and ticketing systems the moment an alert fires. Use header-based auth where it is available, and template the credentials from a vault rather than the URL itself.

Performance implications

A doc that is “frozen” at the moment of writing is a liability. The cost of maintaining the discipline is small: a CI check that verifies URLs resolve and queries return data, run on every change to either the doc or the platform. The cost of not maintaining it shows up in incident duration.

The cost of running the validation against production is also small. URL checks and read-only PromQL queries do not modify state. If the runbook depends on metrics that are expensive to query (label-join explosions, full table scans), the validation step will surface that and the runbook can be edited to a cheaper query.

Production guidance

• Write the runbook after the alert and the dashboard exist, not before. The doc describes what is there.
• Use Go templates for every URL that depends on alert labels. A static URL is a future 404.
• Run the URL-and-query validation in CI. A doc that does not pass the check does not merge.
• Treat runbooks and rule files as a single change. Editing one without editing the other is the failure shape.

Verification

• Why is a runbook written in a meeting room a liability?
• What is the CI check that catches a stale runbook link?
• How is the runbook’s PromQL excerpt validated against the alert’s underlying expression?
• What is the failure mode when a dashboard_url annotation is a static link rather than a Go-templated one?

Quiz