Runbook Link

A page arrives at 03:00. The on-call reads the alert summary: “orders-api 5xx ratio above 5% in eu-west-1.” The alert payload carries a dashboard_url annotation. They open the dashboard. The 5xx panel confirms the alert. The on-call now needs to know what to do. The alert payload does not carry a runbook_url. They open the team wiki. They search for “orders-api 5xx.” The search returns seventeen results. The first three are post-mortem documents from past incidents. The fourth is a draft that was never published. The fifth is the right one, but the title is “orders-api availability troubleshooting” and the search query does not match. The on-call has now spent three minutes looking for the runbook they should have been reading.

A runbook link is the alert’s claim that a single URL, when clicked, lands the on-call on the runbook for the alert that fired. The discipline that produces that URL is small: the link is a stable URL embedded in the alert’s runbook_url annotation, and the runbook page covers the entire alert’s failure shape.

What it is

A runbook link is a URL with three properties:

1. The path identifies the runbook page by a stable URL. The URL does not change when the runbook is renamed; it changes only when the runbook is moved or deleted.
2. The scope is the entire alert rule. The link does not change per region, per instance, or per label. The runbook is generic across the scope the alert covers.
3. The destination is the runbook opening, not a deep section. The on-call reads the runbook top-to-bottom the first time. They anchor on the what-failed and impact sections and proceed.

A runbook link that is templated on alert labels is a bug: the runbook is generic; the alert is the entry point.

Why a sysadmin cares

The runbook link and the dashboard link do different work. The dashboard link lands the on-call on a panel already filtered to the incident, with the right time range, ready to look. The runbook link lands the on-call on the procedure: what failed, what the impact is, what to do. Both links are necessary. A dashboard without a runbook tells the on-call what is wrong but not what to do. A runbook without a dashboard tells the on-call what to do but not whether the alert is the one the runbook describes.

The cost of a missing runbook link is paid every incident. The on-call hunts for the right wiki page under time pressure. The first match in the wiki is rarely the right one. The cost of a correct link is one extra annotation in the rule file.

How it works

The runbook link is a stable URL embedded in the alert’s runbook_url annotation:

  Alert payload (machine-readable)
  --------------------------------
  alertname:    OrdersApiHighErrorRate
  service:      orders-api
  region:       eu-west-1
  severity:     critical
  summary:      orders-api 5xx ratio above 5% in eu-west-1
  runbook_url:  https://runbooks.example.com/checkout/orders-api-5xx.html
  dashboard_url: https://grafana.example.com/d/orders-api/orders-api-overview?var-region=eu-west-1
              |
              v
  Runbook page (human-readable)
  -----------------------------
  orders-api 5xx ratio above 5%

  What failed: ...
  Impact: ...
  Telemetry to read first: ...
  Where to look next: ...
  Mitigations: ...

The mapping is mechanical:

• runbook_url is a static URL. The same URL works for every region, every service, every label set the alert covers.
• The runbook page covers the entire alert rule. The what-failed section uses {{ $labels.region }} to scope the prose; the rest of the runbook is generic across regions.
• The alertmanager route does not touch the runbook_url annotation. Alertmanager routes by label, not by annotation; the runbook link is metadata that rides with the alert.

A runbook link that is per-region (templated on region) is a bug: the rule generates a different URL for every alert instance, and the team has to maintain a separate runbook page per region.

How to configure it

The alert rule, with a runbook_url annotation:

groups:
  - name: orders-api.slo
    rules:
      - alert: OrdersApiHighErrorRate
        expr: |
          sum by (service, region) (
            rate(http_requests_total{service="orders-api", status=~"5.."}[5m])
          )
          /
          sum by (service, region) (
            rate(http_requests_total{service="orders-api"}[5m])
          )
          > 0.05
        for: 5m
        labels:
          severity: critical
          team: checkout
          service: orders-api
        annotations:
          summary: 'orders-api 5xx ratio above 5% in {{ $labels.region }}'
          description: |
            The orders-api service in region {{ $labels.region }}
            has returned a 5xx ratio above 5% over the last
            5 minutes. Current ratio: {{ $value | humanizePercentage }}.
          runbook_url: 'https://runbooks.example.com/checkout/orders-api-5xx.html'
          dashboard_url: 'https://grafana.example.com/d/orders-api/orders-api-overview?var-region={{ $labels.region }}&from=now-1h&to=now'

The matching runbook page:

# orders-api 5xx ratio above 5%

Alert: OrdersApiHighErrorRate
Severity: critical
Owner: checkout team
Dashboard: https://grafana.example.com/d/orders-api/orders-api-overview

## What failed

The orders-api service in {{ $labels.region }} is returning HTTP
5xx responses at a ratio above 5% over a rolling 5-minute window.

## Impact

[See impact section in the runbook template.]

## Telemetry to read first

[See telemetry section, which references the dashboard link
above and the PromQL excerpt the alert is based on.]

## Mitigations

[See mitigations section, ordered by likelihood.]

Each element earns its place:

• runbook_url is static. The same URL works for every region.
• The runbook page is generic across regions; only the what-failed section uses {{ $labels.region }} in prose.
• The alertmanager route does not touch the annotation; the URL rides with the alert from Prometheus to the receiver.

How to validate it

Three checks, in order. The first is a static check against the rule files; the second is read-only against the runbook server; the third is read-only against the live alert payload.

# 1. Does every alert rule with a runbook_url annotation have a stable URL?
#    A simple lint that flags templated runbook URLs.
grep -rE 'runbook_url.*\{\{' /etc/prometheus/rules/ \
  | grep -v 'runbook_url.*{{ \$labels' \
  || echo 'no templated runbook URLs found'

A rule whose runbook_url is templated on alert labels generates a different URL per firing series. The discipline is to keep the URL static. The grep above flags any URL that uses Go template variables.

# 2. Does the runbook URL resolve?
rule='OrdersApiHighErrorRate'
url=$(curl -s "http://alertmanager:9093/api/v2/alerts?filter=alertname%3D%22${rule}%22" \
  | jq -r '.[0].annotations.runbook_url')
curl -sI "$url" | head -1

Expected output:

HTTP/1.1 200 OK

A non-2xx response means the URL is stale. The runbook page must be restored, or the rule must be updated to the new URL.

# 3. Does the rule file contain the runbook_url annotation?
grep -F 'runbook_url' /etc/prometheus/rules/orders-api.yml

Expected output:

          runbook_url: 'https://runbooks.example.com/checkout/orders-api-5xx.html'

A missing annotation means the alert fires without a link to the procedure. The on-call has to find the runbook themselves.

How it can fail

Six failure modes, each observable:

1. The runbook_url annotation is missing. Symptom: the alert fires, the payload has dashboard_url but no runbook_url. Cause: the rule was authored without the annotation. Confirm by inspecting the alert payload.

2. The runbook URL is templated on alert labels. Symptom: the rendered URL contains eu-west-1 in the path. Cause: the author assumed the URL needs to carry the region. The runbook is generic across regions; the URL should be too. Confirm by inspecting the rendered URL.

3. The runbook URL is a 404. Symptom: the on-call clicks the link and gets a 404. Cause: the runbook was moved, renamed, or deleted. Confirm by hitting the URL directly.

4. The runbook URL is for a draft or unpublished page. Symptom: the URL resolves but the page is empty or marked “draft.” Cause: the author linked to the wrong page. Confirm by inspecting the page content.

5. The runbook URL is to the wiki root, not a specific page. Symptom: the URL is https://runbooks.example.com/ and the on-call lands on the wiki home page. Cause: the author copy-pasted a partial URL. Confirm by inspecting the URL path.

6. The runbook URL embeds credentials. Symptom: the URL contains ?token=... or ?api_key=.... Cause: the runbook server is behind an auth layer that uses URL parameters. Confirm by inspecting the URL.

How to troubleshoot it

In order:

1. Is there a runbook_url annotation? grep runbook_url /etc/prometheus/rules/. A missing annotation means the alert has no link.
2. Does the URL resolve? curl -sI <url>. A 404 means the page is gone.
3. Is the URL templated on alert labels? Inspect the URL for {{ $labels.X }}. The URL should be static.
4. Does the rendered URL match a published runbook? Inspect the page content. A draft or empty page is a bug.
5. Does the rule file match the alertmanager payload? Compare the rule file’s annotation to the rendered annotation on the firing alert.

Security implications

The runbook link is a URL. It is not security-sensitive by itself. It becomes sensitive when it embeds credentials in the query string or when it points at a runbook page that exposes internal hostnames, customer identifiers, or production credentials.

The discipline is to use the on-call’s session for auth, not to embed credentials in the URL. A link with ?token=... leaks the token into chat transcripts, alert payloads, and ticketing systems the moment the alert fires. If the runbook server requires auth, use header-based auth via a service account.

A runbook whose content includes credentials, internal hostnames, or production endpoints is a doc that should be moved behind the same access controls as the production system it describes. The simpler path is to keep the runbook generic and template sensitive values from the alert labels rather than hard-coding them in the prose.

Performance implications

The runbook link is clicked once per incident. Performance implications are about the time-to-procedure, not the link size. A precise runbook link compresses the time-to-procedure because the on-call lands on the runbook opening with no hunting required. A missing or broken runbook link extends it because the on-call has to find the runbook themselves.

The cost of adding a runbook_url annotation is one extra line in the rule file. The cost of a missing annotation is paid every incident.

Production guidance

• Add a runbook_url annotation to every alert rule that pages the on-call. A rule without a runbook URL is a rule that pages and abandons.
• Keep the URL static. The runbook is generic across the alert’s scope; the URL is too.
• Validate the URL in CI. A URL that 404s or that points at a draft page fails the check.
• Maintain the runbook URL and the alert in the same repository. A rename in the runbook is a rename in the rule.

Verification

• What three properties must a runbook link carry?
• Why is a runbook link static rather than templated on alert labels?
• How is a missing runbook_url annotation detected in CI?
• What is the symptom in Alertmanager when the runbook URL is a 404?

Quiz