Annotations: Runbook, Dashboard, Description

A page arrives at 03:00. The on-call reads the summary, follows the runbook link, opens the dashboard link, and within two minutes has the dashboard open to the right region and the runbook open to the right procedure. The alert was useful because three pieces of metadata arrived with it: the summary, the runbook URL, and the dashboard URL. None of those pieces are required by Prometheus. All of them are required by the on-call.

What it is

Annotations are key-value pairs on an alert rule, sibling to labels:. They ride with the alert through to Alertmanager and onward to every receiver. Their job is human-facing context: what the alert means, where to read more, what to do first. They are not used for routing; that is labels:. The two are deliberately separate.

The standard annotation keys:

• summary — one short line describing the symptom. Plain language. Read first.
• description — a paragraph or two. Includes the current value, the affected series, and any context the operator needs before opening a runbook.
• runbook_url — a stable URL to the operator-facing runbook. Used by Alertmanager receivers that surface a runbook link (PagerDuty, Opsgenie, Slack notifications).
• dashboard_url — a stable URL to the dashboard for the affected series, with template variables filled in.

Less common but useful:

• playbook_url — alternative name for runbook. Pick one; do not carry both.
• summary_template and description_template — names used in Grafana-managed alert rules to distinguish them from plain strings.
• impact, action — used in some shops to carry a one-line statement of user impact and a one-line statement of what to do.

Why a sysadmin cares

The on-call rota reads annotations under cognitive load, often half-awake. The annotation set is what turns the alert from a notification into an investigation. A rule that fires without annotations reaches the rota as a heartbeat: something is wrong, but not what, where, or how to start. The rotation pays the cost in minutes per page and in misroutes.

The cost is amplified by the lack of routing. Annotations do not route the alert; they only describe it. A missing annotation does not produce a visible failure; it produces an invisible slowdown.

How it works

Annotations are Go-template strings. Prometheus evaluates each annotation against the alert’s labels and the current sample value at the time of evaluation. The template language is the standard text/template with a small whitelist of functions (humanize, humanizeDuration, humanizePercentage, printf, pathPrefix, externalURL, match, value, query).

Three template variables are always available:

• {{ $value }} — the numeric value of the expr for the affected series at the evaluation time.
• {{ $labels }} — the map of labels attached to the alert. Access with {{ $labels.service }}, {{ $labels.region }}, and so on.
• \{\{ $externalLabels \}\} — the labels set in the global: external_labels: block of prometheus.yml. Used for environment markers like env: prod.

Template syntax in YAML:

annotations:
  summary: 'orders-api 5xx ratio above 5% in {{ $labels.region }}'
  description: |
    The orders-api service in region {{ $labels.region }} has
    returned a 5xx ratio of {{ $value | humanizePercentage }} over
    the last 5 minutes.
  runbook_url: 'https://runbooks.example.com/checkout/orders-api-5xx'

Multi-line annotations use YAML literal-block scalars (|). Single-line annotations can use plain scalars but should be quoted because the template delimiters look like YAML flow indicators.

How to configure it

A rule with the standard four annotations:

groups:
  - name: orders-api.slo
    rules:
      - alert: OrdersApiHighErrorRate
        expr: |
          sum by (service, region) (
            rate(http_requests_total{service="orders-api", status=~"5.."}[5m])
          )
          /
          sum by (service, region) (
            rate(http_requests_total{service="orders-api"}[5m])
          )
          > 0.05
        for: 5m
        keep_firing_for: 30m
        labels:
          severity: critical
          team: checkout
          service: orders-api
        annotations:
          summary: 'orders-api 5xx ratio above 5% in {{ $labels.region }}'
          description: |
            The orders-api service in region {{ $labels.region }} has
            returned a 5xx ratio of {{ $value | humanizePercentage }}
            over the last 5 minutes. Check the dependency-latency
            panel and the recent deploys feed before paging upstream.
          runbook_url: 'https://runbooks.example.com/checkout/orders-api-5xx'
          dashboard_url: 'https://grafana.example.com/d/orders-api/orders-api-overview?var-region={{ $labels.region }}'

Each annotation has a distinct job:

• summary is what shows up in the PagerDuty or Slack title. Keep it under 80 characters.
• description is what the on-call reads first. Include the current value, the affected label set, and a one-line pointer to the next step.
• runbook_url is the canonical runbook. The runbook should exist; an annotation that 404s is worse than no annotation.
• dashboard_url is a deep link to the dashboard with the variables filled in. The template variables (?var-region={{ $labels.region }}) are URL-unsafe characters in some setups; for safer URLs, use the dashboard’s share feature to generate a parameterised link and copy the parameter syntax.

A more advanced annotation that uses the whitelist functions:

annotations:
  summary: 'orders-api p99 above SLO for {{ $labels.region }}'
  description: |
    orders-api region {{ $labels.region }} has a p99 latency of
    {{ $value | humanizeDuration }} over the last 5 minutes,
    breaching the SLO budget of 300ms. Check upstream latency
    and the connection-pool panel.
  runbook_url: 'https://runbooks.example.com/checkout/orders-api-latency'

humanizeDuration formats a number of seconds as a human duration. humanizePercentage formats a ratio as a percentage. Both functions are part of the Prometheus whitelist and cannot be replaced with custom Go-template functions.

How to validate it

Three checks. The first is static; the second and third are live.

# 1. Static check: does the rule parse, and do the templates parse?
promtool check rules /etc/prometheus/rules/orders-api.yml

Expected output:

Checking /etc/prometheus/rules/orders-api.yml
  SUCCESS: found 1 rules, 1 alerts

A template parse error (for example {{ $service }} instead of {{ $labels.service }}) is reported as a parse error and the rule does not load. Fix the template and rerun.

# 2. Inspect the rendered annotations on a live alert.
curl -s http://prometheus:9090/api/v1/alerts \
  | jq '.data.alerts[]
        | select(.labels.alertname == "OrdersApiHighErrorRate")
        | .annotations'

Expected output:

{
  "summary": "orders-api 5xx ratio above 5% in eu-west-1",
  "description": "The orders-api service in region eu-west-1 has returned a 5xx ratio of 6.2% over the last 5 minutes. Check the dependency-latency panel and the recent deploys feed before paging upstream.",
  "runbook_url": "https://runbooks.example.com/checkout/orders-api-5xx",
  "dashboard_url": "https://grafana.example.com/d/orders-api/orders-api-overview?var-region=eu-west-1"
}

If summary reads orders-api 5xx ratio above 5% in {{ $labels.region }} literally, the template did not render. Common causes: a typo in the variable name ($lables instead of $labels), or the rule did not reload after a template edit.

# 3. Inspect the annotations from Alertmanager's side.
curl -s http://alertmanager:9093/api/v2/alerts \
  | jq '.[] | select(.labels.alertname == "OrdersApiHighErrorRate") | .annotations'

Expected output: identical to step 2. Alertmanager does not re-render; what Prometheus rendered is what the receiver sees.

How it can fail

Six failure modes:

1. Empty summary. Symptom: the PagerDuty title is blank. Cause: the template parsed but $labels.region is empty on this rule (the rule’s expr does not produce a region label). Confirm by inspecting the result series in Grafana Explore.

2. {{ $value }} is meaningless. Symptom: the description reads current value 1 for a ratio that should be 6.2%. Cause: the expr was a bool (the comparison produced 0 or

   1. instead of a numeric ratio. Return the ratio in the expr and put the comparison in the expr: of the rule using bool-style aggregation, or split into a recording rule.

3. Runbook URL 404s. Symptom: on-call clicks the runbook link and gets a 404. Cause: the runbook was renamed or archived, or the URL was hard-coded without label interpolation. Add a redirect at the old URL or fix the template.

4. Dashboard URL with literal {{ }} characters. Symptom: Grafana opens a dashboard with var-region={{ $labels.region }} in the URL. Cause: the template did not render because the annotation value is itself a URL with embedded braces. Confirm by inspecting the live alert annotations; if the template is missing, fix the surrounding quotes.

5. Multi-line annotation not preserved. Symptom: the description arrives as a single line with \n characters. Cause: the YAML literal-block scalar (|) was not used; a folded scalar (>) was used instead. Use | for multi-line annotations.

6. External label not visible in the template. Symptom: the template references {{ $externalLabels.env }} but renders as empty. Cause: external_labels: is not set in prometheus.yml, or the key is missing. Confirm by inspecting up{job="prometheus"}; it carries the external labels.

How to troubleshoot it

In order:

1. Inspect the rendered annotations. /api/v1/alerts and /api/v1/rules both expose the rendered annotation map. Empty strings or literal {{ }} text means the template did not render.
2. Inspect the labels on the alert. The same endpoints show the alert’s labels. If a template references {{ $labels.region }} but region is not in the labels, the render is empty.
3. Inspect the rule’s expr result. In Grafana Explore, compute the expr with Instant evaluated. Confirm the value is numeric and the label set includes the keys the template uses.
4. Re-run promtool check rules. It parses the templates at load time and reports parse errors. If the rule loaded, the templates parsed.
5. Reload after edit. Templates are not live-reloaded by Prometheus; only the rule files are. After editing a template, SIGHUP Prometheus to pick up the change.

Security implications

Annotations are user-visible. Every URL in an annotation ends up in a chat transcript, a ticketing system, or a PagerDuty notification. Treat them as content:

• Do not embed credentials in URLs. PagerDuty service keys in runbook URLs are a real-world anti-pattern.
• Confirm that the runbook URL points at a trusted, internal source. A runbook URL on a third-party wiki can leak the existence and nature of an incident.
• Be careful with templating. {{ $value }} for a metric that contains a token or a key will end up in the transcript.

Annotations do not affect routing, so a templating bug cannot send an alert to the wrong team. The cost is human time, not data leakage.

Performance implications

Annotations are templates and are evaluated at alert creation, not at every evaluation. A complex template (for example one that calls {{ query "up{job=...") }}) costs CPU at the moment the alert fires, not at every tick. The cost is bounded because the alert fires once per state transition, not once per evaluation.

Still, keep templates simple. The query function in particular can be expensive; prefer computing the value in the expr and rendering it with {{ $value }}.

Production guidance

• Make the four standard annotations mandatory in the rule-review checklist: summary, description, runbook_url, dashboard_url.
• Keep summary under 80 characters so it survives Slack truncation and SMS gateways.
• Use humanizeDuration, humanizePercentage, and humanize in description for numeric values; raw floats are unreadable on a phone.
• Quote annotation values that contain template delimiters. Single-quoted YAML scalars ('text {{ $x }} text') are the safest form; double-quoted scalars interpret escape sequences.

Verification

• What four annotation keys should every paging alert carry?
• What is the difference between {{ $value }} and {{ $labels.x }} in an annotation template?
• What is the labelling-versus-documenting split, and how do you keep the two clean?
• What does Alertmanager do with template rendering, and what does it pass through unchanged?

Quiz