Actionable Alerts

At 03:00 the on-call engineer receives a PagerDuty notification. The notification reads “High CPU.” The engineer wakes, opens Grafana, clicks through four dashboards, and finally concludes that the spike is benign. Twenty minutes of sleep have been spent investigating a metric the alert should have explained in a sentence.

The discipline of actionable alerts is that the alert itself should be the first step of the investigation, not a notification that triggers one. The alert name should narrate the action. The summary should name the impact. The description should provide the evidence and the remediation steps. The annotations should link to the dashboard and the runbook. The labels should name the owner.

What it is

An actionable alert is one that includes every field the on-call engineer needs to act on the page without first loading context from somewhere else. The fields are: a title that names the action, a summary that names the impact, a description that provides evidence and remediation steps, a runbook URL that documents the recovery procedure, a dashboard URL that allows the engineer to drill into the cause, an owner label that identifies the team, and a severity label that determines routing.

A rule that is missing any one of these fields is decoration. A rule that has all of them is operational.

Why a sysadmin cares

The on-call engineer at 03:00 is not in a state to load context. Waking, context-switching, and orienting consume the SLO mitigation window. The alert is the only context the engineer receives without effort. If the alert does not provide it, the engineer will either act on incomplete information (and possibly make the incident worse) or spend ten minutes loading context (and consume the mitigation window before acting).

How it works

The alert-to-incident pipeline moves through four stages, and each stage has a specific payload:

  Prometheus firing state
       |
       |  labels: severity, team, service, slo
       |  annotations: summary, description,
       |               runbook_url, dashboard_url
       v
  Alertmanager route tree
       |
       |  match severity=page -> pagerduty receiver
       |  match severity=ticket -> jira receiver
       |  template renders summary + description
       |  into the notification body
       v
  PagerDuty / OpsGenie / Slack
       |
       |  on-call acks, opens dashboard, follows runbook
       v
  Incident: mitigation, post-incident review

Each stage carries information forward. The Prometheus rule sets the labels and annotations. The Alertmanager route decides the receiver. The template renders the body. The receiver delivers the body. The on-call engineer reads the body and acts.

How to configure it

A rule that meets the actionable standard:

groups:
- name: checkout.rules
  interval: 30s
  rules:
  - alert: CheckoutErrorBudgetBurn
    expr: |
      (
        sum(rate(http_requests_total{job="checkout",status=~"5.."}[5m]))
        /
        sum(rate(http_requests_total{job="checkout"}[5m]))
      ) > 0.02
    for: 5m
    labels:
      severity: page
      team: payments
      service: checkout
      region: '{{ $labels.region }}'
      slo: availability
      pager: payments-oncall
    annotations:
      summary: 'Roll back checkout deploy or scale payments-svc in {{ $labels.region }}'
      description: |
        Checkout 5xx rate in {{ $labels.region }} is
        {{ $value | humanizePercentage }} (threshold 2%).
        Error budget will exhaust in roughly two hours at
        current burn.

        Likely causes:
          - recent deploy introduced a regression
          - payments-svc dependency is timing out
          - database connection pool exhausted

        Steps:
          1. Open the dashboard and confirm the burn rate.
          2. Check the deploy log for the last release.
          3. If deploy, roll back via `kubectl rollout undo`.
          4. If dependency, scale payments-svc to 2x.
          5. If pool, restart db-proxy pods.
      runbook_url: 'https://runbooks.example.com/checkout/5xx'
      dashboard_url: 'https://grafana.example.com/d/checkout?var-region={{ $labels.region }}'

Reading the rule line by line:

• Title (alert: CheckoutErrorBudgetBurn) reads as a sentence that names the metric and the service.
• summary narrates the action (“Roll back or scale”) and includes the region label. The on-call engineer reads this before anything else.
• description lists likely causes and concrete steps. The steps are ordered; the on-call engineer can execute them in order without re-reading the rule.
• runbook_url points to the team’s runbook repository. The on-call engineer clicks through for the deeper procedure.
• dashboard_url includes the region label so the dashboard opens pre-filtered to the affected region.
• severity: page routes to PagerDuty.
• team: payments and pager: payments-oncall identify the owner and the on-call rotation.

The Alertmanager template that renders this payload:

# alertmanager.yml (excerpt)
templates:
- '/etc/alertmanager/templates/default.tmpl'
receivers:
- name: pagerduty
  pagerduty_configs:
  - service_key: '<redacted>'
    description: '{{ .CommonAnnotations.summary }'
    details:
      firing: '{{ .Alerts.Firing }'
      runbook: '{{ .CommonAnnotations.runbook_url }'
      dashboard: '{{ .CommonAnnotations.dashboard_url }'

The default.tmpl file:

{{ define "pagerduty.default" }}
{{ range .Alerts }}
*{{ .Annotations.summary }}*
{{ .Annotations.description }}
Runbook: {{ .Annotations.runbook_url }}
Dashboard: {{ .Annotations.dashboard_url }}
Severity: {{ .Labels.severity }}
Owner: {{ .Labels.team }}
{{ end }}
{{ end }}

The PagerDuty incident receives the rendered Markdown body. The on-call engineer sees the summary as the title, the description as the body, and the runbook / dashboard as clickable links.

How to validate it

Validate the rule and the Alertmanager template:

# SEVERITY: READ-ONLY
promtool check rules /etc/prometheus/rules/checkout.rules.yml
amtool check-config /etc/alertmanager/alertmanager.yml

Render the Alertmanager template against a synthetic alert to verify the payload is complete:

# SEVERITY: READ-ONLY
amtool alert render-template \
  --config.file=/etc/alertmanager/alertmanager.yml \
  --template=default.tmpl \
  alertname=CheckoutErrorBudgetBurn \
  severity=page team=payments service=checkout region=eu-west-1 \
  summary='Roll back checkout deploy or scale payments-svc in eu-west-1'

Expected output ends with the runbook URL, the dashboard URL, and the owner label visible in the rendered body. If any field is missing, the template is incomplete.

Trigger a synthetic firing and inspect the PagerDuty payload:

# SEVERITY: READ-ONLY (test alert; not delivered)
amtool alert add \
  --alertmanager.url=http://alertmanager:9093 \
  --config.file=/etc/alertmanager/alertmanager.yml \
  alertname=CheckoutErrorBudgetBurn \
  severity=page team=payments service=checkout region=eu-west-1 \
  summary='Synthetic test'

Then verify in the Alertmanager UI under Status -> Active Alerts that all labels and annotations are present.

How it can fail

Five failure modes appear when rules are missing fields:

1. Title names the metric, not the action. “HighCPUCheckout” instead of “Roll back checkout deploy or scale.” The on-call has to infer the action from the dashboard. Symptom: MTTA above five minutes for known issues.
2. No runbook URL. The on-call has to search the runbook repository by service name. Symptom: MTTA above ten minutes for known issues; the runbook may not be findable.
3. No dashboard URL. The on-call has to construct the dashboard URL by hand. Symptom: the engineer opens the wrong dashboard or a generic overview.
4. Description is a copy of the summary. The engineer reads the description and learns nothing new. Symptom: the alert is treated as a notification rather than an investigation starting point.
5. Annotations interpolate sensitive labels. The description includes a user email or a request URL. Symptom: the page leaks PII to the receiver channel. The fix is to interpolate only the labels the on-call needs to act.

How to troubleshoot it

When a page is delivered without a field the on-call needs, the order is:

1. Inspect the alert in the Alertmanager UI. Compare the labels and annotations to the rule file. Identify the missing field.
2. If the field is missing from the rule, add it to the rule file and reload Prometheus.
3. If the field is present in the rule but missing from the notification, the Alertmanager template is dropping it. Inspect templates/*.tmpl; ensure the annotation is referenced by name.
4. If the receiver integration is dropping the field (e.g. PagerDuty has a custom payload mapper that strips annotations), update the integration config and re-test.
5. Add a unit test that pins the field set per rule. The promtool test rules framework can assert that the rendered template includes the runbook URL, the dashboard URL, and the owner label.

Security implications

The summary, description, and annotation fields are delivered to the receiver channel. If the receiver is a phone, the field may appear on a locked screen. If the receiver is Slack, the field appears in a channel that may be read by a broader audience than the on-call rotation. Annotations must not interpolate user-identifying labels (user_id, email, request_id, api_key). The rule must interpolate only the operational labels (service, region, instance, severity, team).

The runbook URL is internal. If the runbook repository is not authenticated, the URL leaks the internal topology. The Alertmanager receiver integration must enforce authentication on the runbook link.

Performance implications

The template rendering is in-memory and bounded by the size of the annotations. Annotations are typically a few hundred bytes; the rendering cost is negligible. The performance cost of incomplete annotations is the operational cost of the on-call loading context; this is the same cost as alert fatigue, but per-incident rather than over time.

Long descriptions (above 4 KB) can be truncated by some receiver integrations (Slack, PagerDuty). Keep the description focused; link out to the runbook for the longer procedure.

Production guidance

• The summary is the most important field. It is the only thing the on-call reads in the first five seconds. The summary must name the action.
• The description must include likely causes and ordered steps. The on-call can execute the steps without re-reading the rule.
• The runbook URL and dashboard URL must be present on every page. The on-call must not have to construct them by hand.
• The owner label (team) must be present on every alert. The Alertmanager route must match on it.
• Templates must be unit-tested. A template that drops a field in production is invisible until the first page without that field.

Verification

You should now be able to answer:

• What seven fields must a page-worthy alert include, and which one is the single most important?
• How does the Alertmanager template render the Prometheus annotations into the receiver body, and which stage is most likely to drop a field?
• Why must the description list likely causes and ordered steps, rather than a copy of the summary?
• What is the privacy rule for label interpolation in annotations, and which labels are forbidden?

Quiz