SLO Alert Templates

A team owns twelve services and runs SLOs on each. The first SLO was hand-rolled — six recording rules, two alert rules, and a dashboard pasted from a vendor blog. The second SLO looked similar but used a different recording-rule naming convention. The third SLO used a different burn-rate threshold because the engineer who wrote it had read a different blog post. By the fourth SLO the team had four different conventions; by the eighth, the on-call rotation could not recognise an alert at sight.

Templates are the cure. One convention, one rule shape, one dashboard panel per SLO. The hand-rolled first SLO was the right amount of work; the tenth should be the same amount of work.

What it is

An SLO alert template is a complete, drop-in package for one service: the SLI recording rule, the error budget percentage metric, the four burn-rate alerts (page-worthy fast burn, page-worthy slow burn, ticket-worthy fast burn, ticket-worthy slow burn), and the Grafana dashboard panels that read the recording rules. The template is parametrised on the service name, the SLI expression, and the SLO target so the same file drives the SLO package for any service.

Service SLO package
  +-- 1 SLI recording rule   (the metric)
  +-- 1 budget % rule        (consumed vs window)
  +-- 4 burn-rate alerts     (page + ticket, fast + slow)
  +-- 1 Grafana dashboard    (the panel)
  +-- 1 Alertmanager route   (the receiver)

Each piece has a single convention. The recording rule is named slo:<service>:<type>:<aggregation>. The alert is named <Service>SLO<BurnKind> and carries a severity: page|ticket|policy-* label. The dashboard panel is named <Service> SLO — <Description>. The conventions are what on-call engineers recognise at sight.

Why a sysadmin cares

The on-call rotation does not have time to re-learn each service. A team with twelve SLOs and four different rule shapes produces a rotation that cannot recognise a real incident by reading the alert. The team that adopts templates ships alerts the rotation recognises at a glance — the format, the labels, and the runbook link are all standard.

The category of incident this prevents is “alert that nobody recognises”: a real alert that arrives in the right severity but with the wrong labels, routed to the wrong queue, with a runbook that does not exist for this service. The on-call engineer treats it as malformed noise. By the time the incident is identified as a real SLO violation, the budget is gone.

How it works

The template has six logical artefacts. Each has a single convention; the conventions compose.

slo:<service>:errors:ratio_rate<window>   recording rule
slo:<service>:budget_30d_consumed_pct     recording rule
<Service>SLO<BurnKind>                    alert rule
slo:<service>                              dashboard row
<slo_label>                                Alertmanager route

The template is parameterised on three values:

• service: the service name as it appears in Prometheus labels (e.g. orders).
• sli_expression: the PromQL that produces the good-events / total-events ratio.
• slo_target: the target as a decimal in [0, 1] (e.g. 0.999 for 99.9%).

For a templating layer, the team can use Sloth, Jsonnet with the prometheus-jsonnet-lib, or a manual sed substitution. Sloth is the simplest path because it generates everything from a single spec file.

How to configure it

The canonical template, rendered for the orders service. The template is the same shape for every service; only the SLI expression and the SLO target change.

# /etc/prometheus/rules/slo-orders.yml
# Canonical SLO package for the orders service.
# SLO target: 99.9% availability (1 - SLO = 0.001).
groups:
  # Recording rules: SLI, over each burn-rate window.
  - name: slo.orders.recording
    interval: 30s
    rules:
      - record: slo:orders:errors:ratio_rate5m
        expr: |
          sum(rate(http_requests_total{
            service="orders", code=~"5.."
          }[5m]))
          /
          sum(rate(http_requests_total{
            service="orders"
          }[5m]))

      - record: slo:orders:errors:ratio_rate30m
        expr: |
          sum(rate(http_requests_total{
            service="orders", code=~"5.."
          }[30m]))
          /
          sum(rate(http_requests_total{
            service="orders"
          }[30m]))

      - record: slo:orders:errors:ratio_rate1h
        expr: |
          sum(rate(http_requests_total{
            service="orders", code=~"5.."
          }[1h]))
          /
          sum(rate(http_requests_total{
            service="orders"
          }[1h]))

      - record: slo:orders:errors:ratio_rate6h
        expr: |
          sum(rate(http_requests_total{
            service="orders", code=~"5.."
          }[6h]))
          /
          sum(rate(http_requests_total{
            service="orders"
          }[6h]))

      - record: slo:orders:errors:ratio_rate24h
        expr: |
          sum(rate(http_requests_total{
            service="orders", code=~"5.."
          }[24h]))
          /
          sum(rate(http_requests_total{
            service="orders"
          }[24h]))

      - record: slo:orders:errors:ratio_rate72h
        expr: |
          sum(rate(http_requests_total{
            service="orders", code=~"5.."
          }[72h]))
          /
          sum(rate(http_requests_total{
            service="orders"
          }[72h]))

      # Budget consumed over the 30-day window.
      - record: slo:orders:errors:budget_30d_consumed_pct
        expr: |
          (
            sum(increase(http_requests_total{
              service="orders", code=~"5.."
            }[30d]))
            /
            (0.001 * sum(increase(http_requests_total{
              service="orders"
            }[30d])))
          ) * 100

  # Alert rules: page, ticket, policy.
  - name: slo.orders.alerts
    interval: 30s
    rules:
      # Page-worthy fast burn: 1h and 6h both over threshold.
      - alert: OrdersSLOFastBurnPage
        expr: |
          (
            slo:orders:errors:ratio_rate1h > (14.4 * 0.001)
            and
            slo:orders:errors:ratio_rate6h > (6 * 0.001)
          )
        for: 2m
        labels:
          severity: page
          slo: orders-availability
          pager: pagerduty-orders
        annotations:
          summary: 'Orders SLO: fast burn (page)'
          description: |
            Both 1h and 6h windows over threshold.
            Budget would be exhausted within 2-5 days.
          runbook_url: 'https://runbooks/slo/orders-fast-burn'

      # Ticket-worthy slow burn: 24h and 72h both over threshold.
      - alert: OrdersSLOSlowBurnTicket
        expr: |
          (
            slo:orders:errors:ratio_rate24h > (3 * 0.001)
            and
            slo:orders:errors:ratio_rate72h > (1 * 0.001)
          )
        for: 1h
        labels:
          severity: ticket
          slo: orders-availability
          ticket_queue: slo-quarterly-review
        annotations:
          summary: 'Orders SLO: slow burn (ticket)'
          description: |
            Slow cumulative burn across 24h and 72h windows.
            Budget would be exhausted within the 30-day window.
          runbook_url: 'https://runbooks/slo/orders-slow-burn'

      # Policy-threshold: 25% budget consumed.
      - alert: OrdersBudgetPolicy25
        expr: slo:orders:errors:budget_30d_consumed_pct > 25
        for: 1h
        labels:
          severity: policy-slowdown
          slo: orders-availability
        annotations:
          summary: 'Orders budget: 25% consumed'
          description: 'Slowdown policy applies.'
          runbook_url: 'https://runbooks/slo/orders-policy'

      # Policy-threshold: 100% budget consumed.
      - alert: OrdersBudgetPolicy100
        expr: slo:orders:errors:budget_30d_consumed_pct > 100
        for: 30m
        labels:
          severity: policy-freeze
          slo: orders-availability
        annotations:
          summary: 'Orders budget: 100% consumed'
          description: 'Release freeze policy applies.'
          runbook_url: 'https://runbooks/slo/orders-policy'

The Grafana dashboard row that matches. Each panel reads the recording rule directly — no PromQL at panel time, no scan of raw counters.

{
  "title": "Orders SLO",
  "panels": [
    {
      "title": "Error budget remaining (30d)",
      "type": "gauge",
      "targets": [{
        "expr": "100 - slo:orders:errors:budget_30d_consumed_pct",
        "legendFormat": "remaining"
      }],
      "fieldConfig": {
        "defaults": {
          "thresholds": {
            "mode": "absolute",
            "steps": [
              { "color": "red",    "value": null },
              { "color": "yellow", "value": 25 },
              { "color": "green",  "value": 50 }
            ]
          }
        }
      }
    },
    {
      "title": "Burn rate (multi-window)",
      "type": "timeseries",
      "targets": [
        {
          "expr": "slo:orders:errors:ratio_rate1h / 0.001",
          "legendFormat": "1h"
        },
        {
          "expr": "slo:orders:errors:ratio_rate6h / 0.001",
          "legendFormat": "6h"
        },
        {
          "expr": "slo:orders:errors:ratio_rate24h / 0.001",
          "legendFormat": "24h"
        },
        {
          "expr": "slo:orders:errors:ratio_rate72h / 0.001",
          "legendFormat": "72h"
        }
      ],
      "fieldConfig": {
        "defaults": {
          "thresholds": {
            "mode": "absolute",
            "steps": [
              { "color": "yellow", "value": 1 },
              { "color": "red",    "value": 6 }
            ]
          }
        }
      }
    },
    {
      "title": "SLI over 30d",
      "type": "stat",
      "targets": [{
        "expr": "1 - (sum(increase(http_requests_total{service=\"orders\",code=~\"5xx\"}[30d])) / sum(increase(http_requests_total{service=\"orders\"}[30d])))",
        "legendFormat": "availability"
      }],
      "fieldConfig": {
        "defaults": {
          "unit": "percentunit",
          "thresholds": {
            "mode": "absolute",
            "steps": [
              { "color": "red",    "value": 0 },
              { "color": "yellow", "value": 0.999 },
              { "color": "green",  "value": 0.9995 }
            ]
          }
        }
      }
    }
  ]
}

The Alertmanager route that matches the alert labels:

# /etc/alertmanager/alertmanager.yml
route:
  group_by: ['alertname', 'slo']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  receiver: default-receiver
  routes:
    - matchers:
        - severity = "page"
      receiver: orders-oncall-page
      continue: true
    - matchers:
        - severity = "ticket"
      receiver: slo-ticket-queue
    - matchers:
        - severity = "policy-slowdown"
      receiver: orders-eng-manager
    - matchers:
        - severity = "policy-freeze"
      receiver: orders-release-captain

The conventions are: severity: page routes to the page rotation; severity: ticket routes to the ticket queue; severity: policy-slowdown routes to the engineering manager; severity: policy-freeze routes to the release captain. The slo: label is carried through to the notification so the on-call rotation sees the SLO context in the alert.

How to validate it

# Step 1: validate rule syntax.
promtool check rules /etc/prometheus/rules/slo-orders.yml
# expected: SUCCESS: 11 rules found

# Step 2: confirm Alertmanager config loads.
amtool check-config /etc/alertmanager/alertmanager.yml
# expected: SUCCESS

# Step 3: reload Prometheus and confirm recording rules
#          produce values.
curl -s http://prometheus:9090/-/reload
sleep 60
promtool query instant http://prometheus:9090 \
  'slo:orders:errors:ratio_rate1h'
# {service="orders"} 0.000341

Synthetic incident: drive a 5% error rate and confirm the page-worthy alert fires within for:.

# Lab-only injector. Never run against production.
curl -s http://localhost:9001/inject?service=orders&rate=0.05

# After 2 minutes (for: 2m):
amtool alert query --alertmanager.url=http://alertmanager:9093 \
  'severity=page,slo=orders-availability'
# active  OrdersSLOFastBurnPage   fired_at ...

# Confirm the alert is routed to the page receiver:
amtool alert query 'severity=page,slo=orders-availability' \
  | grep -E 'receiver|state'
# receiver=orders-oncall-page state=active

Synthetic slow burn: drive a 0.3% error rate for several hours and confirm the ticket alert fires while the page does not.

curl -s http://localhost:9001/inject?service=orders&rate=0.003

# After several hours:
amtool alert query 'severity=ticket,slo=orders-availability'
# active  OrdersSLOSlowBurnTicket  fired_at ...
amtool alert query 'severity=page,slo=orders-availability'
# (empty)

Synthetic policy threshold: drive enough error volume to consume 25% of the monthly budget in a short time.

# Drive error rate high enough to consume budget fast.
curl -s http://localhost:9001/inject?service=orders&rate=0.10

# After 1 hour:
amtool alert query 'severity=policy-slowdown,slo=orders-availability'
# active  OrdersBudgetPolicy25

The validation is a three-state cycle: page alert on fast burn, ticket alert on slow burn, policy alert on budget consumption. Each synthetic scenario must produce the expected alert within the configured for: window, routed to the expected receiver.

How it can fail

1. Recording rule interval wrong for the burn-rate window. A interval: 5m recording rule on a 6h recording window loses resolution. The 6h window produces a value computed from 72 samples rather than ~720 samples. Symptom: the burn rate is staircase-like; fine-grained incidents are missed. Fix: interval: 30s is the canonical default; raise it to 1m only when Prometheus load justifies.

2. for: not scaled to the window. A for: 30s on a 72h burn rate promotes noise to alerts. A for: 6h on a 1h burn rate delays urgent pages. Symptom: slow-burn alerts trip on transient drift; fast-burn alerts arrive after the incident. Fix: for: 2m (1h), for: 5m (6h), for: 1h (24h), for: 6h (72h).

3. Recorder and alert split across different Prometheus instances. The recording rules are loaded into the recording instance but the alerts are loaded into the alerting instance; the alerting instance does not see the recording rules. Symptom: alert is No data. Fix: consolidate using rule_files: in a single Prometheus, or use remote-write to fan out to both.

4. Dashboard panel reads raw counters, not recording rules. The dashboard re-computes the SLI from http_requests_total every page load. Symptom: dashboard load is slow; the panel and the alert diverge if the counter reset behaviour is different at panel load time. Fix: dashboard reads the recording rule.

5. severity: label typo. A label severity: Page (note the capital P) does not match the Alertmanager route severity = "page". Symptom: the alert routes to the default receiver; nobody is paged. Fix: enforce label values in a rule-linter (e.g. promtool lint mode in 2.55.x, or a CI step).

6. Runbook link returns 404. A typo in the runbook URL. Symptom: on-call rotation clicks the link at 03:00 and lands on a 404. Fix: verify the link during deploy; use linkinator or a similar linter.

7. Multiple recording rules named the same after a sed substitution. A templating error that replaces orders in one expression but not the matching label. Symptom: the rule’s recording produces empty series because the label match is wrong. Fix: render the template with the same substitution throughout.

How to troubleshoot it

When the SLO package is not behaving:

1. Validate rule syntax first. promtool check rules is fast and catches typos.
2. Validate Alertmanager config. amtool check-config catches label-matcher typos.
3. Check recording-rule materialisation. Query each recording rule directly via promtool query instant. An empty result means the rule is misconfigured.
4. Check alert firing. Query each alert (amtool alert query) and inspect the state. An alert in pending state is waiting for for:. An alert in inactive state is below threshold.
5. Check routing. The alert is firing; the receiver is wrong; the route is broken. Inspect amtool config.
6. Check the runbook. The alert is firing; the runbook link returns 404; the runbook URL has changed.

Security implications

The SLO templates add no new attack surface; they consume existing metrics and produce alerts. Verify that the SLI expression does not aggregate away PII while preserving high label cardinality — the recording rule expands the metric into multiple series with the same label set as the source. The dashboard panel and the alert description: are read-only displays of the burn rate.

The Alertmanager routes are config, not secrets. The pagerduty-orders receiver name should not include secrets; the integration key is in the Alertmanager secrets store, not in the route definition.

Performance implications

The recording rules add evaluation cost. A service with one SLO has six recording rules (5m, 30m, 1h, 6h, 24h, 72h) and the budget percentage rule. The 72h window is the most expensive; the rate() spans 8,640 samples at 30s scrape.

On Prometheus 2.55.x with 100 services and one SLO each, the total rule-evaluation budget is ~700 recording rules per 30s interval. This is manageable on a single Prometheus with default limits; beyond that, shard by rule group or use remote-write to a separate Prometheus for alerts.

The dashboard cost is small. The burn-rate panel reads six recording rules; the budget percentage reads one; the SLI reads one; total one PromQL evaluation per panel load. This is the right cost — a dashboard should not be reading raw counters.

Production guidance

• Adopt one convention across all services. The on-call rotation is the consumer; the convention is for them.
• Parametrise the template. Use Sloth, Jsonnet, or a templating layer so the same package serves N services.
• Validate before applying. promtool check rules and amtool check-config are not optional.
• Pair every page alert with a runbook link that resolves. Verify the link at deploy time.
• Add a lint step in CI: every rule must have a severity: label, a slo: label, and a runbook_url: annotation. Lint failures block deploy.

Verification

You should now be able to answer:

• What are the six logical artefacts of the canonical SLO package (recording rules, alert rules, dashboard, route)?
• Why is the convention more important than the specific rule shape?
• How do you parametrise the template on the service name, the SLI expression, and the SLO target?
• Which severity: label values does Alertmanager match against, and what does each route?

Quiz