Severity and Routing Labels

A team rebuilds its route tree after six months of production. The cumulative cost of the old tree shows up in two numbers: how many severity: critical pages the on-call rota received that should have been tickets, and how many severity: critical pages never reached a human because the route match silently failed. Both numbers are non-zero in every estate that grew without a labelled contract. Severity and routing labels are that contract.

What it is

Routing labels are the dimensions an Alertmanager route tree matches on to decide who receives a given alert. The two labels that almost every production route tree depends on are:

• severity — the operational priority of the alert. Common values: info, warning, critical. The label drives whether the route ends at a page, a ticket, or a chat notification.
• team — the owning team. Common values: the team name in lowercase (checkout, platform, data). The label drives which rota or which Slack channel receives the alert.

Two further labels appear in most estates but are not strictly required:

• service — the owning service (often identical to team in small shops, diverging as the org scales).
• env or stage — prod, staging, dev. Used to keep non-production alerts off the production route.

The labels are set on the Prometheus rule itself, not on the metric. The expr returns metric series with whatever labels the exporter set; the alert rule adds labels: at the rule top level, which are merged into the final alert.

Why a sysadmin cares

Severity is the contract between the rule author and the on-call. A rule marked severity: critical says: someone will be paged at 03:00 for this. A rule marked severity: warning says: this is ticket-shaped; we want it visible but not waking anyone. If the labels do not match what the route tree expects, the contract breaks silently. The on-call rota may be over-paged by tickets, or under-paged by real incidents.

The team label exists for the same reason. A page that arrives in the wrong team’s inbox is a 15-minute delay. The cost of a wrong team label compounds with every misroute.

How it works

Alertmanager evaluates alerts against a routing tree. The tree is matched top-down; the first route whose matchers: block matches the alert’s labels wins, and the alert is sent to that route’s receiver. A continue: true on a route lets siblings also match.

   Prometheus                    Alertmanager
   -----------                   ------------
   alert fires      ---->        route tree
     labels:                       |
       severity=critical           +-- root (catch-all)
       team=checkout               |     |
                                   |     +-- match severity=critical
                                   |     |     -> pager-primary
                                   |     +-- match team=checkout
                                   |     |     -> slack-checkout
                                   |     +-- match severity=warning
                                   |           -> jira-tickets
                                   +-- default -> email-oncall

The matchers block is a list of label matchers. Three forms:

matchers:
  - severity = "critical"
  - severity =~ "crit|high"
  - service != "test-payments"

Equality, regex, and inequality. Matchers within a single route are AND-ed; an alert matches the route only when every matcher matches.

How to configure it

Two sides to this: what the rule sets, and what Alertmanager matches.

The rule side. Add a stable set of labels to every rule:

groups:
  - name: orders-api.slo
    rules:
      - alert: OrdersApiHighErrorRate
        expr: |
          sum by (service, region) (
            rate(http_requests_total{service="orders-api", status=~"5.."}[5m])
          )
          /
          sum by (service, region) (
            rate(http_requests_total{service="orders-api"}[5m])
          )
          > 0.05
        for: 5m
        labels:
          severity: critical
          team: checkout
          service: orders-api
        annotations:
          summary: 'orders-api 5xx ratio above 5% in {{ $labels.region }}'
          runbook_url: 'https://runbooks.example.com/checkout/orders-api-5xx'

      - alert: OrdersApiElevatedErrorRate
        expr: |
          sum by (service, region) (
            rate(http_requests_total{service="orders-api", status=~"5.."}[10m])
          )
          /
          sum by (service, region) (
            rate(http_requests_total{service="orders-api"}[10m])
          )
          > 0.01
        for: 15m
        labels:
          severity: warning
          team: checkout
          service: orders-api
        annotations:
          summary: 'orders-api 5xx ratio above 1% for 15 minutes'
          runbook_url: 'https://runbooks.example.com/checkout/orders-api-warn'

Two rules, two severities. The first pages at 5 minutes; the second files a ticket at 15. Both belong to the checkout rota via the team label.

The Alertmanager side. The corresponding alertmanager.yml fragment:

route:
  receiver: default-receiver
  group_by: [alertname, region]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers:
        - severity = "critical"
        - team = "checkout"
      receiver: pager-checkout-primary
      continue: false

    - matchers:
        - severity = "critical"
        - team = "platform"
      receiver: pager-platform-primary
      continue: false

    - matchers:
        - severity = "warning"
      receiver: jira-tickets
      continue: false

    - matchers:
        - severity = "info"
      receiver: slack-low-priority
      continue: false

receivers:
  - name: default-receiver
    email_configs:
      - to: '[email protected]'

  - name: pager-checkout-primary
    pagerduty_configs:
      - service_key: '<redacted>'

  - name: pager-platform-primary
    pagerduty_configs:
      - service_key: '<redacted>'

  - name: jira-tickets
    webhook_configs:
      - url: 'https://jira.example.com/webhook/alertmanager'

  - name: slack-low-priority
    slack_configs:
      - api_url: '<redacted>'
        channel: '#alerts-low'

Three things to notice. First, the route order matters: more specific routes go first. Second, continue: false (the default) stops the walk after the first match, which is why the severity=warning route does not also fall through to the catch-all. Third, every receiver has a backstop: email for the catch-all, PagerDuty for paging, a webhook for tickets, Slack for chat.

How to validate it

Three checks.

# 1. Is the route tree syntactically valid?
amtool check-config alertmanager.yml

Expected output on a clean file:

Checking 'alertmanager.yml'
  SUCCESS

# 2. Does the route actually match a real alert? Use amtool to render.
amtool config routes test --config.file=alertmanager.yml \
  --alertmanager.url=http://alertmanager:9093 \
  severity=critical team=checkout service=orders-api

Expected output:

receiver: pager-checkout-primary

If the response says default-receiver, the route tree does not match the labels you expected. Fix the matcher spelling, the label spelling, or the route order.

# 3. End-to-end: list the live alerts Alertmanager is currently holding.
curl -s http://alertmanager:9093/api/v2/alerts \
  | jq '.[] | {name: .labels.alertname, severity: .labels.severity,
                team: .labels.team, receiver: .receivers[].name}'

Expected output during an incident:

{
  "name": "OrdersApiHighErrorRate",
  "severity": "critical",
  "team": "checkout",
  "receiver": "pager-checkout-primary"
}

The receivers field confirms the match in production.

How it can fail

Six failure modes:

1. Spelling drift. severity: Critical on the rule, severity = "critical" on the route. The route does not match. Symptom: critical alerts fall through to the catch-all, which is usually email. Confirm by amtool config routes test with the live alert labels.

2. Route order traps. A broad matcher earlier in the tree shadows a specific matcher later. Symptom: critical alerts meant for team=checkout land in team=platform’s pager. Reorder the routes so the most specific is first.

3. Single severity scheme. Every rule is severity: critical because that is what gets paged. Symptom: ticket-shaped alerts wake the on-call; the rota burns out; pages start getting snoozed. Add warning and info, retune the route tree.

4. continue: true cascade. A route with continue: true matches an alert that should have stopped one level higher. Symptom: the same alert appears in three receivers. Confirm by listing receivers for a live alert.

5. Missing team label. A rule omits team: and lands in the catch-all. Symptom: nobody owns the alert; nobody acts on it. Make team mandatory in the rule-review checklist.

6. Public-vs-private confusion. A rule marked severity: critical but tied to a status page incident gets routed as a page but never reaches the status-page receiver. Symptom: the customer sees no incident while the rota pages. Separate the internal severity label from a public_severity annotation that the status-page integration reads.

How to troubleshoot it

In order:

1. What labels reached Alertmanager? curl http://alertmanager:9093/ api/v2/alerts | jq '.[] | .labels'. Confirm severity, team, service, alertname are present and lower-case.
2. Which receiver did the dispatcher pick? Same endpoint, look at receivers[]. If it is default-receiver, the matcher chain failed somewhere.
3. Would the matcher pick the right receiver in isolation? Run amtool config routes test with the same labels. If the answer differs from what production did, the route tree in production is stale; reload Alertmanager with SIGHUP.
4. Did the receiver try to notify? Tail Alertmanager logs for the fingerprint; notify log lines show webhook attempts and their HTTP status codes.
5. Did the receiver succeed? A 200 from the webhook may still be a logical failure (e.g. PagerDuty returns 200 but reports the service key is invalid). Check the receiver-specific integration dashboard.

Security implications

Routing labels are not sensitive on their own, but the receivers they point at are. A misconfigured route that sends a critical alert to a public Slack channel leaks the existence and nature of the incident. Treat the receiver list as part of the platform access control surface: who can read what, who can mute what, who can ack what.

The webhook URLs in receivers frequently carry API tokens. Alertmanager supports the *_file variants (webhook_config_file, pagerduty_config_file) so the secret never lives in alertmanager.yml. Use them.

Performance implications

The route tree is walked on every alert dispatch. A tree with hundreds of routes and complex regex matchers can dominate the dispatcher. Common mitigations:

• Anchor regex matchers (=~ "^checkout$" rather than =~ "checkout").
• Prefer equality (=) over regex (=~).
• Keep the tree small; collapse narrow routes into a single matchers block where possible.

Grouping (group_by: [alertname, region]) reduces the number of notifications the receiver has to handle, but does not reduce the dispatcher CPU cost.

Production guidance

• Pick the label set once: severity, team, service, optionally env. Document it. Review rules against it.
• Three severity values cover almost every estate: info, warning, critical. Resist a fourth unless the business really demands it; the cognitive load of four values is heavier than the precision it adds.
• Make severity drive the receiver, not the page-vs-ticket decision in isolation. A critical alert still goes to a ticket queue if the team rota is offline and the receiver has a fallback.
• Separate internal severity from public severity. The label severity is for the on-call rota. The annotation public_severity (or a label like customer_impact) feeds the status page integration.

Verification

• What three labels should a production alert estate carry on every rule?
• How does an Alertmanager route match an alert, and what is the effect of continue: true?
• Why does a single-severity estate tend to fail as the rota grows?
• How does internal severity differ from public severity, and where does each live?

Quiz