Alert Ownership

An alert fires at 04:11. The on-call engineer’s phone buzzes. The notification says “DiskSpaceLow on host-prod-db-03.” No team. No service. No severity. No runbook. The engineer opens PagerDuty, sees the alert, and types “who owns host-prod-db-03?” into the team Slack channel. Eight minutes elapse before someone from the database team answers. By that time, the disk is full and the database has stopped accepting writes. The post-mortem asks the obvious question: why did an alert that pointed at a database engine page the platform on-call instead of the database on-call?

This is what alert ownership prevents. An alert without an owner is a noise complaint: a signal that fires without a named recipient, without a severity, without a runbook, and without a dashboard. The on-call engineer is paid to wake up and start looking. The discipline is to make the alert self-describing: every alert carries the team, the service, the severity, the runbook, the dashboard, the SLO link, and the action.

What alert ownership is

Alert ownership is the assignment of a named team to every alert rule. The team is responsible for:

• The alert rule — that the threshold reflects the current SLO, the current architecture, and the current incident history.
• The alert annotations — that the runbook URL, the dashboard URL, and the summary are current.
• The alert response — that someone on the team answers the page, acknowledges within the response-time SLO, and runs the runbook.
• The alert review — that the alert is reviewed quarterly for relevance, threshold, and noise.
• The alert retirement — that the alert is removed when the service is retired or the SLO is decommissioned.

The ownership is not a comment in the rule file. The ownership is the conjunction of seven labels, four annotations, and a live Alertmanager route.

Why a sysadmin cares

Alerts are the highest-cost telemetry the platform emits. A page wakes an engineer; the engineer costs the organisation roughly $1 per minute of context-switching. An alert that fires 200 times a day is a $200/day tax. An alert that fires once a year and pages the wrong team is a missed incident. The discipline is to make alerts specific (the right team), actionable (the right runbook), and bounded (the right threshold).

The trade-off is between signal density and alert fatigue. The platform team that emits one alert per service per SLO has 200 alerts and a manageable on-call; the team that emits one alert per metric has 8,000 alerts and an on-call that ignores all of them.

How it works

The most common shape in production is two layers of alerts:

              +---------------------------+
              |  User-impact alerts      |  Audience: tier-0
              |  - SLO burn rate          |     on-call
              |  - Error budget           |     Action: page
              |  - Availability breach    |
              +---------------------------+
                        |
                        v
              +---------------------------+
              |  Component-health alerts |  Audience: service
              |  - Per-instance restart   |     owner
              |  - Queue depth            |     Action: ticket
              |  - Certificate expiry     |
              +---------------------------+
                        |
                        v
              +---------------------------+
              |  Platform alerts         |  Audience: platform
              |  - Prometheus down        |     on-call
              |  - Disk pressure          |     Action: page
              |  - Scrape failure         |
              +---------------------------+

The three layers have different audiences, different actions, and different routing. User-impact alerts page immediately; component-health alerts open a ticket; platform alerts page the platform on-call. The right approach is fewer user- impact alerts, more component-health alerts; the user-impact alerts are the ones that should wake someone.

The Alertmanager route is what enforces the routing. The route matches on severity, team, and service; the receiver is the team’s PagerDuty service. A missing label is a missing route.

Under the hood: what every alert must carry

How to configure it

Three files encode alert ownership. The first is the Prometheus rule file, the second is the Alertmanager route, the third is the rule-file CI gate.

Prometheus rule file

# /etc/prometheus/rules/checkout.yml
# CONFIGURATION: every alert carries the seven labels and
# four annotations. CI rejects rule files that omit any.
groups:
  - name: checkout.slo
    interval: 30s
    rules:
      # User-impact alert: 5xx rate above 1% for 10 minutes.
      # Pages the payments on-call.
      - alert: Checkout5xxRateHigh
        expr: |
          sum(rate(checkout_http_requests_total{
            service="checkout",status=~"5.."}[5m]))
          /
          sum(rate(checkout_http_requests_total{
            service="checkout"}[5m]))
          > 0.01
        for: 10m
        labels:
          team: payments
          service: checkout
          severity: critical
          slo: availability
          tier: '1'
          environment: production
        annotations:
          summary: 'Checkout 5xx rate above 1% for 10m'
          description: |
            Checkout 5xx rate is {{ $value | humanizePercentage }}
            over the last 5 minutes. Threshold: 1%.
            Action: page payments on-call; follow the runbook.
          runbook_url: 'https://runbooks.example.com/payments/checkout-5xx'
          dashboard_url: 'https://grafana.example.com/d/team-payments-checkout-service'

      # Component-health alert: latency p95 above SLO for
      # 15 minutes. Opens a ticket; does not page.
      - alert: CheckoutLatencyP95High
        expr: |
          histogram_quantile(0.95,
            sum(rate(checkout_http_request_duration_seconds_bucket{
              service="checkout"}[5m])) by (le)
          ) > 0.3
        for: 15m
        labels:
          team: payments
          service: checkout
          severity: warning
          slo: latency
          tier: '1'
          environment: production
        annotations:
          summary: 'Checkout latency p95 above 300ms for 15m'
          description: |
            p95 latency is {{ $value | humanizeDuration }}.
            Threshold: 300ms. SLO breach; investigate.
          runbook_url: 'https://runbooks.example.com/payments/checkout-latency'
          dashboard_url: 'https://grafana.example.com/d/team-payments-checkout-service'

The two alerts differ in severity: critical pages, warning opens a ticket. The team, service, slo, tier, and environment labels are the routing and escalation keys.

Alertmanager route

# /etc/alertmanager/alertmanager.yml
# CONFIGURATION: routes pages by team, then by severity.
# Critical pages immediately; warning opens a ticket.
route:
  receiver: 'default-null'
  group_by: ['alertname', 'team', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    # Critical pages the team's PagerDuty service.
    - matchers:
        - team = "payments"
        - severity = "critical"
      receiver: 'pd-payments-oncall'
      continue: false
    # Warning opens a ticket in the team's queue.
    - matchers:
        - team = "payments"
        - severity = "warning"
      receiver: 'jira-payments-warn'
      continue: false
    # Default catch-all routes un-owned alerts to triage.
    - matchers:
        - team =~ ".*"
      receiver: 'pd-default-triage'
      continue: false

receivers:
  - name: 'pd-payments-oncall'
    pagerduty_configs:
      - service_key: '<redacted>'
        severity: 'critical'
        # Tier-0 services get a shorter escalation; tier-1
        # uses the default 30-minute escalation.
        details:
          escalation_policy: 'payments-tier1-30m'
  - name: 'jira-payments-warn'
    webhook_configs:
      - url: 'http://alertmanager-bridge:8080/jira'
        send_resolved: true
  - name: 'pd-default-triage'
    slack_configs:
      - channel: '#observability-triage'

The route is the action. severity=critical and team=payments maps to pd-payments-oncall. severity=warning maps to jira-payments-warn. The catch-all keeps un-owned alerts visible.

CI gate

# /usr/local/bin/check-alert-labels.sh
# CONFIGURATION: CI gate that rejects rule files missing
# any of the seven required labels.
set -euo pipefail

REQUIRED_LABELS=(team service severity slo tier environment)
REQUIRED_ANNOTATIONS=(summary description runbook_url dashboard_url)

failed=0
for rule_file in /etc/prometheus/rules/*.yml; do
  for label in "${REQUIRED_LABELS[@]}"; do
    if ! grep -q "^          $label:" "$rule_file"; then
      echo "FAIL: $rule_file missing label '$label'"
      failed=1
    fi
  done
  for annotation in "${REQUIRED_ANNOTATIONS[@]}"; do
    if ! grep -q "^          $annotation:" "$rule_file"; then
      echo "FAIL: $rule_file missing annotation '$annotation'"
      failed=1
    fi
  done
done

exit "$failed"

The gate runs in CI on every change to /etc/prometheus/rules/. A rule file that omits any of the seven labels or four annotations fails the build.

How to validate it

Validation is the conjunction of the seven labels and four annotations. The audit confirms every alert has all eleven.

# READ-ONLY. List every firing alert and its labels.
curl -s http://prometheus:9090/api/v1/alerts | \
  jq '.data.alerts[] | {labels: .labels}'

# READ-ONLY. List every rule file and confirm the seven
# labels are present.
for rule_file in /etc/prometheus/rules/*.yml; do
  echo -n "$rule_file: "
  for label in team service severity slo tier environment; do
    grep -q "^          $label:" "$rule_file" && continue || \
      echo "missing $label"
  done
  echo "ok"
done

# READ-ONLY. List every Alertmanager route and confirm each
# team in the catalogue has a critical and a warning receiver.
amtool config routes show --alertmanager.url=http://alertmanager:9093

# READ-ONLY. Confirm every runbook_url resolves. A 404 is a
# gap in the ownership.
for url in $(grep -RH "runbook_url:" /etc/prometheus/rules/ \
              | awk -F"'" '{print $2}'); do
  status=$(curl -s -o /dev/null -w '%{http_code}' "$url")
  echo "$status $url"
done

Illustrative output for the alert audit:

$ curl -s http://prometheus:9090/api/v1/alerts | \
    jq '.data.alerts[] | .labels'
{
  "alertname": "Checkout5xxRateHigh",
  "team": "payments",
  "service": "checkout",
  "severity": "critical",
  "slo": "availability",
  "tier": "1",
  "environment": "production"
}

$ for url in $(grep -RH "runbook_url:" /etc/prometheus/rules/ \
                | awk -F"'" '{print $2}'); do
    status=$(curl -s -o /dev/null -w '%{http_code}' "$url")
    echo "$status $url"
  done
200 https://runbooks.example.com/payments/checkout-5xx
200 https://runbooks.example.com/payments/checkout-latency
404 https://runbooks.example.com/payments/checkout-cache-miss

The 404 is the gap: the runbook URL resolves for two alerts but not for the third. The audit catches it before the page fires.

How it can fail

Six failure shapes appear repeatedly when alerts are unowned:

1. Missing team label. The alert routes to the catch- all. Symptom: the platform on-call is paged for every alert, including ones they cannot fix.
2. Generic runbook URL. The annotation points to https://runbooks.example.com/payments instead of the specific alert’s runbook. Symptom: the on-call engineer opens the runbook index and spends five minutes finding the right page.
3. No severity. Alertmanager routes by severity; without it, every alert goes to the default receiver. Symptom: a component-health alert pages the on-call when it should have opened a ticket.
4. Threshold without history. The alert was set with a threshold that made sense for the previous quarter’s traffic. The current traffic is two-thirds of the previous quarter’s; the alert fires every day. Symptom: alert fatigue; the team silences the alert; the next real breach is missed.
5. Stale SLO link. The annotation’s slo label points to an SLO that has been decommissioned. Symptom: the on-call engineer follows the SLO link and gets a 404.
6. Alert without runbook. The annotation has no runbook_url. Symptom: the on-call engineer wakes up, reads the alert, and has no documented action.

How to troubleshoot it

The diagnostic order for “who pages for this alert?”:

1. Confirm the labels. Open Prometheus’s /alerts page. Note the seven labels. Any missing label is a gap.
2. Confirm the route. Run amtool config routes show. Find the matcher for team and severity. If the matcher does not exist, the alert falls through.
3. Confirm the runbook. Open the runbook_url annotation. If the URL 404s, the runbook has been moved or deleted.
4. Confirm the dashboard. Open the dashboard_url annotation. The dashboard should resolve to the team’s service-level dashboard.
5. Confirm the SLO link. Open the SLO document. The SLO should still be active and the alert threshold should match.
6. Form the diagnosis. Missing label, or missing route, or stale runbook, or stale dashboard, or stale SLO. Each is a separate fix.

Security implications

Alert ownership intersects with security at the credential boundary. The Alertmanager PagerDuty service keys are secrets; the webhook URLs to Jira or Slack are secrets; the runbook_url annotation may include credentials in the URL (a runbook that requires basic auth). The discipline:

• PagerDuty service keys are stored in a secret manager and injected at Alertmanager startup, not in the YAML.
• Webhook URLs are stored in the same secret manager.
• The runbook_url annotation is an internal URL; basic auth is enforced at the runbook site, not in the URL.
• The alert payload does not include secrets. The description template uses metric values only; no credentials are interpolated.

Performance implications

Alertmanager performance is bounded by alert volume and grouping. An Alertmanager that receives 8,000 alerts a day with no grouping fans out 8,000 notifications; one that groups by ['alertname', 'team', 'service'] fans out 200. The discipline:

• Group by alertname, team, and service. The grouping collapses multiple instances of the same alert into one notification.
• Use repeat_interval to bound re-pages. A four-hour repeat is a common default; tier-0 services use one hour.
• Tune group_wait and group_interval to the team’s response time. A team with a 30-minute SLO does not need 10-second group_wait.

Production guidance

• Require the seven labels and four annotations in CI. The gate is one shell script that grep’s for the labels.
• Group by alertname, team, and service. The grouping is what makes Alertmanager usable at scale.
• Review alerts quarterly. The review removes alerts for decommissioned SLOs and tunes thresholds that have drifted.
• Treat the alert as a contract. The alert says “this team answers this page within this response time”; the team answers.
• Document the alert. The runbook URL is the documentation; the dashboard URL is the context; the description is the action.

Verification

You should now be able to answer:

• What seven labels and four annotations must every alert carry?
• What is the difference between user-impact and component- health alerts?
• Why must every alert rule live behind a CI gate that enforces the labels?
• What is the failure shape when a critical alert has no runbook URL?
• How do you audit an Alertmanager deployment for alert ownership?

Quiz