Owner

A page arrives at 03:00. The on-call reads the alert summary: “orders-api 5xx ratio above 5% in eu-west-1.” They open the runbook. The what-failed and impact sections confirm the alert is the one they are looking at. The mitigations section lists five steps. The on-call executes them. The first mitigation is “restart the orders-api deployment.” They run it. The 5xx ratio holds. They escalate. They look for the next mitigation. The runbook does not say who the team is. There is no Slack channel linked. There is no escalation path. The on-call pings the #incident channel. Twenty minutes later, an engineer from the checkout team responds. The owner was never declared on the runbook; the on-call had to find the team by guessing.

An owner field is the runbook’s answer to the question every escalating on-call asks eventually: who else is on the hook for this? The answer must be a team, with a contact channel and an escalation path. The cost of declaring the owner is one extra line in the rule file and one extra line in the runbook frontmatter. The cost of leaving it undeclared is paid every time the alert escalates.

What it is

An owner field is a triple:

1. The team is the named group responsible for the alert and the runbook. The team name matches the team label on the alert rule and the owner: field in the runbook frontmatter.
2. The channel is the primary contact path. A Slack channel, a PagerDuty rotation, or an email alias. The channel is staffed when the alert fires.
3. The escalation is the path the on-call follows when the primary channel does not respond. A senior engineer, a manager, or a sister team. The escalation is named, not inferred.

An owner field that names only one of the three is incomplete. A team without a channel cannot be reached. A channel without an escalation stalls. An escalation without a team is just a name.

Why a sysadmin cares

The owner field is the runbook’s contract with the rest of the org. Without it, every incident that escalates pays a coordination tax: the on-call has to find the team, find the channel, and find the escalation by asking around. The tax is paid under time pressure, by the engineer least able to pay it.

The same discipline pays compound interest. Once the alert carries a team label, the Alertmanager route can match the label and route the alert to the right PagerDuty rotation automatically. The on-call manager dashboard can group alerts by team. The SLO report can attribute incidents to the team that owns them. None of that works without a team label on the rule.

How it works

The owner field is the bridge between the alert and the rest of the organisation:

  Alert rule (machine-readable)
  -----------------------------
  alert: OrdersApiHighErrorRate
  labels:
    severity: critical
    team: checkout         <-- owner label
    service: orders-api
              |
              v
  Alertmanager route (matches on labels)
  --------------------------------------
  routes:
    - matchers:
        - team = "checkout"
      receiver: checkout-pagerduty
              |
              v
  PagerDuty rotation (the named team)
  -----------------------------------
  checkout team:
    primary:   @alice, @bob (rotating weekly)
    secondary: @carol (escalation)
    channel:   #team-checkout

The mapping is mechanical:

• The team label is the canonical owner identifier. The Alertmanager route matches on it and routes the alert to the team’s receiver.
• The receiver is a PagerDuty rotation, a Slack channel, or an email alias. The team is responsible for staffing it.
• The escalation path is declared in the runbook and in the PagerDuty rotation. The on-call manager dashboard surfaces it.

A runbook whose owner field names an individual rather than a team is a bug: the individual leaves, the alert routes to a non-existent rotation.

How to configure it

The alert rule, with a team label:

groups:
  - name: orders-api.slo
    rules:
      - alert: OrdersApiHighErrorRate
        expr: |
          sum by (service, region) (
            rate(http_requests_total{service="orders-api", status=~"5.."}[5m])
          )
          /
          sum by (service, region) (
            rate(http_requests_total{service="orders-api"}[5m])
          )
          > 0.05
        for: 5m
        labels:
          severity: critical
          team: checkout
          service: orders-api
          slo: availability
        annotations:
          summary: 'orders-api 5xx ratio above 5% in {{ $labels.region }}'
          runbook_url: 'https://runbooks.example.com/checkout/orders-api-5xx.html'
          dashboard_url: 'https://grafana.example.com/d/orders-api/orders-api-overview?var-region={{ $labels.region }}&from=now-1h&to=now'

The corresponding Alertmanager route, matching on team:

route:
  receiver: default
  group_by: ['alertname', 'team']
  routes:
    - matchers:
        - team = "checkout"
      receiver: checkout-pagerduty
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 4h
    - matchers:
        - team = "platform"
      receiver: platform-pagerduty
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 4h

receivers:
  - name: default
    email_configs:
      - to: 'oncall@example.com'
  - name: checkout-pagerduty
    pagerduty_configs:
      - service_key: '<pd-integration-key>'
  - name: platform-pagerduty
    pagerduty_configs:
      - service_key: '<pd-integration-key>'

The matching runbook frontmatter:

---
title: 'orders-api 5xx ratio above 5%'
alert: OrdersApiHighErrorRate
severity: critical
owner: checkout
channel: '#team-checkout'
escalation:
  - '@checkout-oncall-manager'
  - '@checkout-engineering-manager'
---

Each element earns its place:

• The team: checkout label on the alert is the canonical identifier. Alertmanager routes the alert to the checkout-pagerduty receiver by label match.
• The channel: '#team-checkout' field in the runbook frontmatter is the primary contact path. The on-call pings the channel when they need help.
• The escalation: field is the named escalation path. The on-call manager dashboard surfaces it during incidents.

How to validate it

Three checks, in order. The first is a static check against the rule files; the second is read-only against Alertmanager; the third is read-only against the live alert payload.

# 1. Does every alert rule with severity in {critical, warning} have a team label?
#    A simple lint that flags high-severity rules without a team.
yq eval-all '.groups[].rules[] | select(.labels.severity == "critical" or .labels.severity == "warning") | select(.labels.team == null) | .alert' /etc/prometheus/rules/*.yml

Expected output: empty. A rule listed here is a high-severity alert without a team label; the on-call cannot be routed to a team without it.

# 2. Does every team label on a firing alert match an Alertmanager route?
rule='OrdersApiHighErrorRate'
team=$(curl -s "http://alertmanager:9093/api/v2/alerts?filter=alertname%3D%22${rule}%22" \
  | jq -r '.[0].labels.team')
curl -s http://alertmanager:9093/api/v2/routes \
  | jq -r '.[].matchers[] | select(.name=="team") | .value' \
  | sort -u | grep -F "$team"

Expected output: the team name. An empty result means the alertmanager route does not match the team’s label; the alert falls through to the catch-all receiver.

# 3. Does the runbook frontmatter declare the same team as the alert label?
yq eval '.owner' runbooks/checkout/orders-api-5xx.md
yq eval '.labels.team' /etc/prometheus/rules/orders-api.yml | head -1

Expected output: checkout from both. A mismatch means the alert and the doc disagree on the owner; reconcile by editing one or the other.

How it can fail

Six failure modes, each observable:

1. The alert has no team label. Symptom: the alert fires and routes to the catch-all receiver. Cause: the rule was authored without the label. Confirm by inspecting the alert payload.

2. The team label names a team that does not exist. Symptom: the alert routes to a non-existent PagerDuty rotation. Cause: the team was renamed or disbanded. Confirm by listing Alertmanager receivers.

3. The team label names a team but the runbook owner is a different team. Symptom: the alert fires, the page goes to one team, and the runbook lists another. Cause: the alert or the doc was edited without the other. Confirm by comparing the rule’s team label to the runbook’s owner field.

4. The owner is an individual, not a team. Symptom: the team label is alice or the runbook owner is @alice. Cause: the rule was authored before the team was formalised. Confirm by inspecting the label and the frontmatter.

5. The escalation path is empty. Symptom: the runbook frontmatter has owner: but no escalation: field. Cause: the author did not declare an escalation. The on-call has to find the escalation by asking around.

6. The contact channel is a dead link. Symptom: the channel listed in the runbook is archived or read-only. Cause: the channel was retired when the team moved. Confirm by sending a test message to the channel.

How to troubleshoot it

In order:

1. Does the alert have a team label? grep team /etc/prometheus/rules/. A missing label means the alert cannot be routed by team.
2. Does the team label match an Alertmanager route? Inspect the routes file. A team without a route falls through to the catch-all.
3. Does the team label match the runbook owner? Compare the rule’s team label to the runbook’s owner: field. A mismatch is the bug.
4. Does the team have a staffed rotation? Inspect PagerDuty. A rotation with no current on-call is a dead route.
5. Does the escalation path resolve? Send a test ping to each escalation contact. A dead escalation path is a liability.

Security implications

The owner field is not security-sensitive by itself. It becomes sensitive when it names an individual whose contact information should not be in a broadly-shared doc, or when the escalation path includes a channel that exposes internal communications.

The discipline is to name teams, not individuals. A team name in the owner field is durable; an individual name is not. An escalation path that names a person is a doc that has to be updated every time the person changes role; an escalation path that names a role (@checkout-oncall-manager) is durable.

A runbook whose escalation path includes a customer’s contact or a vendor’s pager is a doc that should be moved behind the same access controls as the customer data it describes. The simpler path is to keep the runbook internal and route external contacts through a separate escalation procedure.

Performance implications

The owner field is read once per incident. Performance implications are about the time-to-correct-pager, not the doc size. A precise owner field compresses the time-to-correct-pager because the alert routes to the right team automatically. A missing owner field extends it because the alert routes to the catch-all and waits for a human to forward it.

The cost of declaring the owner is one extra label in the rule file and one extra field in the runbook frontmatter. The cost of a missing owner is paid every incident.

Production guidance

• Add a team label to every alert rule that pages the on-call. A rule without a team label cannot be routed.
• Keep the owner field on the runbook in sync with the team label on the alert. Mismatches are bugs.
• Name the escalation path explicitly. A runbook without an escalation is a runbook the on-call cannot extend.
• Validate the team label and the Alertmanager route in CI. A team label without a matching route fails the check.

Verification

• Why is team ownership preferred over individual ownership on an alert rule?
• What three shapes must an owner field declare?
• How is a missing team label detected in CI?
• What is the symptom in Alertmanager when the team label names a team without a route?

Quiz