Service Ownership

The checkout service pages at 02:14. The alert is named “Checkout5xxRateHigh.” The runbook says “see payments team Slack channel.” The Slack channel has not had a payment-team member in it for nine months; the team was reorganised into “commerce” and “billing” six months ago. The current on-call rotation belongs to the billing team, who did not know the checkout service was theirs until the page arrived. The alert goes round-robin between three engineers, none of whom have shipped a change to checkout in the past year. The runbook links to a Confluence page last edited by an ex-employee.

This is what happens when service ownership is implied rather than declared. The team that owns the code owns the service; the team that owns the service owns its observability. The boundary is not a wiki page. The boundary is the on-call rotation, the alert receivers, the Grafana folder, and the runbook URL — all wired to the same team.

What service ownership is

Service ownership is the assignment of a named team to every production service. The team is responsible for:

• The code that the service runs (the application).
• The deployment configuration (the manifests, the Helm values, the Compose file, the VM image).
• The dependency choices (the libraries, the external services).
• The observability of the service: its RED instrumentation, its dashboards, its alerts, its runbooks, its SLOs.
• The on-call rotation for the service.
• The response to incidents that affect the service.
• The follow-up remediation after incidents.

Everything else — the host, the network, the database engine, the shared observability stack — is owned by another team. The application team does not run the cluster; the platform team does not own the application.

Why a sysadmin cares

The trade-off is between expertise and coordination cost. A platform team that owns observability for every application has a coordination problem at every change. A platform team that owns the platform and lets application teams own their services has a coordination problem only at the boundary.

The cost of explicit service ownership is the overhead of maintaining a service catalogue. The benefit is that every service has a named owner who is on-call and accountable. Without the catalogue, the on-call engineer for a misrouted page is whichever team happens to be available, which is usually no team at all.

How it works

The discipline is “you build it, you run it,” attributed to operations teams that have run it well for two decades. The shape is:

                Service Catalogue
                ------------------
   +-------------+    +-------------+    +-------------+
   | service:    |    | service:    |    | service:    |
   | checkout    |    | payment-svc |    | fraud-detect|
   |             |    |             |    |             |
   | owner:      |    | owner:      |    | owner:      |
   | payments    |    | payments    |    | trust-and-  |
   |             |    |             |    | safety      |
   | oncall:     |    | oncall:     |    | oncall:     |
   | pd-payments |    | pd-payments |    | pd-tas      |
   |             |    |             |    |             |
   | slo:        |    | slo:        |    | slo:        |
   | avail 99.9% |    | avail 99.95|    | avail 99.5% |
   | lat p95<300 |    | lat p95<200 |    | lat p95<500 |
   +-------------+    +-------------+    +-------------+
         |                  |                   |
         v                  v                   v
   +-------------+    +-------------+    +-------------+
   | Dashboards  |    | Dashboards  |    | Dashboards  |
   | Alerts      |    | Alerts      |    | Alerts      |
   | Runbooks    |    | Runbooks    |    | Runbooks    |
   | folder:     |    | folder:     |    | folder:     |
   | team-payments    | team-payments    | team-tas    |
   +-------------+    +-------------+    +-------------+

The catalogue is a single document, machine-readable, that links every service to its owner. The link is enforced by the configuration: every alert rule carries the team label, every Grafana dashboard lives in the team’s folder, every Alertmanager receiver maps to the team’s PagerDuty service.

The most common shape in production is one team owns several services (payments owns checkout, payment-svc, and billing- svc). The other shape, one service is jointly owned by two teams, is a smell: the boundaries need to be re-drawn.

Under the hood: what “ownership” actually means in the

configuration

How to configure it

Three files encode service ownership. The first is the service catalogue, the second is the Prometheus rule-group that owns the alerts, the third is the Alertmanager route that pages.

Service catalogue

# /etc/services/catalogue.yaml
# CONFIGURATION: every production service and its owner.
# The file is the source of truth; CI derives the alert
# routing and the dashboard folders from it.
services:
  - name: checkout
    owner: payments
    oncall: pd-payments-oncall
    tier: 1
    slo:
      availability: 0.999
      latency_p95_ms: 300
    repo: github.com/payments/checkout
    runbook_root: 'https://runbooks.example.com/payments'
    dependencies:
      - payment-svc
      - fraud-detect
      - postgres-orders

  - name: payment-svc
    owner: payments
    oncall: pd-payments-oncall
    tier: 0
    slo:
      availability: 0.9995
      latency_p95_ms: 200
    repo: github.com/payments/payment-svc
    runbook_root: 'https://runbooks.example.com/payments'

  - name: fraud-detect
    owner: trust-and-safety
    oncall: pd-tas-oncall
    tier: 1
    slo:
      availability: 0.995
      latency_p95_ms: 500
    repo: github.com/tas/fraud-detect
    runbook_root: 'https://runbooks.example.com/tas'

The catalogue is generated into Prometheus rule-group labels, Alertmanager routes, Grafana folders, and runbook URLs. The generation is CI; the catalogue is the only file a human edits.

Prometheus: derive the alert rules from the catalogue

# /etc/prometheus/rules/checkout.yml
# CONFIGURATION: every rule for the checkout service carries
# team=payments and service=checkout. Alertmanager routes
# by team; Grafana folder lookup is by team; runbook URL is
# built from the catalogue.
groups:
  - name: checkout.slo
    interval: 30s
    rules:
      - alert: Checkout5xxRateHigh
        expr: |
          sum(rate(checkout_http_requests_total{
            service="checkout",status=~"5.."}[5m]))
          /
          sum(rate(checkout_http_requests_total{
            service="checkout"}[5m]))
          > 0.01
        for: 10m
        labels:
          team: payments        # from catalogue: checkout.owner
          service: checkout
          severity: critical
          slo: availability
        annotations:
          summary: 'Checkout 5xx rate above 1% for 10m'
          # The runbook URL is built from catalogue.runbook_root
          # and the alert name.
          runbook_url: 'https://runbooks.example.com/payments/checkout-5xx'
          dashboard_url: 'https://grafana.example.com/d/team-payments-checkout'

The team and service labels are non-negotiable. A rule that omits them is rejected by CI; a rule that uses the wrong team is rejected by review. The labels are what the rest of the platform keys off.

Alertmanager: route by service and team

# /etc/alertmanager/alertmanager.yml
# CONFIGURATION: routes pages by team, with a deeper match on
# service for tier-0 services that have their own escalation.
route:
  receiver: 'default-null'
  group_by: ['alertname', 'team', 'service']
  routes:
    - matchers:
        - team = "payments"
        - service = "payment-svc"   # tier-0: deeper escalation
      receiver: 'pd-payments-tier0'
      continue: true
    - matchers:
        - team = "payments"
      receiver: 'pd-payments-oncall'
      continue: false
    - matchers:
        - team = "trust-and-safety"
      receiver: 'pd-tas-oncall'
      continue: false
    - matchers:
        - team =~ ".*"
      receiver: 'pd-default-triage'
      continue: false

The deeper match for tier-0 services means the payment-svc alert pages a separate PagerDuty service with its own escalation policy. Other payments services (checkout, billing-svc) use the standard payments on-call. The split is encoded in the catalogue (tier: 0 vs tier: 1) and generated into the route.

How to validate it

Validation is the conjunction of the four artefacts. The audit confirms each service has all four.

# READ-ONLY. List every catalogue entry and its owner.
yq '.services[] | {name, owner, oncall}' /etc/services/catalogue.yaml

# READ-ONLY. List every alert rule that references a service
# in the catalogue, and confirm it carries the catalogue's
# owner as its team label.
for svc in $(yq '.services[].name' /etc/services/catalogue.yaml); do
  owner=$(yq ".services[] | select(.name == \"$svc\") | .owner" \
          /etc/services/catalogue.yaml)
  echo -n "$svc (owner=$owner): "
  found=$(grep -RH "service=\"$svc\"" /etc/prometheus/rules/ \
          | wc -l)
  matches=$(grep -RH "service=\"$svc\"" /etc/prometheus/rules/ \
            | grep -c "team: $owner")
  echo "rules=$found team_matches=$matches"
done

# READ-ONLY. List every Alertmanager route and confirm each
# catalogue owner has a route.
amtool config routes show --alertmanager.url=http://alertmanager:9093

# READ-ONLY. List every Grafana folder and confirm the team
# prefix matches an owner.
curl -s -u admin:$GRAFANA_PASS \
  http://grafana:3000/api/folders \
  | jq '.[] | .title' | grep '^team-'

Illustrative output for the ownership audit:

checkout (owner=payments): rules=12 team_matches=12
payment-svc (owner=payments): rules=8 team_matches=8
fraud-detect (owner=trust-and-safety): rules=5 team_matches=5

Every catalogue entry has rules, every rule matches its service’s owner. The audit passes. A row like checkout (owner=payments): rules=12 team_matches=11 would flag one rule with the wrong team label.

How it can fail

Six failure shapes appear repeatedly when service ownership is fragmented:

1. Two teams, one service. The payments team owns the code; the billing team owns the deployment; the data team owns the database. Symptom: a 5xx alert pages three teams, none of whom can fix it alone.
2. One team, no on-call. The catalogue says owner: payments, but the payments team has no rotation; the alert routes to a PagerDuty service no one is on. Symptom: pages go to voicemail.
3. Stale catalogue. The catalogue says owner: growth-team; the growth team was disbanded eighteen months ago. Symptom: pages go to a defunct Slack channel.
4. Service sprawl. The catalogue has 200 services; 30 of them are zombie services that no one has run in a year. Symptom: dashboards are stale; alerts fire on cold endpoints; runbooks link to deleted repos.
5. Runbook drift. The alert fires; the runbook URL returns a generic “see SRE” page. Symptom: the on-call engineer spends twenty minutes figuring out what to do.
6. Tier mis-classification. A tier-0 service is treated as tier-1. Its alert routes to a single PagerDuty service with a six-hour escalation. Symptom: a tier-0 outage waits six hours for escalation.

How to troubleshoot it

The diagnostic order for “who owns this alert?”:

1. Confirm the alert’s team label. Open Prometheus’s /alerts page. Note the team and service labels.
2. Confirm the catalogue. Look up the service in /etc/services/catalogue.yaml. The catalogue should name the team.
3. Confirm the route. Run amtool config routes show. Find the route that matches the team label. If the route does not exist, the alert falls through to the triage channel.
4. Confirm the dashboard. Open the dashboard_url annotation. The dashboard should live under the team’s Grafana folder.
5. Confirm the runbook. Open the runbook_url annotation. The URL should resolve to a runbook that names the team and the service.
6. Form the diagnosis. Stale catalogue, or missing Alertmanager route, or wrong folder, or missing runbook. Each is a separate fix.

Security implications

Service ownership intersects with security at the credential boundary. The application team that owns the service also owns the service’s secrets: database credentials, API keys, TLS certificates. The secrets are not the platform’s responsibility. The platform provides the secret store (Vault, AWS Secrets Manager, sealed-secrets in Kubernetes); the application team rotates, scopes, and revokes.

The audit confirms: every secret used by the service has a documented owner (the team), a rotation cadence, and a revocation procedure on team-offboarding. A service whose secret has no owner is the same shape as a service whose dashboard has no owner — the configuration exists, the accountability does not.

Performance implications

The service ownership model has performance implications for the catalogue and the rule generation:

• Catalogue size — a 200-service catalogue is a 200-line YAML file; trivial. A 5,000-service catalogue (every microservice in a large platform) is a 5,000-line file; non-trivial. The discipline is to merge services that share an owner and a tier; “checkout-api” and “checkout-worker” owned by the same team and tier-1 can be one entry with two components.
• Rule generation — generating rule files from the catalogue at CI time means a service catalogue change triggers an alert config reload. The reload is hot for Prometheus (SIGHUP) but slow if every rule is regenerated. Validate the generation cost on a copy of the catalogue before shipping.

Production guidance

• Keep the catalogue small. Merge services that share an owner and a tier. The catalogue is a navigation tool, not a service registry.
• Require a live PagerDuty service for every catalogue entry. CI fails the catalogue diff if the PagerDuty service does not exist.
• Audit the four-artefact conjunction quarterly. The audit script is one shell loop; the output is a CSV that goes to the engineering leadership review.
• Treat the catalogue as code. Reviews, version control, and change log entries apply.
• Re-classify the tier annually. A tier-1 service that becomes tier-0 (a payment flow is added) should trigger the tier-0 escalation policy, not the tier-1 one.

Verification

You should now be able to answer:

• What four artefacts prove service ownership in an observability programme?
• What is the most common shape of service ownership in production?
• How does the catalogue drive Alertmanager, Grafana, and Prometheus configuration?
• What is the failure shape when ownership is fragmented across teams?
• How do you audit a service against the ownership checklist?

Quiz