The Operating Model

An incident fires at 03:11. PagerDuty wakes an engineer on the infrastructure rota. They look at the alert: “HTTP 5xx rate elevated on checkout service.” They do not own the checkout service. They do not know who does. They page the platform Slack channel. Twelve minutes elapse before someone from the payments team sees the page and starts investigating. Mean time to acknowledge: 12 minutes. Mean time to engage the right team: 14. Mean time to resolve: 47 minutes. None of those minutes were spent actually fixing the problem.

This is what the observability operating model exists to prevent. The operating model is the assignment of every dashboard, alert, and runbook to a named team, plus the routing rules that put the right team in front of the right page. It is the contract between “the platform collects telemetry” and “a named team is responsible for what the telemetry shows.”

What the operating model is

The observability operating model is a RACI matrix for telemetry. For every observability artefact (dashboard, alert rule, log stream, recording rule, runbook, SLO), the model names:

• Responsible — the team that owns the artefact, writes it, reviews it, and is paged for it.
• Accountable — the team whose budget pays for the underlying service (usually the application team that owns the service the artefact observes).
• Consulted — teams that need to be in the loop when the artefact changes (platform, security, compliance).
• Informed — teams that need to see the output but do not own it (leadership dashboards, status pages).

The matrix is a document. The artefact is the wiring that enforces it: Alertmanager routes, Grafana folder permissions, Loki tenant labels, Prometheus rule-file team label. The matrix without the wiring is aspiration; the wiring without the matrix is chaos.

Why a sysadmin cares

The first incident of every production observability programme is a misrouted page. The team that built the dashboard is on holiday; the team that paged does not own the service; the person who knows the runbook left six months ago. Mean time to resolve triples. The fix is not “another on-call rotation”; the fix is an explicit operating model that names the team and the wiring that routes the page to that team.

The trade-off is real. An explicit model requires the platform team to maintain per-team routing rules, per-team Grafana folders, and per-team contact lists. The cost is paid by the platform team; the benefit is paid by every application team that pages correctly. Without executive sponsorship, the cost often wins.

How it works

The model separates who runs the platform from who uses the platform.

                 Observability Operating Model
                 ---------------------------------

   +-----------------+      +-------------------+
   |  Observability  |      |   Application     |
   |  / Platform     |      |   Teams           |
   |  Team           |      |                   |
   +-----------------+      +-------------------+
            |                       |
            | owns                  | own
            v                       v
   +-----------------+      +-------------------+
   |  Shared Stack   |      |  Per-service      |
   |  - Prometheus   |      |  - Dashboards     |
   |  - Loki         |      |  - Alerts         |
   |  - Tempo        |      |  - Runbooks       |
   |  - Grafana      |      |  - SLOs           |
   |  - OTel Coll.   |      |  - Drop rules     |
   |  - Alloy        |      |                   |
   +-----------------+      +-------------------+
            |                       |
            +----------+------------+
                       |
                       v
              Tenant boundary / RBAC
                  (one per team)

The boundary is tenant isolation (Loki multi-tenant, Prometheus per-team external_labels or Cortex/Mimir tenant) and permission isolation (Grafana folders per team, RBAC binding per team). Without these, “ownership” is a label that no system enforces.

The right approach is “you build it, you run it.” The team that writes the application writes the application dashboards, the application alerts, and the application runbook. The platform team provides the stack, the policies, and the review. Nothing in this model is novel; the discipline is to enforce it.

Under the hood: where ownership is actually stored

How to configure it

Three files encode the operating model. The first is the Alertmanager route, the second is the Prometheus rule-group header, the third is the Grafana folder provisioning.

Alertmanager: route by team label

# /etc/alertmanager/alertmanager.yml
# CONFIGURATION: routes pages by the team label on the alert.
global:
  resolve_timeout: 5m

route:
  receiver: 'default-null'
  group_by: ['alertname', 'team', 'service']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    # Payments team owns checkout, payment-svc, billing-svc.
    - matchers:
        - team = "payments"
      receiver: 'pd-payments-oncall'
      continue: false
      # Tighter repeat because payments pages often indicate
      # revenue impact; the on-call should be re-paged if
      # silence is not declared within an hour.
      repeat_interval: 1h
    # Platform team owns the shared observability stack.
    - matchers:
        - team = "platform"
      receiver: 'pd-platform-oncall'
      continue: false
    # Default catch-all routes anything without a team label
    # to a manual triage queue. Items here are a backlog of
    # artefacts that have not been assigned.
    - matchers:
        - team =~ ".*"
      receiver: 'pd-default-triage'
      continue: false

receivers:
  - name: 'pd-payments-oncall'
    pagerduty_configs:
      - service_key: '<redacted>'
        severity: 'critical'
  - name: 'pd-platform-oncall'
    pagerduty_configs:
      - service_key: '<redacted>'
        severity: 'warning'
  - name: 'pd-default-triage'
    slack_configs:
      - channel: '#observability-triage'
        send_resolved: true

The catch-all with no explicit receiver is a deliberate choice: alerts that match no team label land in #observability-triage, a Slack channel the observability team watches. The backlog of un-owned alerts is itself a metric.

Prometheus: declare the team in the rule group

# /etc/prometheus/rules/payments.yml
# CONFIGURATION: every alert rule and recording rule carries a
# team label so Alertmanager can route by team.
groups:
  - name: payments.slo
    interval: 30s
    rules:
      - alert: Checkout5xxRateHigh
        expr: |
          sum(rate(checkout_http_requests_total{
            service="checkout",status=~"5.."}[5m]))
          /
          sum(rate(checkout_http_requests_total{
            service="checkout"}[5m]))
          > 0.01
        for: 10m
        labels:
          team: payments        # routes to pd-payments-oncall
          severity: critical
          service: checkout
          slo: availability
        annotations:
          summary: 'Checkout 5xx rate above 1% for 10m'
          description: |
            Checkout 5xx rate is {{ $value | humanizePercentage }}
            over the last 5 minutes. Page the payments on-call.
          runbook_url: 'https://runbooks.example.com/payments/checkout-5xx'
          dashboard_url: 'https://grafana.example.com/d/checkout-overview'

The team label on the alert is what Alertmanager matches. A missing team label routes the alert to the triage queue. The discipline is: every rule-group header encodes team, every alert rule preserves that label, every recorder carries it forward.

Grafana: folder-per-team provisioning

# /etc/grafana/provisioning/folders/payments.yaml
# CONFIGURATION: every team gets a folder with explicit
# permissions. Provisioned, not clickable, so the state is
# reproducible from version control.
apiVersion: 1
orgs:
  - orgId: 1
    folders:
      - title: 'team-payments'
        uid: 'team-payments'
        permissions:
          - team: 'payments'
            permission: 2      # Editor
          - team: 'sre-readonly'
            permission: 1      # Viewer

The folder UID matches the Alertmanager receiver name suffix. A grep across /etc/grafana/provisioning should reveal every team folder that has a corresponding Alertmanager route. If the two drift, the operating model is broken.

How to validate it

Validation here is consistency across the three encodings. Three commands confirm the model is wired:

# READ-ONLY. List every Alertmanager route and the team it
# matches. The output should have one entry per team.
amtool config routes show --alertmanager.url=http://alertmanager:9093

# READ-ONLY. Validate every Prometheus rule file and confirm
# each group declares a team label.
promtool check rules /etc/prometheus/rules/*.yml
grep -RH '^  - alert:' /etc/prometheus/rules/ \
  | while read -r line; do
      file=$(echo "$line" | cut -d: -f1)
      # Confirm each alert carries a team label in its labels
      # block. CI rejects rule files that omit it.
      grep -q 'team:' "$file" || echo "MISSING team in $file"
    done

# READ-ONLY. List every Grafana folder with a team prefix and
# confirm each has a permission entry. Missing permissions are
# a configuration drift.
curl -s -u admin:$GRAFANA_PASS \
  http://grafana:3000/api/folders | jq '.[] | {uid, title}'
curl -s -u admin:$GRAFANA_PASS \
  http://grafana:3000/api/folders/team-payments/permissions \
  | jq '.[] | {role, permission}'

Illustrative output of the Alertmanager route check:

$ amtool config routes show --alertmanager.url=http://alertmanager:9093
Routing tree:
  default-route receiver: default-null
  |- matchers: team="payments" receiver: pd-payments-oncall continue: false
  |- matchers: team="platform" receiver: pd-platform-oncall continue: false
  \- matchers: team=~".*"     receiver: pd-default-triage  continue: false

The shape is n+1 routes: one per application team, plus one default. If the count grows past that, a team is missing from the document.

How it can fail

Six failure shapes appear repeatedly when the operating model is implicit rather than wired:

1. Wiki-only ownership. A Confluence page lists every team and every dashboard. The page is six months stale. Three dashboards are owned by ex-employees. Symptom: alerts route to a PagerDuty service that no one is on.
2. No team label on rules. Rule files omit the team label. Alertmanager has no matcher; everything lands in the default triage queue. Symptom: the triage channel has 200 alerts a day and is ignored.
3. Folder sprawl. Every engineer creates a top-level dashboard. There are 800 dashboards and 12 folders. The “payments folder” exists twice, with different permissions. Symptom: a new dashboard is unowned because there is no default folder to inherit from.
4. Default-null catch-all catches everything. Alertmanager has a permissive default route that pages the platform team. Symptom: the platform on-call is paged for every alert, including ones they cannot fix.
5. Cross-team dashboards with no joint owner. A dashboard shows checkout and fraud-detection. Two teams need it. Neither updates it. Symptom: the dashboard is frozen at the version two teams agreed on, then rots.
6. Promotion without ownership migration. A service moves from team A to team B (reorg, split, M&A). The dashboards stay under team A. Symptom: team B pages for an alert they cannot see the dashboard for.

How to troubleshoot it

The diagnostic order is fixed. Start with the artefact that is on fire; trace the owner; confirm the wiring.

1. Confirm the alert fired. Open Prometheus’s /alerts page and locate the firing alert. Note the team label and the service label.
2. Confirm the route. Run amtool config routes show and find the matcher. If the matcher does not exist, the alert will fall through to the catch-all and end up in the triage channel.
3. Confirm the dashboard exists. Open the dashboard_url annotation in the alert. If the URL 404s, the dashboard has been deleted or moved.
4. Confirm the runbook exists. Open the runbook_url annotation. If the URL 404s or returns a generic runbook index, the runbook is missing or generic.
5. Confirm the Grafana permission. Check that the team named in the alert has Editor permission on the folder referenced by the dashboard.
6. Form the diagnosis. The alert lacks a team label, or the Alertmanager route does not match, or the folder is missing permission, or the runbook URL is broken. Each is a separate remediation.

Security implications

The operating model intersects with security at the tenant boundary. Loki’s auth_enabled: true enables multi-tenancy; the tenant ID is derived from the X-Scope-OrgID header set by the gateway (Grafana’s Loki datasource). If the gateway is misconfigured, one team can query another’s logs. Validate the gateway explicitly:

# READ-ONLY. Confirm the gateway sets the tenant header.
curl -sI -H 'X-Scope-OrgID: payments' \
  http://loki-gateway:3100/loki/api/v1/query?query='{service="checkout"}'

The Grafana RBAC for folders is the second security boundary. A Viewer on a folder can see the dashboard and its variables. A folder with Editor permission for “all users” is an exposure. Audit the folder permissions quarterly.

Performance implications

The operating model has performance implications in two places: Alertmanager grouping and Grafana dashboard rendering. Alertmanager groups by ['alertname', 'team', 'service']; if the team label is missing on many alerts, the grouping collapses to alertname alone, which produces more groups and more notifications. The fix is the missing label, not the group_by. Grafana renders one folder at a time; folders with 500 dashboards take longer to load than folders with 50. The fix is to split, not to increase the rendering budget.

Production guidance

• Require a team label in the rule-group header; CI rejects rule files without it.
• Provision Grafana folders per team; never click-create them.
• Route in Alertmanager by team first, then by service; never by individual user.
• Audit the agreement quarterly: every team in the Alertmanager routes should have a Grafana folder and a PagerDuty service.
• Make the operating model visible. A single page listing every team, every Alertmanager route, every Grafana folder UID, and the PagerDuty service is the reference document.

Verification

You should now be able to answer:

• What three roles does the observability operating model separate?
• Where in the configuration is ownership actually stored?
• What is the failure shape when ownership is documented but not wired?
• How does the model appear in Alertmanager, Prometheus, and Grafana, and what happens when the three disagree?
• How do you audit an existing platform against the model?

Quiz