The Operations Layer

The on-call rotation has been paged eleven times in the last hour. Seven of the pages are critical alerts that share the same root cause. Three of the pages are warning alerts that have been firing for three days. One of the pages is for a test alert that was never removed from the routing tree. The on-call engineer is paid to triage pages, not to fix the system. The fourth page is the real one. It is the one that the team will discuss in the post-incident review.

The operations layer is the layer that turns the other five into something a team can run. The technology is necessary; the discipline is sufficient.

What it is

The operations layer is the set of people, processes, and disciplines that operate the stack. In this course:

• On-call rotations. The schedule, the escalation path, the handover.
• Alert quality. The rubric that an alert must pass before it pages someone.
• Runbooks. The documents that the on-call engineer reads when the page arrives.
• Post-incident review. The meeting that produces a remediation backlog.
• Change management. The discipline that owns the deployment cadence.
• Ownership. The name attached to every dashboard, every alert, every runbook.

The operations layer is the only layer that requires the team to change habit. The other layers are configurations. The operations layer is a culture.

Why a sysadmin cares

The operations layer is where the other five layers are either used or abandoned. The five failure shapes:

1. Alert fatigue. The pages fire too often. The on-call engineer mutes the channel. The real alert arrives. The channel is muted. The page is missed.
2. Unowned alerts. The alert fires. The page goes to the rotation. The rotation has no owner. The page is acknowledged. The investigation does not start.
3. Stale runbooks. The runbook documents a fix that no longer works. The on-call engineer follows the runbook. The system does not recover.
4. No post-incident review. The incident is closed. The root cause is not documented. The next incident repeats the same root cause.
5. No change management. Deployments happen when an engineer has time. The rollback path is not documented. The on-call engineer is paged for a deployment that was never reviewed.

The operations layer is the only layer where the team that maintains the discipline is the team that operates the stack.

How it works

The mental model: the page is the start of the investigation, not the end of the triage. The alert must arrive with the context that the on-call engineer needs to start the investigation.

  +------------ symptom alert fires -------------+
  |                                                |
  |  Prometheus rule  ->  Alertmanager  ->  page   |
  |                                                |
  +-----------------------+------------------------+
                          |
                          v
  +------------ on-call engineer receives ---------+
  |                                                |
  |  page includes:                                |
  |    summary                                     |
  |    dashboard URL                               |
  |    runbook URL                                 |
  |    severity                                    |
  |    service + cluster + instance                |
  |                                                |
  +-----------------------+------------------------+
                          |
                          v
  +------------ engineer starts investigation -----+
  |                                                |
  |  1. read runbook                               |
  |  2. read dashboard                             |
  |  3. hypothesis, evidence, test                 |
  |  4. action or escalate                         |
  |  5. document                                   |
  |                                                |
  +-----------------------+------------------------+
                          |
                          v
  +------------ post-incident review --------------+
  |                                                |
  |  1. timeline                                   |
  |  2. contributing factors                       |
  |  3. what went well                             |
  |  4. what went wrong                            |
  |  5. remediation backlog                        |
  |                                                |
  +------------------------------------------------+

The page is the start. The investigation is the work. The post-incident review is the learning.

Under the hood

The five-question alert rubric. Every alert must answer “yes” to all five before it pages someone:

1. Does it fire on a symptom? A user-visible failure or a service-level objective breach. CPU above 80% is not a symptom; checkout error rate above 5% is.
2. Is it actionable? The on-call engineer can take an action that changes the outcome. An alert that just informs is not a page; it is a dashboard.
3. Is it owned? A team or a service is responsible for the alert. The page goes to the owner, not the rotation.
4. Does it link to a runbook? The runbook_url annotation is the on-call engineer’s first action.
5. Is it deduplicated? The alertmanager groups and dedupes the alert. The on-call engineer receives one page, not ten.

The three-question runbook rubric. Every runbook must answer “yes” to all three:

1. Is it current? The commands work against the current system. The version numbers are present.
2. Is it owned? A team or a service is responsible for the runbook. The ownership is documented.
3. Is it tested? The runbook has been executed against a staging environment. The last-tested date is documented.

The post-incident review template. Five sections:

1. Timeline. What happened, when, with what evidence.
2. Contributing factors. The technical, process, and human factors that contributed.
3. What went well. The detection, the response, the tooling that worked.
4. What went wrong. The detection, the response, the tooling that failed.
5. Remediation backlog. The actions, the owners, the dates.

How to configure it

The operations layer is the only layer that is not configured in YAML. The configuration is the document.

Alert ownership document. The right pattern is a single committed document that maps every alert to an owner. The owner is the team that receives the page.

# /etc/operations/alert-ownership.yaml
# This file is the source of truth for alert ownership.
# Changes require a review by the SRE team.

alerts:
  - name: CheckoutHighErrorRate
    owner: payments-team
    escalation: payments-on-call
    severity: critical
    runbook: https://runbooks.internal/checkout-high-error-rate

  - name: PostgresTooManyConnections
    owner: database-team
    escalation: database-on-call
    severity: critical
    runbook: https://runbooks.internal/postgres-connections

  - name: NodeExporterDown
    owner: platform-team
    escalation: platform-on-call
    severity: warning
    runbook: https://runbooks.internal/node-exporter-down

rotations:
  - name: payments-on-call
    schedule: https://schedules.internal/payments
    escalation:
      - tier: primary
        timeout: 5m
      - tier: secondary
        timeout: 10m
      - tier: manager
        timeout: 30m

On-call runbook. The right pattern is a single Markdown file per alert, with a documented structure.

# Runbook: CheckoutHighErrorRate

## What this alert means
The checkout service is returning 5xx for more than 5% of
requests for 2 minutes. The user-visible impact is failed
checkouts.

## First 60 seconds
1. Open the dashboard: https://grafana.internal/d/checkout-prod
2. Check the dependency panel: payment-svc, inventory-svc
3. Check the recent deploys: https://deploys.internal/checkout
4. Check the change log: https://changes.internal/checkout

## If the cause is a recent deploy
1. Roll back the deploy: `argocd app rollback checkout`
2. Confirm the error rate drops within 5 minutes
3. File a post-incident review

## If the cause is a dependency
1. Open the dependency's dashboard
2. Follow the dependency's runbook

## If the cause is unknown
1. Escalate to the secondary on-call
2. File a post-incident review

## Last tested
2026-08-01 by on-call engineer, in staging

## Owner
payments-team

Post-incident review template. The right pattern is a single document per incident, with a documented structure.

# Post-Incident Review: Checkout outage 2026-08-13

## Summary
The checkout service was returning 5xx for 30% of requests
from 03:12 to 03:47. The root cause was a misconfigured
connection pool in the payment-svc dependency.

## Timeline
- 03:12: First failed checkout reported in #support
- 03:14: CheckoutHighErrorRate alert fires
- 03:15: On-call engineer paged
- 03:18: On-call engineer identified the dependency
- 03:22: Database team paged
- 03:31: Connection pool configuration reverted
- 03:47: Error rate returns to baseline

## Contributing factors
- The connection pool change was deployed without a review
- The staging environment does not replicate the production
  connection pool size
- The alert did not include the dependency's health

## What went well
- The alert fired within 2 minutes of the user-visible impact
- The on-call engineer identified the dependency within 4 minutes
- The rollback was effective

## What went wrong
- The change was not reviewed
- The staging environment did not catch the issue
- The alert did not include the dependency's health

## Remediation backlog
- [ ] Add a staging environment that matches the production
      connection pool size (owner: platform-team, due: 2026-08-20)
- [ ] Add a dependency health panel to the checkout dashboard
      (owner: payments-team, due: 2026-08-15)
- [ ] Add a connection pool alert to the payment-svc
      (owner: payments-team, due: 2026-08-20)

How to validate it

READ-ONLY — confirm the alert ownership is current.

grep -lE 'owner:\s*$' /etc/operations/alert-ownership.yaml
# Empty output confirms every alert has an owner.

grep -c 'severity:' /etc/operations/alert-ownership.yaml
# 28 (matches the alert count)

READ-ONLY — confirm the runbook is current.

find /etc/operations/runbooks -name '*.md' | while read rb; do
  last_tested=$(grep -A1 'Last tested' "$rb" | tail -1)
  echo "$rb: $last_tested"
done
# /etc/operations/runbooks/checkout-high-error-rate.md: 2026-08-01

READ-ONLY — confirm the alert passes the five-question rubric.

for alert in $(grep '^  - name:' /etc/operations/alert-ownership.yaml | awk '{print $3}'); do
  has_symptom=$(grep -A20 "name: $alert" /etc/operations/alert-ownership.yaml | grep -c 'symptom: true')
  has_owner=$(grep -A20 "name: $alert" /etc/operations/alert-ownership.yaml | grep -c 'owner:')
  has_runbook=$(grep -A20 "name: $alert" /etc/operations/alert-ownership.yaml | grep -c 'runbook:')
  echo "$alert: symptom=$has_symptom owner=$has_owner runbook=$has_runbook"
done

SERVICE-IMPACT — fire a synthetic alert to validate the delivery path.

amtool alert add \
  alertname=TestAlert \
  service=checkout \
  severity=critical \
  cluster=prod-eu-west-1
# Alert added.

# Wait 30 seconds
sleep 30

amtool alert list
# TestAlert, severity=critical, state=suppressed (active inhibition)

READ-ONLY — confirm the post-incident review was held.

find /etc/operations/post-incident-reviews -name '*.md' -mtime -30 | wc -l
# 4 (matches the number of incidents in the last 30 days)

How it can fail

1. Alert fatigue. The on-call rotation is paged every 10 minutes. The rotation mutes the channel. The real alert arrives. The page is missed.
2. Unowned alert. The alert fires. The page goes to the rotation. The rotation has no owner. The page is acknowledged. The investigation does not start.
3. Stale runbook. The runbook documents a fix that no longer works. The on-call engineer follows the runbook. The system does not recover.
4. No post-incident review. The incident is closed. The root cause is not documented. The next incident repeats the same root cause.
5. No change management. Deployments happen when an engineer has time. The rollback path is not documented. The on-call engineer is paged for a deployment that was never reviewed.

How to troubleshoot it

The diagnostic order for “the alert stack is not helping the operator”:

1. Is the page reaching the rotation? The alertmanager /api/v2/alerts is the source of truth. The PagerDuty UI is the next step.
2. Is the runbook current? The Last tested date is the first check. The version numbers are the next.
3. Is the alert owned? The /etc/operations/alert-ownership.yaml is the source of truth. The owner is the team that receives the page.
4. Is the alert passing the five-question rubric? The audit is the right tool. The output is the backlog.
5. Is the post-incident review being held? The /etc/operations/post-incident-reviews is the source of truth. The review date is the next check.

Security implications

The operations layer is the layer that holds the credentials. The risks:

• PagerDuty credentials. The alertmanager holds the service key. The key is a secret. The key is rotated annually.
• Runbook URLs. The runbook may contain internal-only infrastructure. The runbook is not for the public internet.
• Slack webhook. The alertmanager holds the webhook URL. The URL is a secret. The URL is rotated annually.
• On-call schedule. The on-call schedule is a confidentiality-sensitive document. The schedule is behind authentication.
• Post-incident review. The post-incident review may contain confidential information about the system. The review is behind authentication.

Performance implications

The operations layer is the layer that consumes the engineering time. The four knobs:

• Number of alerts. The right pattern is 20-30 alerts per rotation. More than 50 is a failure mode.
• Mean time to acknowledge. The right pattern is under 5 minutes. The PagerDuty SLA is the contract.
• Mean time to resolve. The right pattern is documented per severity. The SLO is the contract.
• Number of incidents per quarter. The right pattern is a downward trend. The post-incident review is the feedback loop.

Production guidance

• Audit the alert stack quarterly. The five-question rubric is the audit tool. The output is the backlog.
• Test the on-call handover. The handover is the document that the on-call engineer reads. The handover is current.
• Hold the post-incident review. The review is the learning. The remediation backlog is the action.
• Track the remediation backlog. The backlog is the contract. The backlog is closed.
• Document the change management. The change management is the discipline. The change log is the evidence.

Verification

You should now be able to answer:

• What is the five-question alert rubric, and why is it the right tool for the audit?
• Why is the runbook the on-call engineer’s first action?
• What is the difference between a symptom alert and a cause alert in the routing tree?
• Why is the post-incident review the feedback loop, not the blame document?
• What is the failure mode when the alert fires without context?

Quiz