Impact

A 03:00 page arrives: OrdersApiHighErrorRate. The what-failed section says the service is returning 5xx above 5% in eu-west-1. The on-call reads the next section, “impact,” and sees the word “users affected.” They read it again. There is no number. They go back to the alert payload; the alert payload carries region=eu-west-1 but does not say how many users are in that region. The on-call is now estimating the impact of an incident they have already been asked to fix.

An impact section is the runbook’s answer to the question every manager, escalation engineer, and post-incident review asks first: how bad is this? The answer must be a number over a window, derived from a telemetry signal, and stated in the same units the alert payload carries. The cost of getting the section right is small. The cost of getting it wrong is paid in missed escalations, misclassified incidents, and incident reviews that cannot reconstruct what actually happened.

What it is

An impact section is a short paragraph that states the user-visible effect of the failure in three concrete shapes:

1. The count — how many requests, users, or transactions are affected. Expressed as an absolute number over a defined window.
2. The ratio — what fraction of total traffic or user base the affected count represents. Expressed as a percentage.
3. The window — the time range the count covers. The window must be tied to a metric and a label set, not to a colloquial phrase like “since the outage started.”

A runbook whose impact section omits any of the three forces the on-call to derive it under time pressure. The derivation is usually wrong.

Why a sysadmin cares

The impact section drives three operational decisions:

1. Severity escalation. A 0.1% ratio for two minutes is a P3. A 30% ratio for thirty minutes is a P0. The numbers are the difference between paging the on-call manager and not.
2. Comms routing. The customer-facing comms template triggers on impact thresholds. A runbook whose impact section is vague forces the comms lead to estimate.
3. Post-incident review. The PIR asks “how many users were affected?” If the runbook impact section already had the number, the answer is one query away. If it did not, the answer is reconstructed from logs weeks later.

The cost of a missing impact section is paid every time the alert fires. The cost of a precise impact section is one extra paragraph of authoring.

How it works

The impact statement derives from the same telemetry that produced the alert. The alert expression is a ratio over a window. The impact section multiplies that ratio by the volume of traffic in the affected scope:

  Alert expression (ratio over a window)
  ---------------------------------------
  sum by (service, region) (
    rate(http_requests_total{service="orders-api", status=~"5.."}[5m])
  )
  /
  sum by (service, region) (
    rate(http_requests_total{service="orders-api"}[5m])
  )
  > 0.05
              |
              | (multiply by request volume in scope)
              v
  Impact statement (count and ratio over a window)
  ------------------------------------------------
  Approximately 1,200 failed checkout requests in eu-west-1
  over the last 5 minutes, which is roughly 7% of orders-api
  traffic in the region (the ratio shown by the alert).

The mapping is the same PromQL the alert uses, plus a volume counter that converts the ratio into a count. The volume counter is typically a separate recording rule or a label selector on the same metric.

How to configure it

The alert rule, with a recording rule that pre-computes the request volume:

groups:
  - name: orders-api.slo
    rules:
      - record: orders_api:request_ratio:5m
        expr: |
          sum by (service, region) (
            rate(http_requests_total{service="orders-api", status=~"5.."}[5m])
          )
          /
          sum by (service, region) (
            rate(http_requests_total{service="orders-api"}[5m])
          )

      - record: orders_api:request_volume:5m
        expr: |
          sum by (service, region) (
            rate(http_requests_total{service="orders-api"}[5m])
          )

      - alert: OrdersApiHighErrorRate
        expr: orders_api:request_ratio:5m > 0.05
        for: 5m
        labels:
          severity: critical
          team: checkout
          service: orders-api
          slo: availability
        annotations:
          summary: 'orders-api 5xx ratio above 5% in {{ $labels.region }}'
          runbook_url: 'https://runbooks.example.com/checkout/orders-api-5xx.html'
          dashboard_url: 'https://grafana.example.com/d/orders-api/orders-api-overview?var-region={{ $labels.region }}&from=now-1h&to=now'

The matching runbook impact section:

# orders-api 5xx ratio above 5%

## What failed

The orders-api service in {{ $labels.region }} is returning HTTP
5xx responses at a ratio above 5% over a rolling 5-minute
window.

## Impact

When the alert fires, the platform reports:

- 5xx ratio: orders_api:request_ratio:5m, currently above 0.05.
- Request volume in scope: orders_api:request_volume:5m, in
  requests per second for {{ $labels.region }}.
- Approximate failed requests per minute: ratio times volume,
  expressed as a count over the same window the alert evaluates.

Severity tier implied by the numbers:

- Ratio above 5% and volume above 100 RPS: P1.
- Ratio above 5% and volume below 100 RPS: P2.
- Ratio above 20% sustained for 10 minutes: P0 regardless of
  volume.

Customer-facing impact: approximately the share of
orders-api traffic in {{ $labels.region }} that fails for the
duration of the alert. Other regions are not affected.

Three shapes present:

• Count is implicit in the request volume metric. The on-call multiplies the ratio by the volume to get a count.
• Ratio is the alert’s underlying value, expressed as the recording rule.
• Window is the 5-minute window the alert evaluates, tied to the same metric name.

The severity tier is a function of the numbers, not an assertion. A P0 at low volume is wrong; the runbook says so.

How to validate it

Three checks, in order. The first two are read-only against the runbook; the third is read-only against the live platform.

# 1. Does the impact section name a metric, a count, and a window?
#    A simple lint that flags runbooks without a numeric statement.
grep -L -E 'ratio|count|volume|RPS' runbooks/checkout/orders-api-5xx.md

Expected output: empty. A file listed here is a runbook whose impact section has no numeric anchor.

# 2. Does the alert's severity label match the impact tier the section implies?
rule='OrdersApiHighErrorRate'
expected_severity='critical'
actual_severity=$(curl -s "http://alertmanager:9093/api/v2/alerts?filter=alertname%3D%22${rule}%22" \
  | jq -r '.[0].labels.severity')
[ "$actual_severity" = "$expected_severity" ] && echo OK || echo MISMATCH

Expected output: OK. A MISMATCH means the alert severity and the runbook’s implied tier disagree; reconcile by adjusting either the rule or the impact section.

# 3. What is the live ratio and volume for the affected region?
curl -sG http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=orders_api:request_ratio:5m' \
  | jq '.data.result[] | select(.metric.region=="eu-west-1")'

curl -sG http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=orders_api:request_volume:5m' \
  | jq '.data.result[] | select(.metric.region=="eu-west-1")'

Expected output during an incident:

{ "metric": {"service":"orders-api","region":"eu-west-1"},
  "value": [1723524870, "0.073"] }
{ "metric": {"service":"orders-api","region":"eu-west-1"},
  "value": [1723524870, "287.4"] }

A ratio of 0.073 and a volume of 287 RPS imply roughly 21 failed requests per second, or about 1,260 failures over the 5-minute window. That is the number the impact section should anchor to.

How it can fail

Six failure modes, each observable:

1. The impact section uses a colloquial phrase for scope. Symptom: “a lot of users,” “many regions,” “the whole world.” Cause: the author wrote the section from memory rather than from a query. Confirm by checking that the section names a label-driven scope.

2. The impact section omits the window. Symptom: the section says “300 errors” without saying over what period. Cause: the author copied a count from a one-off dashboard panel. Confirm by reading the impact section for a duration expression.

3. The impact section contradicts the alert severity. Symptom: the alert fires at severity=critical but the impact section describes numbers that imply a P3. Cause: the alert severity was tightened without updating the doc. Confirm by comparing the runbook’s tier mapping to the alert’s severity label.

4. The impact section is overstated. Symptom: the section says “all checkout traffic is failing” when the alert shows 7% in one region. Cause: the author overstated to motivate the runbook to be read. Confirm by comparing the section’s numbers to the alert expression.

5. The impact section is understated. Symptom: the section says “minor degradation” for an alert that has been firing for 30 minutes at 25% ratio. Cause: the doc was written when the impact was small and never updated. Confirm by comparing the section to the live alert expression.

6. The impact section is not derived from a telemetry signal. Symptom: the section has no metric, no query, and no panel reference. Cause: the doc was written in prose. The on-call cannot confirm or refute the statement; it is an assertion.

How to troubleshoot it

In order:

1. Is there an impact section? grep -L '^## Impact'. A runbook without an impact section is missing the question every consumer asks first.
2. Does the section name a metric, a count, and a window? If any is missing, the section is a heading.
3. Does the alert’s severity label match the tier the section implies? Disagreement is the bug.
4. Do the numbers reconcile to the alert expression? Run the alert’s PromQL in Grafana Explore and compare.
5. Is the scope label-driven or colloquial? A scope of “single region” with {{ $labels.region }} is label-driven. A scope of “a few users” is not.

Security implications

The impact section is not security-sensitive by itself. It becomes sensitive when it includes user identifiers, customer segments, or revenue figures that should not appear in a broadly-shared doc. The discipline is to anchor on counts and ratios over label-driven scopes, not on named customers or named accounts.

A runbook whose impact section says “Customer X is affected” is a doc that should be moved behind the same access controls as the customer data it describes. The simpler path is to state the impact in terms of the affected region’s traffic share, not in terms of named customers.

Performance implications

The impact section is read once per incident. Performance implications are about the time-to-derivation, not the doc size. A precise impact section compresses the time-to-derivation because the on-call can match the doc’s numbers to the alert payload in a single read. A vague impact section extends it because the on-call has to reconcile the doc to the live telemetry before quoting a number.

The cost of writing a precise impact section is five minutes of authoring against a recording rule that already exists. The cost of a vague impact section is paid every incident.

Production guidance

• Anchor the impact section on the recording rule that backs the alert. The alert expression and the impact statement use the same metric.
• Express impact as count, ratio, and window. Three sentences, one paragraph.
• Map the numbers to a severity tier explicitly. The tier must reconcile to the alert’s severity label.
• Tie the scope to alert labels, not to colloquial phrases.
• Run the impact-section lint in CI. A runbook whose impact section has no numeric anchor fails the check.

Verification

• What three shapes must an impact section express?
• Why is “many users affected” an unacceptable scope statement?
• How does the impact section tie back to the alert’s severity label?
• What is the symptom when the impact section is overstated relative to the alert expression?

Quiz