Example: Error Rate Spike

At 09:42 UTC the Alertmanager fires: CheckoutHighErrorRate above 5% in us-east-1. The 5xx rate has climbed from 0.4% to 8.2% over the last 8 minutes. The synthetic probe shows the same shape. Customer reports in the support channel mention “checkout error” and “card declined” but the language is ambiguous - the same words describe a 5xx from the checkout service and a 4xx from the payment-svc dependency. The on-call engineer has the page in front of them. This is the worked example.

What it is

An error-rate-spike investigation is the six-phase loop applied to a user-visible rise in the 5xx rate of a service. The investigation distinguishes between a regression the team shipped (deploy) and a regression a dependency imposed (dependency outage) because the two have different mitigations.

The signal that distinguishes the two is the change log window. A spike that begins within five minutes of a deploy to the suspect service is, with high probability, the deploy. A spike that begins without a recent change to the suspect service is, with high probability, the dependency.

Why a sysadmin cares

An error-rate spike is the most common page in a Grafana / Prometheus / Loki / Tempo stack. The mitigation is either a rollback or an upstream fix. The investigation must reach the right mitigation in under fifteen minutes; a wrong mitigation (rolling back a deploy that did not cause the spike) loses fifteen minutes of SLO budget and a week of engineering trust.

The worked example matters because the failure modes of an error-rate investigation are subtle. A spike that looks like a deploy may be a dependency. A spike that looks like a dependency may be a deploy with a delayed effect. The discipline is to verify, not to assume.

How it works

The investigation walks the six phases with the worked example running in parallel:

  09:42 UTC  Phase 1: symptom = "checkout 5xx rate 8.2%, SLO 0.5%"
  09:44 UTC  Phase 2: impact  = "~14% error budget burned in 8 min"
  09:46 UTC  Phase 3: hypoth. = "deploy at 09:35 broke card auth
                            retry; OR dependency card-auth-svc
                            is down"
  09:48 UTC  Phase 4: evidence = (metric, logs, change log,
                              dependency health, trace)
  09:56 UTC  Phase 5: test    = deploy confirmed
  10:01 UTC  Phase 6: cause   = deploy regression;
                            mit. = rollback

Phase 1: Define the symptom

The page message names the symptom in plain language. The on-call engineer writes the symptom down so the team has a single source of truth:

Checkout 5xx rate is 8.2% in us-east-1 since 09:34 UTC, against an SLO of 0.5%. Synthetic probe confirms. Customer reports mention “checkout error” and “card declined” but the language is ambiguous.

The symptom is specific: region, time window, magnitude, SLO. The falsifier is built into the wording: if 5xx rate drops below 0.5% for 5 minutes, the symptom resolves.

Phase 2: Quantify the impact

The SLO burn panel says the checkout service has burned 14% of the 30-day error budget in 8 minutes. At this rate, the budget will be exhausted in roughly 57 minutes.

# SEVERITY: READ-ONLY
curl -s 'http://prometheus:9090/api/v1/query?query=slo:checkout_5xx_rate:burn_rate_5m' \
  | jq '.data.result[0].value[1]'

Expected output:

"61.5"

A burn rate of 61.5 means the 30-day budget will be exhausted in 30 / 61.5 days = 11.7 hours. The 14% in 8 minutes is consistent. The urgency is high.

Phase 3: Form a hypothesis

The change log shows a deploy to the checkout service at 09:35 UTC. The deploy message references a change to the card-auth retry logic: the deploy increased the retry count from 2 to 5 and decreased the per-retry timeout from 800 ms to 200 ms.

Two hypotheses are plausible:

H1 (deploy). The deploy at 09:35 changed the card-auth retry logic. The new configuration retries 5 times at 200 ms each, producing 1000 ms of cumulative latency on the dependency. The card-auth dependency is rate-limited at 500 RPS; the increased retries push the dependency above its rate limit; the dependency returns 429; the checkout service surfaces the 429 as a 5xx.

H2 (dependency). The card-auth-svc dependency is down or degraded independently of the deploy. The checkout service retries per its old configuration and surfaces the dependency failure as a 5xx.

The hypotheses have different falsifiers:

F1. If the dependency (card-auth-svc) returns 429s at a rate that aligns with the deploy start (09:35) and the checkout retry count, H1 is confirmed. If the dependency returns 5xx (not 429), H2 is more likely.

F2. If the dependency’s own error rate panel shows no regression, H1 is more likely. If the dependency’s error rate panel shows a regression at the same time as the checkout spike, H2 is more likely.

Phase 4: Find evidence

Four telemetry surfaces, in this order:

1. Service metric. The 5xx rate by status code:

# SEVERITY: READ-ONLY
curl -s 'http://prometheus:9090/api/v1/query?query=sum%20by%20(code)%20(rate(http_requests_total{job%3D%22checkout%22%2Cregion%3D%22us-east-1%22%2Ccode%3D~%22%5B45%5D..%22}[5m]))' \
  | jq -r '.data.result[] | "\(.metric.code)=\(.value[1])"'

Expected output:

"429"="142.3"
"500"="0.8"
"502"="0.2"
"503"="4.1"
"504"="0.1"

The dominant error code is 429 (Too Many Requests), not 5xx server errors. The checkout service is receiving 429s from a downstream. H1 (deploy with retry increase) is consistent with this distribution: the retry storm pushes the dependency past its rate limit, the dependency returns 429, and the checkout surfaces the 429 as a 5xx.

2. Dependency metric. The card-auth-svc rate-limit counter:

# SEVERITY: READ-ONLY
curl -s 'http://prometheus:9090/api/v1/query?query=rate(card_auth_rate_limit_exceeded_total[5m])' \
  | jq -r '.data.result[0].value[1]'

Expected output:

"312.5"

The card-auth-svc is rejecting requests due to its rate limit. The rate-limit events began at 09:35 UTC, matching the deploy start.

3. Logs. The checkout service logs filtered for retry events:

# SEVERITY: READ-ONLY
logcli query '{job="checkout"} |= "card_auth_retry" | line_format "{{.ts}} {{.msg}}"' \
  --since=09:30 --until=09:50

Expected output:

2026-08-13T09:35:01Z card_auth_retry attempt=1 target=card-auth-svc latency_ms=210
2026-08-13T09:35:01Z card_auth_retry attempt=2 target=card-auth-svc latency_ms=198
2026-08-13T09:35:01Z card_auth_retry attempt=3 target=card-auth-svc latency_ms=205
2026-08-13T09:35:01Z card_auth_retry attempt=4 target=card-auth-svc latency_ms=202
2026-08-13T09:35:01Z card_auth_retry attempt=5 target=card-auth-svc latency_ms=199
2026-08-13T09:35:01Z card_auth_retry exhausted target=card-auth-svc error="rate_limited"
... (8 minutes of similar logs)

The logs confirm: the checkout service is retrying 5 times per request, each retry hitting the card-auth dependency, the dependency returning 429 after the rate limit. The deploy’s new retry configuration is the cause.

4. Change log. Confirm the deploy at 09:35 changed the retry configuration:

# SEVERITY: READ-ONLY
kubectl rollout history deployment/checkout -n payments

Expected output:

deployment.apps/checkout with image revision
REVISION  CHANGE-CAUSE
2         deploy at 09:35: increase card-auth retry 2->5, timeout 800ms->200ms
1         initial

The change log confirms: the deploy at 09:35 changed the retry configuration. H1 is consistent with all four pieces of evidence.

Phase 5: Test the hypothesis

H1 predicted: 429s dominate the 5xx mix; the dependency rate limit triggers; the checkout retry storm matches the deploy. The metric, the dependency metric, the logs, and the change log all confirm. H1 is confirmed; H2 is refuted (the dependency is healthy apart from the rate limit, which is caused by H1’s retry storm).

Phase 6: Locate root cause and document

Root cause: the deploy at 09:35 increased the card-auth retry count from 2 to 5 and decreased the per-retry timeout from 800 ms to 200 ms. The retry storm pushed the card-auth dependency past its 500 RPS rate limit; the dependency returned 429; the checkout service surfaced the 429 as a 5xx.

Mitigation: rollback. The 09:35 deploy is reverted via kubectl rollout undo. Checkout 5xx rate returns to 0.4% within 90 seconds.

Runbook log entry, written at 10:05 UTC:

# Runbook: CheckoutHighErrorRate

## Symptom (Phase 1)
Checkout 5xx rate 8.2% in us-east-1 since 09:34 UTC, SLO 0.5%.
Synthetic probe confirms. Customer reports ambiguous.

## Impact (Phase 2)
5-minute SLO burn rate 61.5; ~14% of 30-day error budget in
8 minutes. Budget exhaustion projected in 11.7 hours at
current rate.

## Hypothesis (Phase 3)
H1: deploy at 09:35 changed card-auth retry count 2 to 5 and
per-retry timeout 800 ms to 200 ms; retry storm pushes
card-auth dependency past its 500 RPS rate limit; dependency
returns 429; checkout surfaces 429 as 5xx.
H2: card-auth-svc dependency is degraded independently.
Falsifier F1: if dependency returns 429 (not 5xx) and rate
limit events align with deploy, H1 confirmed. If dependency
returns 5xx, H2 more likely.
Falsifier F2: if dependency error rate panel is healthy, H1
more likely. If dependency error rate panel is degraded, H2.

## Evidence (Phase 4)
- 5xx by code: 429 = 142.3 RPS, 503 = 4.1 RPS, others < 1.
  429 dominates; dependency is rate-limiting, not erroring.
- card-auth rate-limit events: 312.5 events/sec since 09:35.
- checkout logs: 5 retries per request at ~200 ms each,
  exhausted with error rate_limited.
- change log: revision 2 deployed at 09:35 with retry 2 to 5
  and timeout 800 ms to 200 ms.

## Test (Phase 5)
H1 confirmed. H2 refuted (dependency is healthy apart from
the rate limit caused by H1 retry storm).

## Root cause + mitigation (Phase 6)
Root cause: retry configuration regression in 09:35 deploy.
Mitigation: rollback. 5xx returned to 0.4% at 10:02 UTC.

## Follow-up
- Add pre-deploy check: card-auth retry count must be less
  than or equal to (dependency rate limit / expected RPS).
- Add recording rule: checkout_card_auth_retry_total.
- Add alert: card-auth retry exhaustion rate above 5/sec for
  5 minutes (severity: ticket).
- Coordinate with card-auth-svc team to raise rate limit if
  the new retry policy is the long-term intent.

How to configure it

The investigation produces three configuration artefacts. The immediate artefact is the rollback. The follow-up artefacts are a pre-deploy check, a recording rule, and a ticket-class alert.

A pre-deploy check that enforces the retry invariant:

# deploy-checks/checkout-retry-budget.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: checkout-retry-budget-check
spec:
  template:
    spec:
      restartPolicy: Never
      containers:
      - name: check
        image: curlimages/curl:8
        command:
        - sh
        - -c
        - |
          set -eu
          # READ-ONLY: verify retry budget against dependency rate limit
          RETRY_COUNT=$(curl -s \
            'http://config.example.com/checkout/card_auth.retry_count')
          RETRY_TIMEOUT_MS=$(curl -s \
            'http://config.example.com/checkout/card_auth.retry_timeout_ms')
          CARD_AUTH_RATE_LIMIT=$(curl -s \
            'http://config.example.com/card-auth-svc/rate_limit_rps')
          EXPECTED_RPS=$(curl -s \
            'http://metrics.example.com/checkout/expected_rps')
          # Effective retry rate = expected_rps * retry_count
          EFFECTIVE_RATE=$(echo "$EXPECTED_RPS * $RETRY_COUNT" | bc)
          if [ "$(echo "$EFFECTIVE_RATE > $CARD_AUTH_RATE_LIMIT" | bc)" -eq 1 ]; then
            echo "FAIL: effective retry rate ($EFFECTIVE_RATE) > card-auth rate limit ($CARD_AUTH_RATE_LIMIT)"
            exit 1
          fi
          echo "OK: effective retry rate ($EFFECTIVE_RATE) <= card-auth rate limit ($CARD_AUTH_RATE_LIMIT)"

A recording rule that captures the retry exhaustion rate:

groups:
- name: checkout.rules
  interval: 30s
  rules:
  - record: checkout:card_auth_retry_exhausted:rate5m
    expr: sum by (region) (
      rate(checkout_card_auth_retry_exhausted_total[5m])
    )

A ticket-class alert on the recording rule:

  - alert: CheckoutCardAuthRetriesExhausted
    expr: checkout:card_auth_retry_exhausted:rate5m > 5
    for: 5m
    labels:
      severity: ticket
      team: payments
      service: checkout
    annotations:
      summary: 'Checkout card-auth retries exhausted above 5/sec for 5 minutes'
      runbook_url: 'https://runbooks.example.com/checkout/retry-exhaustion'

How to validate it

Validate that the investigation reached phase 6 by replaying each step against the live platform:

# SEVERITY: READ-ONLY
# 1. Confirm the alert rule is back in steady state.
curl -s 'http://prometheus:9090/api/v1/query?query=sum%20by%20(region)%20(rate(http_requests_total{job%3D%22checkout%22%2Ccode%3D~%22%5B45%5D..%22}[5m]))%20%2F%20sum%20by%20(region)%20(rate(http_requests_total{job%3D%22checkout%22}[5m]))' \
  | jq '.data.result[] | "\(.metric.region)=\(.value[1])"'

Expected output:

us-east-1=0.004
us-west-2=0.004
eu-west-1=0.004

# SEVERITY: READ-ONLY
# 2. Confirm the rollback completed via the change log.
kubectl rollout history deployment/checkout -n payments

Expected output:

deployment.apps/checkout with image revision
REVISION  CHANGE-CAUSE
3         deploy at 09:35 reverted (rollback)
2         deploy at 09:35 (reverted)
1         initial

# SEVERITY: READ-ONLY
# 3. Confirm the recording rule is live and returning 0.
curl -s 'http://prometheus:9090/api/v1/query?query=checkout:card_auth_retry_exhausted:rate5m' \
  | jq '.data.result'

Expected output:

{"metric":{"region":"us-east-1"},"value":[1747215000,"0"]}

The four checks confirm: the symptom has resolved (5xx rate back to SLO), the cause has been removed (deploy reverted), and the follow-up signal is live (recording rule returns 0).

How it can fail

Five specific failure shapes for an error-rate-spike investigation:

1. Mitigating on H1 without testing H2. The on-call sees the deploy at 09:35 and rolls back without testing the dependency health. The dependency is actually down; the rollback does not improve the 5xx rate. The 5xx rate is now both deploy- and dependency-driven. Symptom: the rollback completes; the 5xx rate stays elevated; the on-call is confused.

2. Confusing 4xx and 5xx. The customer reports mention “card declined” - a 4xx language. The on-call filters for 5xx in the metric and concludes the customers are describing something else. The actual cause is a 5xx that the customers are interpreting as a decline. Symptom: the on-call dismisses the customer reports; the actual cause is not investigated.

3. Aggregating all error codes into one panel. The 5xx panel shows 8.2% but does not show that 142 RPS are 429s and 4 RPS are 503s. The on-call assumes a 5xx and investigates the wrong cause. Symptom: the metric by status code (which should be the first panel) is not consulted; the investigation takes 30 minutes longer.

4. Skipping the dependency health panel. The on-call assumes the deploy is the cause and rolls back. The dependency (card-auth-svc) is independently degraded; the rollback does not fix the issue. Symptom: the dependency dashboard was never opened; the rollback was the only action taken.

5. Following the customer language literally. “Card declined” is a customer phrase; the underlying error may be 5xx (the dependency returning 5xx) or 4xx (the dependency returning a decline code). The on-call treats “card declined” as 4xx and does not investigate. The actual cause is a 5xx. Symptom: the on-call dismisses the spike because “customers are just seeing legitimate declines”; the 5xx rate is not investigated.

How to troubleshoot it

When the error-rate-spike investigation is taking longer than expected, the diagnostic order is:

1. Confirm the symptom is still observable. Has the 5xx rate dropped back to SLO? If yes, the spike may be transient and the investigation may not need to continue.
2. Check the distribution by code. If 429 dominates, the dependency is rate-limiting; check the dependency’s rate limit counter. If 5xx dominates, the dependency is erroring; check the dependency’s error rate.
3. Read the change log. The deploy window is the first thing to confirm; the second is the dependency’s own change log.
4. Check the dependency health panel. The dependency’s own 5xx rate, rate-limit events, and saturation metrics are the next evidence surface.
5. Pull a trace of one failed request. The trace localises the failure to a span and to a dependency call. A trace of a single 429 shows which dependency returned the 429.

Security implications

The error-rate investigation may surface per-user information in trace spans and logs. The trace of a single failed checkout may include the user ID, the session ID, and (in extreme cases) the card auth token. Limit trace query access to operators with a recorded purpose; audit the queries.

The logs of the card-auth dependency may include the card auth response codes; these are not PII but they are sensitive enough that the log retention policy must enforce the aggregated metric path before the log lines are stored.

Performance implications

The investigation queries are bounded by the 5-minute rate window and the cardinality of the error code label. The metric is cheap; the trace query is bounded by the sampling rate. The dependency metric is a counter that is incremented per rate-limit event; the rate query is cheap.

The pre-deploy check is a single HTTP fetch per service at deploy time; the cost is negligible. The recording rule evaluates every 30 seconds against the retry exhaustion counter; the cost is negligible on a healthy platform.

Production guidance

• The error-rate spike has two plausible hypotheses (deploy vs dependency). Test both before mitigating.
• The first metric to open is error rate by code. The distribution by code is the cheapest falsifier.
• The dependency health panel is the second surface. The dependency may be healthy, rate-limited, or erroring; each has a different mitigation.
• Roll back before investigating when the SLO burn rate exceeds 10x and the change log shows a deploy within the last hour. The change log is the cheapest evidence.
• The follow-up is configuration, not just a ticket. The pre-deploy check, the recording rule, and the alert are the artefacts that prevent the regression from shipping again.
• Coordinate with the dependency team. The fix may require raising the dependency rate limit; that is a separate conversation with the dependency owners.

Verification

You should now be able to answer:

• Which metric is the cheapest falsifier between a deploy hypothesis and a dependency hypothesis, and why?
• Why is the 5xx-by-code distribution consulted at phase 4 rather than the 5xx aggregate?
• What is the second piece of evidence to consult after the service metric, and what does it show?
• Why is the follow-up alert at severity: ticket and not at severity: page?
• Why does coordination with the dependency team belong in the follow-up rather than in the mitigation?

Quiz