SLO Alerting Risks

A team ran a clean SLO program for a year. SLI was the 5xx rate. SLO was 99.9% availability. Burn-rate alerts worked. The on-call rotation trusted them. Then a correlation-ID rewrite in the access log broke the SLI: requests that previously returned 5xx suddenly returned 200 because the response code was set after the access log. The SLI flipped from “violated” to “compliant” in minutes. The alerting went silent. The service was still broken — the indicator and the reality had diverged.

SLO-only alerting has gaps. The gaps are not failures of the pattern; they are properties of the pattern. The team that recognises the gaps can layer complementary alerting on top of SLOs without losing the SLO benefits.

What it is

SLO alerting risks are the failure modes where the SLO indicator and the production reality diverge. The four most common shapes:

1. Heterogeneous-window averaging. The 30-day SLI averages over a window that includes deploys, traffic patterns, and incident periods the team does not separately control. The SLI looks acceptable while incidents recur.
2. Slow-burn alert shape. The 24h/72h ticket alert catches cumulative burns but does not produce the same urgency as a page. A real incident that the slow-burn alert catches may be left untriaged.
3. Aspirational SLI. The team defines an SLI that approximates (not measures) the user experience. The SLO is met on the indicator while users are unhappy.
4. Uncalibrated SLO. The SLO target is set without reference to the actual current performance. The budget is either exhausted every quarter (target too tight) or never meaningfully engaged (target too lax).

The category is the same in each shape: the indicator and the reality diverge. The fix is complement (saturation alerts, dual alerting, indicator audits) rather than substitute (drop the SLO).

Why a sysadmin cares

SLOs are a brilliant simplification but only when the indicator stays close to the reality. When the indicator drifts, the SLO silently lies — the dashboard is green, the budget looks intact, the team is happy — and the production truth is that users are unhappy, the service is slow, or the infra is saturated. By the time the indicator catches up, the team’s reputation is damaged and the SLO programme is distrusted.

The category of decision the team needs to make:

• The SLO is silent on a problem the team knows about. Is the SLI measuring the right thing?
• The SLO is firing on an apparent problem the team cannot find. Is the SLI measuring the right thing?
• The SLO budget is exhausted every quarter. Is the SLO target realistic?
• The SLO budget is never touched. Is the SLO target useful?

How it works

The four shapes are mechanical, not emotional. Each has a diagnostic symptom.

Indicator                              Reality
-----------                            --------
Heterogeneous-window      : incident during deploy; SLI averages
  SLO                     : over deploys and looks fine
                           : users see "slow Tuesdays"
                           :
Slow-burn alert shape     : cumulative burn tickets opened;
  SLO                     : investigation postponed; SLI does
                           : not match the urgency
                           :
Aspirational SLI          : SLI is the 5xx rate; users see 5xx
                           : as 200 in the access log; SLI
                           : shows compliant
                           :
Uncalibrated SLO          : budget exhausted every quarter;
                           : SLI is met but to a target the
                           : team can never achieve

The mitigation in each case is recognition plus complement:

• Heterogeneous-window averaging: pair the SLI with a per-deploy indicator (e.g. a 5m SLO during the rollout window).
• Slow-burn alert shape: add a saturation alert on the resource that the slow burn is degrading (e.g. CPU pressure on the dependency).
• Aspirational SLI: audit the SLI end-to-end every quarter; correlate the SLI to the user experience (NPS, support tickets, refund rate).
• Uncalibrated SLO: revise the target against observed performance in the quarterly review.

How to configure it

The dual-alerting pattern: the SLO alert for user-visible shape, the saturation alert for resource pressure. Both alerts are needed; neither alone is complete.

# /etc/prometheus/rules/slo-orders-risk-mitigation.yml
groups:
  - name: slo.orders.dual
    interval: 30s
    rules:
      # SLO alert: page on fast burn.
      - alert: OrdersSLOFastBurnPage
        expr: |
          (
            slo:orders:errors:ratio_rate1h > (14.4 * 0.001)
            and
            slo:orders:errors:ratio_rate6h > (6 * 0.001)
          )
        for: 2m
        labels:
          severity: page
          slo: orders-availability
          indicator: slo
        annotations:
          summary: 'Orders SLO burning'
          description: 'User-visible 5xx ratio elevated.'

      # Saturation alert: dependency pressure that will
      # eventually burn the SLO.
      - alert: OrdersDependencySaturation
        expr: |
          (
            rate(orders_payment_dependency_timeout_total[5m])
            /
            rate(orders_payment_dependency_requests_total[5m])
          ) > 0.05
          and
          (
            rate(orders_payment_dependency_requests_total[5m])
            /
            1000
          ) > 1
        for: 10m
        labels:
          severity: ticket
          slo: orders-availability
          indicator: saturation
        annotations:
          summary: 'Orders: payment-svc timeouts rising'
          description: |
            5% of payment-svc calls are timing out. Sustained
            for 10m, this will burn the SLO.

      # Per-deploy indicator: SLI over the rollout window
      # only, raised during deploys.
      - alert: OrdersDeploySLI
        expr: |
          (
            increase(http_requests_total{
              service="orders", code=~"5.."
            }[10m])
            /
            increase(http_requests_total{service="orders"}[10m])
          ) > 0.005
          and on()
          changes(http_requests_total{service="orders"}[10m])
        for: 1m
        labels:
          severity: ticket
          slo: orders-deploy-shield
          indicator: deploy
        annotations:
          summary: 'Orders deploy caused elevated 5xx'
          description: 'Rollout auto-rollback if not cleared.'

      # Uncalibrated SLO guard: cumulative 30-day consumption
      # above a flag before the quarter ends.
      - alert: OrdersBudgetUncalibrated
        expr: slo:orders:errors:budget_30d_consumed_pct > 75
        for: 6h
        labels:
          severity: policy-review
          slo: orders-availability
          indicator: calibration
        annotations:
          summary: 'Orders budget: 75% consumed'
          description: |
            Calibration review needed. Either the SLO target
            is too tight, or the team is consuming
            meaningfully. The policy will engage freeze at
            100%.
          runbook_url: 'https://runbooks/slo/orders-calibration'

How to validate it

For each risk shape, drive a synthetic scenario that the SLO alert would miss. Confirm the complementary alert fires.

# Heterogeneous-window: drive a deploy-time burst that
#   the 30-day SLI averages away.
curl -s http://localhost:9001/inject?service=orders&rate=0.02&mode=burst

amtool alert query 'indicator=deploy'
# active  OrdersDeploySLI
amtool alert query 'indicator=slo'
# (empty; the 30-day SLO has not yet moved)

# Saturation: drive the payment dependency to time out.
curl -s http://localhost:9001/inject?dependency=payment-svc&mode=timeouts

amtool alert query 'indicator=saturation'
# active  OrdersDependencySaturation

# Aspirational SLI: simulate the access-log rewrite by
#   zeroing the 5xx counter for a window.
curl -s http://localhost:9001/inject?service=orders&mode=log_rewrite

amtool alert query 'indicator=saturation'
# active  OrdersDependencySaturation
amtool alert query 'indicator=slo'
# (empty; the SLI is silent)

# Uncalibrated SLO: drive error volume to consume 80% of
#   the 30-day budget within a week.
curl -s http://localhost:9001/inject?service=orders&rate=0.05&duration=7d

amtool alert query 'indicator=calibration'
# active  OrdersBudgetUncalibrated

The validation is four-state: each synthetic scenario exercises one of the four risk shapes. The SLO alert alone should miss all four; the complementary alert should catch each.

How it can fail

1. Saturation alert on the wrong resource. The team writes the saturation alert against CPU on the service’s own host, but the degradation is in the upstream database. Symptom: saturation alert silent when the SLO is burning on the database. Fix: alert on the resource that drives the SLO failure, which may not be on the service’s own host.

2. Per-deploy indicator triggered by routine traffic. The deploy indicator fires on every peak hour because the 5xx ratio naturally spikes during traffic peaks. Symptom: the deploy indicator is noisy; the on-call rotation mutes it. Fix: gate on changes(http_requests_total) to detect the actual deploy signal, not just elevated ratio.

3. Calibration alert ignored. The uncalibrated SLO alert fires; the policy-review route has no receiver; the alert is silently delivered. Symptom: 75% budget consumed, nobody knows. Fix: route the severity: policy-review alert to the engineering manager explicitly.

4. Aspirational SLI not audited. The SLI is the 5xx rate; the access log is rewritten by a deploy; nobody audits the SLI end-to-end for a year. Symptom: the SLI has been lying for a year; the team discovers only when users complain. Fix: SLI end-to-end audit at least quarterly — verify that the response that the user sees matches the metric the alert reads.

5. Slow-burn ticket alert not enrolled in the ticket queue. The 24h/72h ticket alert fires; the ticket queue route does not exist; the alert is silently dropped. Symptom: slow burns accumulate; nothing in the ticket queue mentions them. Fix: every severity: ticket route must have an explicit receiver.

6. Indicator label cardinality explosion. The saturation alert uses labels that include user_agent or instance_ip. Symptom: the rule produces millions of series; Prometheus OOMs. Fix: drop high-cardinality labels with labeldrop before the saturation rule evaluates.

How to troubleshoot it

When SLO-only alerting appears to be missing incidents:

1. Confirm the incident is real, not a misreport. The symptom may be a support escalation, not a service problem. Open the access logs, the trace timeline, the dependency metrics.
2. Inspect the SLI directly. What does the SLI say at the time of the incident? If the SLI is fine, the SLI is not measuring the right thing.
3. Inspect the saturation signals. Are any resources near saturation at the incident time? Often yes — the signal is there, the alert is not.
4. Audit the SLI end-to-end. Trace a request from the user to the metric. Where is the divergence?
5. Inspect the canary indicators. What did the canary say during the deploy that immediately preceded the incident?

When SLO alerts fire on apparent noise:

1. Confirm the SLI is correct. Maybe the SLI is measuring something that does not matter.
2. Confirm the SLO target is appropriate. A 99.99% target on a service whose users are indifferent to 99.9% is not useful.
3. Inspect whether the slow-burn alert is reaching the ticket queue. If the alert is firing and the ticket queue is empty, the route is broken.

Security implications

Saturation alerts can leak resource names or service identifiers through their annotations. The labels on the alert should mirror the labels on the source metrics — no PII escalation, no high-cardinality labels like IP addresses. The audit cadence that catches the aspirational SLI does not require reading request content; sample traces and access logs are sufficient.

Performance implications

Saturation alerts are cheaper than SLO alerts because they do not require the multi-window pattern. One alert per resource is enough. A typical service runs one saturation-alert group per service; the rule-evaluation budget is bounded.

The per-deploy indicator is more expensive — the changes() evaluation spans the deployment window. A team with hundreds of deploys per day should disable the per-deploy indicator and rely on the canary system (Jenkins, Spinnaker, Argo) for the deploy-time signal.

Production guidance

• Recognise the four risk shapes. The SLO alone does not catch each; the complementary alert does.
• Audit the SLI end-to-end every quarter. Trace a request from the user to the metric. The SLI must match the reality it approximates.
• Keep the calibration review quarterly. A team that runs SLOs without calibration produces SLOs that do not match the operational truth.
• Layer saturation alerting on the SLO alerts. The two together catch both the user-visible and the resource-pressure shapes.

Verification

You should now be able to answer:

• What are the four canonical risk shapes where SLO alerting misleads the team?
• What is the dual alerting pattern, and why do SLO and saturation alerts complement each other?
• How does an audit of the SLI end-to-end catch the aspirational SLI failure?
• What is the role of the calibration review in preventing the uncalibrated SLO?

Quiz