Burn Rate 101

A team adopted a 99.9% availability SLO on the orders service. Three weeks in, the user-visible 5xx rate spiked to 1.5% for roughly an hour during a deploy. The 30-day budget is now 13% of a single failure. The single 5xx-panel dashboard had nothing useful on it — the question is not “what is the 5xx rate”, but “how fast is that rate consuming a fixed monthly budget?”. A metric that answered the second question would have paged on this incident automatically. Burn rate is that metric.

What it is

Burn rate is the ratio of the current error rate to the error rate allowed by the SLO. It answers “how fast, relative to what the SLO permits, is the budget being consumed?”. A burn rate of 1.0 means the service is consuming budget exactly as fast as the SLO permits. A burn rate of 10 means the service is consuming budget ten times faster than permitted — at that rate, the 30-day budget is gone in three days. A burn rate of 0.2 means the service is comfortably inside the SLO with budget to spare.

burn_rate = (1 - SLI) / (1 - SLO_target)

For a 99.9% availability SLO (1 - SLO_target = 0.001):

• SLI = 0.999 (SLO-compliant): burn_rate = 0.001 / 0.001 = 1.0
• SLI = 0.99 (10x SLO allowance): burn_rate = 0.01 / 0.001 = 10
• SLI = 0.9 (100x SLO allowance): burn_rate = 0.1 / 0.001 = 100
• SLI = 0.9999 (1/10 SLO allowance): burn_rate = 0.0001 / 0.001 = 0.1

Why a sysadmin cares

Alerts on raw error counts are wrong for almost every service. A service with 100 RPS that has 5 errors an hour looks similar to a service with 100,000 RPS that has 5,000 errors an hour — except the first service is catastrophically unreliable and the second is operating well inside its SLO. Burn-rate alerts collapse both shapes into a number with the same operational meaning. The team pages when the budget is being consumed too fast, regardless of how many requests the service is serving.

The category of incident this prevents is the “slow SLO violation” — a service whose errors are not noisy enough to trigger an alarm individually but which accumulate across weeks into a SLO breach the team notices only when the dashboard goes red at month-end. By that point, the budget is gone.

How it works

The math is simple; the application is what does the work.

Given a 30-day SLO budget, a burn rate of k means that, at the current error rate, the entire 30-day budget will be exhausted in 30 / k days. Two canonical page-worthy thresholds emerge from this:

• k = 14.4 over a 1h window — budget exhausted in 30 / 14.4 = 2.08 days. Catches “burning 2% of the monthly budget every hour”.
• k = 6.0 over a 6h window — budget exhausted in 30 / 6 = 5 days. Catches sustained fast burns that the 1h window might miss.

The derivation of 14.4: a 1h window is one of 720 hours in 30 days. Burning 2% of the monthly budget in that hour means the burn rate is 0.02 / (1/720) = 14.4. The derivation of 6.0: a 6h window is one of 120 six-hour chunks in 30 days. Burning 5% of the monthly budget in that window means 0.05 / (1/120) = 6.0. These specific thresholds appear in the Google SRE Workbook and have become the de-facto standard.

Window                     Burn Rate        Days to exhaust
1h (1/720 of 30d)          14.4x            30 / 14.4 = 2.08
6h (1/120 of 30d)            6.0x           30 / 6    = 5.0
24h (1/30 of 30d)            3.0x           30 / 3    = 10.0
72h (1/10 of 30d)            1.0x           30 / 1    = 30.0

The fast-burn pair (14.4x / 6x) trips a page.
The slow-burn pair (3x / 1x) opens a ticket.

A page-worthy alert is the AND of two windows. The 1h window catches sharp incidents. The 6h window rejects single sharp hours that recover. Both must be true simultaneously, which is what lesson 02 (multi-window burning) formalises.

How to configure it

The minimal SLO recording rule and the page-worthy burn-rate alert. The slo recording rule computes the error ratio; the alert compares it to the threshold.

# /etc/prometheus/rules/slo-orders.yml
groups:
  - name: slo.orders
    interval: 30s
    rules:
      # The SLI: error ratio over 5m, the foundation for the alert.
      - record: slo:orders:error:ratio_rate5m
        expr: |
          sum(rate(http_requests_total{service="orders", code=~"5.."}[5m]))
          /
          sum(rate(http_requests_total{service="orders"}[5m]))

      # Page: burn rate over 14.4x on a 1h window.
      - alert: OrdersSLOFastBurn1h
        expr: slo:orders:error:ratio_rate1h > (14.4 * 0.001)
        for: 2m
        labels:
          severity: page
          slo: orders-availability
        annotations:
          summary: 'Orders SLO burning 14.4x over 1h'
          description: |
            Error rate over the 1h window would consume the
            full 30-day error budget in 2.08 days. Page on-call.
          runbook_url: 'https://runbooks/slo/orders-fast-burn'

      # Page: burn rate over 6x on a 6h window.
      - alert: OrdersSLOFastBurn6h
        expr: slo:orders:error:ratio_rate6h > (6 * 0.001)
        for: 5m
        labels:
          severity: page
          slo: orders-availability
        annotations:
          summary: 'Orders SLO burning 6x over 6h'
          description: |
            Sustained error rate over 6h would consume the
            30-day budget in 5 days.
          runbook_url: 'https://runbooks/slo/orders-fast-burn'

The (14.4 * 0.001) term is (burn_rate_threshold * (1 - SLO_target)). For a 99.9% SLO, that is 14.4 * 0.001 = 0.0144. The alert fires when the actual error ratio exceeds 1.44% over a 1h window — which is 14.4x the SLO allowance.

The Sloth equivalent — a single specification drives the whole bundle:

# /etc/sloth/slo/orders.yml
service: orders
description: 'Orders availability SLO'
slos:
  - name: availability
    objective: 99.9
    description: 'Successful (non-5xx) HTTP responses for /orders/*'
    sli:
      events:
        error_query: |
          sum(rate(http_requests_total{service="orders",code=~"5.."}[{{.window}}]))
        total_query: |
          sum(rate(http_requests_total{service="orders"}[{{.window}}]))
    alerting:
      name: orders-availability
      page_alert:
        labels:
          severity: page
      ticket_alert:
        labels:
          severity: ticket

sloth generate -i /etc/sloth/slo/orders.yml \
  -o /etc/prometheus/rules/slo-generated.yml
promtool check rules /etc/prometheus/rules/slo-generated.yml

How to validate it

Run the recording rule through Prometheus and verify the output series appears with the right shape.

promtool check rules /etc/prometheus/rules/slo-orders.yml
# expected: SUCCESS: 3 rules found

Reload Prometheus and query the recording rule directly. The output should be a single time series with the slo recording name and a value between 0 and 1.

slo:orders:error:ratio_rate5m
# {service="orders"} 0.000342  (illustrative, well inside 99.9%)

slo:orders:error:ratio_rate5m > 0.001
# (empty result: SLI is currently 99.97%, inside SLO)

Simulate an incident by raising the error rate in a lab prometheus and confirm the alert transitions to firing.

# Synthetic error injector for the lab (not production).
curl -s http://localhost:9001/inject?service=orders&rate=0.05

# Watch the alert transition:
amtool alert query --alertmanager.url=http://localhost:9093 \
  'alertname=~"OrdersSLO.*"' \
  | grep -E 'state|summary'
# active  OrdersSLOFastBurn1h  summary: Orders SLO burning 14.4x over 1h

The validation is binary: the alert must transition to firing within the configured for: window when the synthetic error rate is applied at or above the threshold, and it must remain inactive when the error rate is below.

How it can fail

1. Threshold derived against the wrong SLO target. The rule uses (14.4 * 0.001) for a service whose SLO is 99.5%. The threshold under-alerts by 5x. Symptom: SLI falls below 0.005 over 1h and the alert is silent. Fix: parametrise on the actual SLO; do not paste 0.001 from another rule.

2. rate() on a too-short window. A 5m rate() against a service that emits a request every 30 seconds has only 10 samples — noise dominates, the SLI looks jittery, the alert flutters. Symptom: alerts firing and clearing every few minutes for the same incident. Fix: prefer 1h/6h/24h/72h recording rules for the alert path.

3. Recording rule mislabelled. The recording is named slo:orders:success:ratio_rate5m (success ratio) but the alert compares it as > 0.001. A success ratio of 0.998 means 2 errors per 1000 — that is exactly the 99.8% SLI, not 99.9%. The alert fires at the wrong direction. Fix: verify the recording rule produces the error ratio or the success ratio and choose the comparison accordingly.

4. Counter resets confused with rate. A rate() after a counter reset will return a value that includes the reset as a partial-sample. Combined with a burn-rate threshold on a short window, the first scrape after a restart looks like an incident. Symptom: page fires at process startup. Fix: irate() is no better here; use a recording rule with a 1h range and add an up == 1 requirement.

5. Burn-rate alert aligned with deployment cadence. The for: 2m window collides with the 1h record rule at a restart boundary, producing a one-off false positive at every rollout. Symptom: page fires during deploys but not during real incidents. Fix: extend for: or align the recording rule interval with the deploy.

6. Alert group key missing. Without labels: { severity: page, slo: orders-availability }, the route in Alertmanager does not match the SLO escalation path. Symptom: alert routes to the default queue with the wrong runbook. Fix: enforce labels in the rules-review checklist.

How to troubleshoot it

The diagnostic order when a burn-rate alert is not behaving:

1. Is the recording rule loaded? promtool check rules returns SUCCESS and the rule appears in /api/v1/rules.
2. Does the recording rule produce a value? up{job="prometheus"} and slo:orders:error:ratio_rate5m both return rows. The Prometheus targets page shows the orders job as up.
3. Is the alert evaluation firing when expected? amtool alert query for the alertname returns the expected state during the synthetic injection.
4. Is the alert routing correctly? The Alertmanager UI shows the alert in the route. If the route is wrong, fix the label.
5. Is the alert’s for: satisfied? The alert is pending for the duration of for:, not yet firing; this is normal.

If the alert is firing when it should not, the order is the reverse: check the recording-rule value first (is the SLI actually breached?), then check the burn-rate threshold (is it correct for the SLO?), then check for: and routing.

Security implications

Burn-rate rules carry the same security profile as the metrics they read. If http_requests_total is scrapeable only by an authenticated Prometheus, the rule is fine. If the SLI is derived from a metric that itself is privileged (e.g. a per-user counter), the recording rule expansion produces a series with the same label set as the original; the rule does not escalate the privilege but does make the series easier to query. Burn-rate alerts sometimes leak request counts through the description: annotation in Alertmanager — keep summaries generic (“the 30-day error budget would be exhausted in 2.08 days”), not request-shaped.

Performance implications

A recording rule that fires every 30 seconds at 1h, 6h, 24h, 72h windows is roughly four evaluations per series. A fleet of 100 services with one SLO each is 400 evaluations per minute. The TSDB cost per evaluation is dominated by the rate() over the longest window (72h = 8,640 samples per series at 30s scrape). On Prometheus 2.55.x with default limits, this is comfortable; on a single Prometheus with 1000 SLOs it becomes a meaningful CPU load — shard by rule group, or use the federation / remote-write pattern from the recording-rules modules.

The dashboard cost is one panel per SLO reading the 1h recording rule. Use the recording rule; do not let dashboards run rate() directly against raw counters.

Production guidance

• Use Sloth or a Sloth-equivalent for any service whose SLO you intend to keep for more than one quarter. Hand-rolling the rule set is fine for a one-off SLO; replacing it later is expensive.
• Pair every page-worthy alert with a runbook. The runbook_url: annotation is what gets clicked at 03:00.
• Audit the threshold against the SLO target quarterly. A team that tightens an SLO without updating the recording rule generates a stream of pages that nobody trusts.
• Do not mix “success ratio” and “error ratio” recording rules on the same dashboard. Pick one convention; document it.

Verification

You should now be able to answer:

• What is burn rate, mathematically, and what does the value 14.4 mean for a 99.9% availability SLO?
• Why does a single-window burn-rate alert produce both false positives (transient spikes) and false negatives (sustained slow burns)?
• What does Sloth generate from a single SLO specification file?
• Why is the page-worthy threshold 14.4x over a 1h window and not, say, 10x or 20x?

Quiz