Multi-Window Burning · Observability

A team ran a 1h/14.4x burn-rate alert for six months. The page fired every Friday afternoon at 17:30 as the workday traffic dip caused transient spikes in the error ratio. The team ignored the alert by month three. By month six, a real incident — a slow-burning memory leak that consumed 0.3% of the budget per hour for three days straight — was silent because the alert was muted. The single window was both too noisy and too blind.

Multi-window burning is the pattern that fixes both.

What it is

Multi-window burning is the practice of alerting on the AND of two burn-rate windows of different lengths. The short window catches sharp incidents; the long window catches sustained degradation. Both must be true simultaneously. The pattern generalises to four windows: a fast-burn pair for paging, a slow-burn pair for ticketing.

                Short window      Long window       Catches
Fast burn 1:    1h  / 14.4x       AND  6h / 6x       Sharp incidents
                                                  Sustained fast burns
Slow burn 1:    24h / 3x          AND  72h / 1x      Slow cumulative burns
                                                  Monthly-budget bleed

A page-worthy alert is the AND of two burn-rate windows of different lengths. Both windows must be over threshold simultaneously. A short spike causes the 1h window to trip but the 6h window to stay flat — the AND remains false and the team is not paged. A sustained slow burn trips both the 24h window and the 72h window simultaneously even though neither is sharp on its own — the AND is true and the team gets a ticket.

Why a sysadmin cares

This is the difference between a page that the on-call rotation trusts and a page they mute. A single-window alert is wrong twice: it fires on noise the team learns to ignore, and it misses the slow cumulative burns that consume the budget silently. The muted alert then has nothing to say when the real incident arrives.

The pattern matters less for the lucky team that never had a slow burn and never had a transient spike at 17:30. It matters for the team that operates services of any scale, where both shapes appear regularly. Multi-window burning reduces false positives by an order of magnitude in production telemetry and catches the slow burns that single-window alerting misses.

How it works

The mechanics are simple. For each SLO, write four recording rules — one per window — and two alert rules per severity. The recording rule produces a stable series name; the alert reads it.

Recording rules (one per window):
  slo:<service>:errors:ratio_rate1h
  slo:<service>:errors:ratio_rate6h
  slo:<service>:errors:ratio_rate24h
  slo:<service>:errors:ratio_rate72h

Alert rules (two per severity):
  Page:   1h/14.4x  AND  6h/6x
  Ticket: 24h/3x   AND  72h/1x

The recording rule is the SLI rate over a window. The naming convention is fixed: <service>:<slo>:<type>:<aggregation> — for example, slo:orders:errors:ratio_rate1h. The name is human-readable, scrape-independent, and stable across rule edits. A change to the recording rule expression should not change the series name; if it does, dashboards and alerts break.

The alert rule that consumes the recording rule uses a multi-line expression:

(
  slo:orders:errors:ratio_rate1h > (14.4 * 0.001)
  and
  slo:orders:errors:ratio_rate6h > (6 * 0.001)
)

The and operator is a vector match — both sides must produce a sample for the alert series within the evaluation window. The for: interval is satisfied only when the AND remains true for the configured duration. A spike that lasts 90 seconds does not satisfy for: 2m and the alert does not page.

Under the hood

Each window requires its own recording rule because Prometheus cannot read a 1h recording rule at a 6h range vector. The recording rule is computed once and stored with the rule evaluation interval (typically 30s); the alert reads the materialised series. The cost is four recording rules per service per SLO; the benefit is that alert evaluation is O(1) per window — no per-evaluation scan of 1h of samples.

The vector and evaluates the LHS and RHS at the same instant. Both samples must exist (i.e. the recording rule produced a value on the most recent evaluation tick) for the alert to consider the conjunction. If the RHS recording rule is stale or fails to evaluate, the conjunction is silently false; the alert does not fire. This is the right behaviour for a production alert — a missing recording rule is itself a signal worth investigating separately.

The for: interval adds hysteresis. A truly noisy 1h/14.4x that flickers at exactly the threshold clears between ticks; the for: 2m prevents the alert from oscillating between firing and inactive. Standard values: 2m for the 1h page, 5m for the 6h page, 1h for the 24h ticket, 6h for the 72h ticket. The for: scales with the window — a 6h window should not promote an alert that has not held for at least 1h.

How to configure it

The canonical multi-window burn-rate alerting package. Four recording rules, two alert rules per severity. This is the same package that Sloth generates; the file below is the hand-rolled equivalent.

# /etc/prometheus/rules/slo-orders.yml
groups:
  - name: slo.orders.recording
    interval: 30s
    rules:
      - record: slo:orders:errors:ratio_rate1h
        expr: |
          sum(rate(http_requests_total{service="orders", code=~"5.."}[1h]))
          /
          sum(rate(http_requests_total{service="orders"}[1h]))

      - record: slo:orders:errors:ratio_rate6h
        expr: |
          sum(rate(http_requests_total{service="orders", code=~"5.."}[6h]))
          /
          sum(rate(http_requests_total{service="orders"}[6h]))

      - record: slo:orders:errors:ratio_rate24h
        expr: |
          sum(rate(http_requests_total{service="orders", code=~"5.."}[24h]))
          /
          sum(rate(http_requests_total{service="orders"}[24h]))

      - record: slo:orders:errors:ratio_rate72h
        expr: |
          sum(rate(http_requests_total{service="orders", code=~"5.."}[72h]))
          /
          sum(rate(http_requests_total{service="orders"}[72h]))

  - name: slo.orders.alerts
    rules:
      # Page: short AND long window both over threshold.
      - alert: OrdersSLOFastBurn
        expr: |
          (
            slo:orders:errors:ratio_rate1h > (14.4 * 0.001)
            and
            slo:orders:errors:ratio_rate6h > (6 * 0.001)
          )
        for: 2m
        labels:
          severity: page
          slo: orders-availability
        annotations:
          summary: 'Orders SLO: fast burn detected'
          description: |
            Both 1h and 6h burn rates exceed threshold. The
            30-day error budget would be exhausted in 2-5
            days if the burn continues.
          runbook_url: 'https://runbooks/slo/orders-fast-burn'

      # Ticket: sustained slow burn across two long windows.
      - alert: OrdersSLOSlowBurn
        expr: |
          (
            slo:orders:errors:ratio_rate24h > (3 * 0.001)
            and
            slo:orders:errors:ratio_rate72h > (1 * 0.001)
          )
        for: 1h
        labels:
          severity: ticket
          slo: orders-availability
        annotations:
          summary: 'Orders SLO: slow burn detected'
          description: |
            Budget would be exhausted within the 30-day window
            even though no single window is alarming.
          runbook_url: 'https://runbooks/slo/orders-slow-burn'

The (14.4 * 0.001) literals are parametrised on the SLO target. For a 99.5% SLO, they become (14.4 * 0.005) etc. — do not paste the 99.9% numbers across SLOs.

How to validate it

promtool check rules /etc/prometheus/rules/slo-orders.yml
# expected: SUCCESS: 6 rules found

Reload Prometheus and verify all four recording rules materialise:

{slo_record=~"slo:orders:errors:ratio_rate(1h|6h|24h|72h)"}
# Expect 4 series, each with a single value in [0, 1].

Synthetic incident: drive the orders service into a 5% error state and confirm the page-worthy alert fires within for:.

# Lab-only injector. Never run against production.
curl -s http://localhost:9001/inject?service=orders&rate=0.05

# Wait for `for: 2m` to elapse, then inspect:
amtool alert query --alertmanager.url=http://localhost:9093 \
  'alertname=OrdersSLOFastBurn' \
  | grep -E 'state|summary|fired_at'
# active  OrdersSLOFastBurn  fired_at 17:42:18

Synthetic slow burn: drive a 0.3% error rate for several hours and confirm the ticket alert fires while the page does not.

# Lab-only injector.
curl -s http://localhost:9001/inject?service=orders&rate=0.003

# After 6 hours:
amtool alert query 'alertname=OrdersSLOSlowBurn'
# active
amtool alert query 'alertname=OrdersSLOFastBurn'
# (empty)

The validation is two-state: a fast burn produces a page only; a slow burn produces a ticket only; a transient single-window spike produces neither.

How it can fail

Two windows chosen from the same time range. A team defines “1h short AND 1h long” with a 30-minute offset on each. The two windows overlap heavily and the AND is effectively a single window. Symptom: noise reduction is nothing like expected. Fix: choose windows whose ratio is at least 4:1 (the canonical 1h:6h and 24h:72h are 6:1 and 3:1 respectively).
for: interval shorter than the scraper’s data availability. A for: 30s against a recording rule evaluated at interval: 30s evaluates on a single sample. A flaky recording rule triggers an immediate page. Fix: for: of at least 4x the rule interval; for the canonical 1h/6h pair, for: 2m is the floor.
Vector matching across recording rules with different labels. The 1h rule sums over {service="orders"}; the 6h rule sums over {service="orders"}. If a label drift happens (e.g. a deploy adds a region label), the two recording rules produce series with different label sets and the AND returns empty. Symptom: alert goes silent after a label change. Fix: enforce label consistency via a recording-rule lint.
for: on the ticket alert too short. A for: 5m on the 24h/72h AND raises tickets on a transient drift that the budget easily absorbs. Symptom: tickets opened and closed the same day for the same service. Fix: for: 1h minimum, for: 6h preferred for the slow-burn pair.
Recording rule interval misaligned with for:. A 60-second recording interval combined with for: 30s is noise. A 5-minute recording interval combined with for: 1m means the alert only fires when the recording rule has confirmed the burn for two consecutive ticks. Fix: choose interval: 30s and for: 2m for the 1h pair as a default.
Alertmanager route for severity: ticket absent. The ticket alert is correctly firing but Alertmanager has no route for it, so it lands in the default queue with no owner. Symptom: tickets are filed against the wrong team (often the platform team) or never opened. Fix: every severity: ticket label must have a matching Alertmanager route with an explicit receiver.

How to troubleshoot it

When an alert is firing when it should not:

Inspect the recording rules. Query each window independently — what is the 1h ratio? what is the 6h ratio?
Confirm the AND is the cause. If either side is below threshold, the AND is failing as designed; investigate the window that is over threshold.
Check for:. A for: 30s against a noisy ratio will fire on every blip. Extend for:.
Check routing. The alert is firing and routed, but to the wrong team.

When an alert is not firing when it should:

Inspect the SLI directly. The metric on which the recording rule depends is missing or zero — the SLI is probably wrong, not the alert.
Inspect each window independently. A window is below threshold that should be over. The metric on which that window depends may be missing one or more labels.
Check label match. If the two recording rules produce different label sets, the AND is empty.

Security implications

The description: annotation may include the error ratio and the burn-rate threshold. Both are derived metrics; neither leaks request content. The label set on the recording rule is the same as the label set on the source metric — no privilege is escalated. Be careful when the source metric is itself privileged (per-user request counts, for example); that the burn-rate rule does not aggregate away PII while keeping label cardinality high.

Performance implications

Four recording rules per SLO times the number of SLOs is the rule-eval budget. A fleet of 50 services with one SLO each is 200 recording rules per evaluation tick (30s default = ~67,200 evaluations/hour). On Prometheus 2.55.x with default limits this is a moderate load — most teams should be fine. The 72h recording rule is the most expensive per evaluation because the rate() range vector spans 8,640 samples at 30s scrape; this is where shard-by-rule-group pays off.

Use Prometheus 2.55.x’s rule group sharding (/api/v1/rules endpoint) to inspect evaluation latency per group and to find the slowest SLO rule. Anything over 5s is worth reframing.

Production guidance

Pick the four canonical windows (1h, 6h, 24h, 72h). Do not invent local variants unless you have a documented reason; the canonical pattern is what on-call engineers are trained on.
Use Sloth (or a Sloth-equivalent) to generate the rule bundle from one SLO spec file. Hand-rolling is fine once. Two SLOs hand-rolled three times each is a maintainability bug waiting to happen.
The page alert must have a runbook link. The ticket alert must have a route with an explicit owner. Both go in the annotations:.
Audit label consistency on the recording rules at every upstream instrumentation change.

Verification

You should now be able to answer:

Why does a single-window burn-rate alert produce false positives AND false negatives simultaneously?
What does the AND of two windows of different lengths accomplish that a single window does not?
Why are the canonical thresholds 1h/14.4x AND 6h/6x for paging and 24h/3x AND 72h/1x for ticketing?
What is the minimum ratio between the two windows of an AND pair, and what is the symptom if the ratio is insufficient?