Recording Rules for SLOs

At 06:48 a customer reports that the API has been returning 503s for forty minutes. The SLO dashboard shows 99.95 percent availability. The on-call engineer opens the burn-rate alert in Alertmanager; the alert has not fired. The post-mortem reveals the cause: the SLO was computed at panel load time. Every dashboard refresh re-evaluated the availability SLI from scratch; the burn-rate alert read a different expression that took a different rate window; the two disagreed by enough that the alert never crossed the threshold.

The fix is to record the SLI as a Prometheus recording rule. The SLI becomes a first-class metric. The error budget becomes a first-class metric. The burn-rate alert reads the recorded SLI. The SLO dashboard reads the recorded error budget. Every consumer sees the same number.

This is the operational shape of SLO recording rules. An SLO is a contract; the recording rule is the contract enforcement. The team that records the SLI once and reuses it across the dashboard, the alert, and the post-mortem gets the right answer under every incident. The team that recomputes the SLI in every consumer gets the wrong answer at least once per incident.

What it is

A SLO recording rule is the Prometheus implementation of a service-level objective. There are three rules in a working SLO pipeline.

1. The SLI rule. Computes the service-level indicator — the measurable property that tracks whether the service is meeting its objective. For an availability SLO, the SLI is the ratio of successful requests to total requests. For a latency SLO, the SLI is the quantile of the latency distribution against a threshold.
2. The error budget rule. Computes the remaining error budget — the fraction of the SLO window during which the service is allowed to fail. For a 99.9 percent availability SLO, the error budget is 0.1 percent of requests.
3. The burn-rate rule. Computes the rate at which the error budget is being consumed. A burn rate of 1 means the budget is being consumed linearly; a burn rate of 14.4 means the budget for a 28-day window would be exhausted in 48 hours.

All three are recorded as Prometheus time series. The SLO dashboard reads the error budget rule. The burn-rate alert reads the burn-rate rule. The post-mortem reads the SLI rule. Every consumer sees the same metric.

The naming convention from lesson 02 applies. A typical SLO rule stack for an availability SLO:

job:api:requests:error_ratio:rate5m      # SLI
job:api:error_budget:remaining:ratio     # error budget
job:api:slo:burn_rate:5m                 # 5-minute burn rate
job:api:slo:burn_rate:30m                # 30-minute burn rate
job:api:slo:burn_rate:1h                 # 1-hour burn rate
job:api:slo:burn_rate:6h                 # 6-hour burn rate

Each rule is named with the SLO target (here, api) and the operation. A consumer can read the stack and infer the SLO from the names.

Why a sysadmin cares

A SLO is a contract between the team and its users. Three operational consequences follow from recording the SLO as a rule.

1. The SLI is computed once per minute and reused. The dashboard, the alert, and the post-mortem all read the same metric. There is no possibility of disagreement between consumers because there is only one producer.
2. The error budget is a single number. The dashboard shows the remaining budget; the alert fires when the remaining budget is at risk. The on-call engineer does not have to compute the budget from the raw metrics during an incident; the rule has already done it.
3. The burn rate is multi-window. A multi-window burn-rate alert reads the burn rate over a short window (e.g. five minutes) and a long window (e.g. one hour). The short window catches sudden regressions; the long window catches slow burns. The recording rules make both windows cheap to query.

A team that records the SLO as a rule gets a single number that drives the dashboard, the alert, and the post-mortem. A team that recomputes the SLI in every consumer gets disagreement between consumers and a slower mean time to recovery.

How it works

The three rules are layered. The SLI rule reads raw metrics. The error budget rule reads the SLI. The burn-rate rule reads the error budget (or the SLI directly).

   http_requests_total{status=~"5.."}
       |
       |  rate over 5 minutes, ratio to total
       v
   job:api:requests:error_ratio:rate5m       # SLI
       |
       +--->  1 - error_ratio
       |     = error_budget_remaining_ratio
       v
   job:api:error_budget:remaining:ratio      # error budget
       |
       +--->  error_ratio / (1 - SLO_target)
       |     = burn rate
       v
   job:api:slo:burn_rate:5m                  # 5-minute burn rate
   job:api:slo:burn_rate:30m                 # 30-minute burn rate
   job:api:slo:burn_rate:1h                  # 1-hour burn rate
   job:api:slo:burn_rate:6h                  # 6-hour burn rate
       |
       +---> Multi-window burn-rate alert
       +---> Burn-rate dashboard panel
       +---> Post-mortem query

The SLO target is a number between 0 and 1. For a 99.9 percent availability SLO, the target is 0.999. The error budget per unit time is 1 - SLO_target, which is 0.001 (one tenth of one percent).

Availability SLI

The SLI for an availability SLO is the ratio of successful requests to total requests. The Prometheus expression:

sum by (job) (
  rate(http_requests_total{status!~"5.."}[5m])
)
/
sum by (job) (
  rate(http_requests_total[5m])
)

The numerator counts successful requests (status not matching 5xx). The denominator counts all requests. The result is a ratio between 0 and 1.

For a latency SLO, the SLI is the fraction of requests faster than the threshold:

sum by (job) (
  rate(http_request_duration_seconds_bucket{le="0.2"}[5m])
)
/
sum by (job) (
  rate(http_request_duration_seconds_count[5m])
)

The numerator counts requests under 200 milliseconds. The denominator counts all requests. The result is the fraction of requests that met the latency target.

Error budget rule

The error budget is the fraction of the SLO window during which the service is allowed to fail. For a 99.9 percent availability SLO, the budget is 0.1 percent of requests.

The error budget rule reads the SLI and computes the remaining budget:

1 - job:api:requests:error_ratio:rate5m

The result is a number between 0 and 1. A value of 0.999 means the service is meeting the SLO. A value of 0.998 means the service has consumed 0.1 percent of its budget — the SLO is still met, but the budget is being spent.

For a latency SLO, the error budget rule is:

1 - job:api:latency:under_threshold:ratio_rate5m

The threshold is the SLO target (e.g. 200 milliseconds for a 200 ms latency SLO).

Burn rate rule

The burn rate is the rate at which the error budget is being consumed. A burn rate of 1 means the budget is being consumed linearly; at this rate, the budget for a 28-day window would be exhausted in 28 days. A burn rate of 14.4 means the budget would be exhausted in 48 hours. A burn rate of 6 means the budget would be exhausted in roughly 4.7 days.

The burn-rate rule reads the error ratio and divides by the allowed error rate:

job:api:requests:error_ratio:rate5m / 0.001

The 0.001 is the error budget for a 99.9 percent SLO (1 - 0.999 = 0.001). The result is the burn rate.

Multi-window burn-rate alerts read the burn rate over multiple windows:

• 5-minute burn rate catches a sudden regression. The alert fires when the 5-minute burn rate exceeds 14.4.
• 30-minute burn rate catches a moderate regression. The alert fires when the 30-minute burn rate exceeds 6.
• 1-hour burn rate catches a slow burn. The alert fires when the 1-hour burn rate exceeds 4 (the classic Google SRE recommendation).
• 6-hour burn rate catches a chronic issue. The alert fires when the 6-hour burn rate exceeds 3.

The exact thresholds come from the Google SRE workbook. The rule of thumb: at a burn rate of N, the budget for a T-day window is exhausted in T / N days.

How to configure it

A complete SLO rule file for an availability SLO on the API:

# File:        slo/api.slo.recording.yml
# Owner:       team-payments
# On-call:     payments-oncall (PagerDuty schedule: payments)
# Source:      https://github.com/example/prometheus-rules
# Consumers:   alerts/api.alerts.yml, dashboards/api-slo.json
# Last review: 2026-07-15 by alice (PR #1234)
#
# SLO target: 99.9 percent availability over 28 days.
# Error budget: 0.1 percent of requests.
#
# Naming convention: level:metric:operations:rate_window
groups:
  - name: api-sli
    interval: 1m
    rules:
      # SLI: error ratio over 5 minutes.
      - record: job:api:requests:error_ratio:rate5m
        expr: |
          sum by (job) (
            rate(http_requests_total{status=~"5.."}[5m])
          )
          /
          sum by (job) (
            rate(http_requests_total[5m])
          )

      # SLI: error ratio over 1 hour (for the 1h burn rate).
      - record: job:api:requests:error_ratio:rate1h
        expr: |
          sum by (job) (
            rate(http_requests_total{status=~"5.."}[1h])
          )
          /
          sum by (job) (
            rate(http_requests_total[1h])
          )

  - name: api-slo
    interval: 1m
    rules:
      # Error budget remaining.
      - record: job:api:error_budget:remaining:ratio
        expr: 1 - job:api:requests:error_ratio:rate5m

      # 5-minute burn rate (page-worthy threshold: 14.4).
      - record: job:api:slo:burn_rate:5m
        expr: job:api:requests:error_ratio:rate5m / 0.001

      # 1-hour burn rate (page-worthy threshold: 4).
      - record: job:api:slo:burn_rate:1h
        expr: job:api:requests:error_ratio:rate1h / 0.001

The corresponding burn-rate alert:

# File:        alerts/api.alerts.yml
groups:
  - name: api-slo-alerts
    rules:
      # Page-worthy alert: 5-minute burn rate above 14.4
      # AND 1-hour burn rate above 4.
      - alert: APIHighBurnRate
        expr: |
          job:api:slo:burn_rate:5m > 14.4
          and
          job:api:slo:burn_rate:1h > 4
        for: 2m
        labels:
          severity: page
          team: payments
        annotations:
          summary: 'API burn rate above SLO threshold ({{ $labels.job }})'
          description: |
            5m burn rate: {{ with query "job:api:slo:burn_rate:5m" }}{{ . | first | value }}{{ end }}
            1h burn rate: {{ with query "job:api:slo:burn_rate:1h" }}{{ . | first | value }}{{ end }}

The alert fires when both the short-window burn rate and the long-window burn rate are above their thresholds. The short-window condition catches sudden regressions; the long-window condition suppresses noise from momentary blips. This is the Google SRE multi-window burn-rate pattern.

A latency SLO example:

groups:
  - name: api-latency-sli
    interval: 1m
    rules:
      # SLI: fraction of requests under 200 ms over 5 minutes.
      - record: job:api:latency:under_200ms:ratio_rate5m
        expr: |
          sum by (job) (
            rate(http_request_duration_seconds_bucket{le="0.2"}[5m])
          )
          /
          sum by (job) (
            rate(http_request_duration_seconds_count[5m])
          )

  - name: api-latency-slo
    interval: 1m
    rules:
      # Error budget remaining.
      - record: job:api:latency:error_budget:remaining:ratio
        expr: 1 - job:api:latency:under_200ms:ratio_rate5m

      # 5-minute burn rate.
      - record: job:api:latency:slo:burn_rate:5m
        expr: job:api:latency:under_200ms:ratio_rate5m / 0.001

The latency SLO uses le="0.2" to count requests under 200 milliseconds. The _count series counts all requests. The ratio is the fraction of requests that met the latency target.

How to validate it

Four checks.

1. promtool validates the rule files.

promtool check rules /etc/prometheus/rules/slo/api.slo.recording.yml
promtool check rules /etc/prometheus/rules/alerts/api.alerts.yml

Expected: SUCCESS: rules are valid for each file.

2. The SLI rule matches the direct expression.

# Rule output (READ-ONLY).
curl -s 'http://localhost:9090/api/v1/query?query=job:api:requests:error_ratio:rate5m' \
  | jq '.data.result[].value[1]'

# Direct expression (READ-ONLY).
curl -s 'http://localhost:9090/api/v1/query?query=
  sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
  /
  sum by (job) (rate(http_requests_total[5m]))' \
  | jq '.data.result[].value[1]'

Expected: identical values. Any difference means the rule and the direct expression disagree; the rule is wrong.

3. The error budget is in the expected range.

job:api:error_budget:remaining:ratio

Expected: a value between 0.999 and 1.0 in steady state (the service is meeting the 99.9 percent SLO). A value below 0.999 means the service has consumed some of its budget.

4. The burn rate is consistent with the error ratio.

job:api:slo:burn_rate:5m * 0.001 == job:api:requests:error_ratio:rate5m

Expected: an empty result. Any non-empty result means the burn-rate rule does not equal the SLI divided by the budget; the rule is wrong.

How it can fail

Six failure modes.

1. SLI rule is inverted. The numerator and denominator are swapped; the SLI is the failure ratio instead of the success ratio. The error budget appears healthy (close to
   1. when the service is failing. Symptom: the dashboard reads 99.9 percent availability while the customer reports 503s.
2. Error budget rule is inverted. The rule reads error_ratio - 1 instead of 1 - error_ratio. The remaining budget appears negative when the service is healthy. Symptom: the dashboard reads a negative budget; the alert fires continuously.
3. Burn rate window mismatch. The SLI rule uses rate(...[5m]); the burn-rate rule references the SLI and divides by 0.001. The burn rate is the 5-minute burn rate. To get a 1-hour burn rate, the SLI rule must use rate(...[1h]). Symptom: the 1-hour burn rate reads the 5-minute value; the alert fires late.
4. Multi-window alert missing a window. The alert reads only the 5-minute burn rate. The 1-hour burn rate is not checked. Symptom: momentary blips page the on-call engineer; the suppress-from-noise property of the multi-window pattern is lost.
5. SLO target wrong. The burn-rate rule divides by 0.01 instead of 0.001. The burn rate is ten times too low; the alert threshold of 14.4 is effectively 144. Symptom: the alert never fires because the threshold is unreachable.
6. SLI rule fails to evaluate. The source metric is missing or the expression has a typo. The SLI rule produces no series; the error budget and burn rate produce no series; the alert does not fire. Symptom: silent SLO breach.

How to troubleshoot it

When an SLO recording rule produces a value that disagrees with the underlying metric, the diagnosis order matters.

1. Confirm the SLI rule matches the direct expression. Query the rule and the direct expression; compare. A mismatch means the rule’s expression is wrong.
2. Confirm the error budget rule is the right arithmetic. 1 - error_ratio is correct for an availability SLO where the SLI is the success ratio. error_ratio - 1 is wrong; error_ratio alone is the failure ratio, not the remaining budget.
3. Confirm the burn rate divides by the right budget. For a 99.9 percent SLO, the budget is 0.001. For a 99 percent SLO, the budget is 0.01. A common mistake is to divide by the SLO target (0.999) instead of the budget (0.001); the result is the inverse burn rate.
4. Confirm the burn rate window matches the SLI window. The 1-hour burn rate must read an SLI computed over 1 hour, not 5 minutes. The rule file must have a separate SLI rule with rate(...[1h]) for the long-window burn rate.
5. Confirm the multi-window alert reads both windows. The alert expression should and the short-window and long-window burn rates. A missing window means the alert fires on blips.
6. Confirm the source metric is present. The SLI rule depends on http_requests_total (or whatever the underlying metric is). If the metric is missing, the rule produces no output and the alert does not fire.

Security implications

SLO recording rules do not introduce a new attack surface beyond what lesson 01 covers. Two considerations apply.

1. The SLO target is a contract. A rule that hardcodes the target (e.g. 0.001 for 99.9 percent) leaks the target through /api/v1/rules. Treat the rule file as part of the team’s access model.
2. The SLI inherits the source metric’s labels. A rule that pulls a sensitive label (e.g. customer_id) into the SLI exposes the label on every consumer. Treat the SLI as carefully as the source metric.

Performance implications

SLO recording rules are usually cheap. The SLI rule is a single rate() over the source metric; the error budget rule is a subtraction; the burn-rate rule is a division. The total cost is one to three milliseconds per evaluation per rule. The cost driver is the SLI rule’s rate() window: a 1-hour window loads more samples than a 5-minute window.

The rule of thumb: keep the SLI rule’s rate() window the same length as the burn rate it feeds. A 5-minute burn rate needs a 5-minute SLI; a 1-hour burn rate needs a 1-hour SLI. Two SLI rules (one for each window) is the standard pattern.

Production guidance

• Record the SLI once and reuse it. The SLI rule is the source of truth for the SLO. Every consumer (dashboard, alert, post-mortem) reads the same metric.
• Use the multi-window burn-rate pattern. The alert reads both a short-window and a long-window burn rate. The short-window catches regressions; the long-window suppresses noise.
• Cross-check the rule against the direct expression. Every rule change should include a verification step that compares the rule’s output to the direct expression.
• Document the SLO target in the file header. The # SLO target: 99.9 percent availability over 28 days comment is part of the contract.
• Review the SLO quarterly. The target may need to change as the service’s user expectations change. A quarterly review catches a target that is no longer appropriate.
• Alert on the SLI rule failing to evaluate. The burn-rate alert does not fire when the SLI rule has no output. A separate alert on absent(job:api:requests:error_ratio:rate5m) catches the silent failure.

Verification

You should now be able to answer:

• What is the difference between the SLI rule, the error budget rule, and the burn-rate rule, and how do they form a layered pipeline?
• Why does the burn-rate rule’s window have to match the SLI rule’s rate window?
• What is the multi-window burn-rate pattern, and why does it suppress noise from momentary blips?
• What is the right arithmetic for the error budget rule, and what is the failure shape if the rule is inverted?

Quiz