The Error Budget Policy

A team adopted SLOs for a year. They defined SLIs, set targets, built burn-rate alerts, and opened tickets whenever the budget crossed thresholds. At a quarterly review, the director asked what the team had done differently when the budget tightened. The team could not answer — the alerts were firing, the budget was being consumed, the work did not change. The SLOs existed; the policy did not.

An error budget policy is what closes the loop between “the budget is depleting” and “the team’s behaviour changes.”

What it is

An error budget policy is a written agreement, agreed by engineering leadership and the affected teams, that specifies what changes when the 30-day error budget crosses defined thresholds. It converts the SLO from a measurement into a decision rule. Without a policy, the SLO is a chart on a dashboard. With a policy, the SLO is a contract that moves engineering time.

A canonical policy has three thresholds on the 30-day error budget:

• 25% consumed: Slowdown. New feature work pauses for one engineering day per week on reliability. The team writes down the top three risks to the SLO and chooses one to mitigate.
• 50% consumed: Tighter scope. No new features whose rollout increases error budget exposure. New rollouts must include a canary plan and a rollback signal.
• 100% consumed (SLO violation): Release freeze. No new deploys to production until a post-incident review identifies the cause and the team agrees on preventive changes.

The exact thresholds vary — the canonical values are 25/50/100, but the policy must be calibrated to the team’s risk appetite and operational tempo. A team that ships five deploys a day runs at a fundamentally different pace from a team that ships five a quarter; the same thresholds produce different behaviour. The point is that the thresholds are written, agreed, and applied uniformly.

Why a sysadmin cares

The on-call rotation does not set the engineering roadmap. The SLO does not change behaviour on its own. Without a policy, the SLO is a measurement on which no decision is made; the team’s behaviour is unchanged when the budget tightens, and the SLO is a decoration. The policy is what makes the SLO operational: it gives the engineering manager authority to slow down feature work, gives the on-call engineer authority to halt a deploy, and gives the team a single language for discussing reliability versus feature work.

The category of decision the policy enables:

• A feature flag rollout that would consume more budget — hold or proceed?
• A migration that may produce a long-tail error — accept or split?
• An infrastructure change that is high-risk — is the budget large enough to absorb it?

Without the policy, these are political conversations. With the policy, they are checklist items.

How it works

The mechanics are a written document, a recording rule that materialises the consumed-budget percentage, a quarterly review meeting, and a list of who has authority to invoke each threshold.

30-day error budget % consumed

   0%                                             100%
   |-----------------------------------------------|
   |                                               |
   |              normal feature work              |
   |                                               |
   |---------------+----------+--------------------|
                   25%       50%
                   |          |
                   v          v
            slowdown       tighter scope
            1 day/week     no new high-risk
            on reliability rollouts
                   |          |
                   +-----+----+
                         |
                         v
                  release freeze
                  (at 100% consumed)

The consumed-percentage recording rule produces the metric that the dashboard and the policy reviews read. The metric is a Prometheus recording rule that runs every minute and materialises the percentage of the 30-day budget consumed since the start of the 30-day window.

How to configure it

The recording rule, the markdown policy, and the route for the policy-trigger notification. The policy itself is a document, not a config; the recording rule is the automation that keeps the policy honest.

# /etc/prometheus/rules/slo-orders-budget.yml
groups:
  - name: slo.orders.budget
    interval: 1m
    rules:
      # Percentage of the 30-day error budget consumed.
      - record: slo:orders:errors:budget_30d_consumed_pct
        expr: |
          (
            sum(increase(http_requests_total{
              service="orders", code=~"5.."
            }[30d]))
            /
            (
              0.001 * sum(increase(http_requests_total{
                service="orders"
              }[30d]))
            )
          ) * 100

      # Threshold alerts that map to the policy.
      - alert: OrdersBudgetSlowdown
        expr: slo:orders:errors:budget_30d_consumed_pct > 25
        for: 1h
        labels:
          severity: policy-slowdown
          slo: orders-availability
        annotations:
          summary: 'Orders budget: 25% consumed'
          description: 'Slowdown policy applies.'
          runbook_url: 'https://runbooks/slo/policy-slowdown'

      - alert: OrdersBudgetReleaseFreeze
        expr: slo:orders:errors:budget_30d_consumed_pct > 100
        for: 30m
        labels:
          severity: policy-freeze
          slo: orders-availability
        annotations:
          summary: 'Orders budget: 100% consumed'
          description: 'Release freeze applies.'
          runbook_url: 'https://runbooks/slo/policy-freeze'

The accompanying policy document (kept in the team’s runbook repository, not in the Prometheus tree):

# Orders Service Error Budget Policy (v3, last reviewed YYYY-MM-DD)

## Thresholds

- **25% consumed:** Slowdown. One engineering day per week
  is reallocated to reliability work. The team writes down
  the top three risks to the SLO and chooses one to
  mitigate inside the next sprint.
- **50% consumed:** Tighter scope. New rollouts that would
  increase error budget exposure require a written canary
  plan and a documented rollback signal. Existing deploys
  are not paused.
- **100% consumed:** Release freeze. No new deploys to
  production until the post-incident review identifies the
  cause and the team agrees on preventive changes. The freeze
  is lifted when the policy-freeze alert clears for 6
  consecutive hours.

## Authority

- **Engineering manager** can invoke slowdown and tighter
  scope on the team's own services.
- **Release captain** (rotating on-call lead) can invoke
  release freeze for their own shift's services.
- **Cross-team freezes** require the engineering director.

## Quarterly review

Every quarter the team reviews:
- 30-day budget consumption across the quarter.
- The largest single contributors to burn.
- Whether the SLO target is still appropriate.
- Whether the policy thresholds produced the intended
  behaviour.

The output is a written decision: keep the policy as is,
update the thresholds, retire the SLO, or tighten the target.

How to validate it

Verify the recording rule produces a sensible percentage and the threshold alerts fire as expected.

promtool check rules /etc/prometheus/rules/slo-orders-budget.yml
# expected: SUCCESS: 3 rules found

Query the budget metric in Grafana or directly:

slo:orders:errors:budget_30d_consumed_pct
# {service="orders"} 8.4 (illustrative; well inside slowdown threshold)

Synthetic consumption: drive an error rate that would consume 2% of the monthly budget in an hour and confirm the slowdown alert fires within for::

# Lab only. Never run against production.
curl -s http://localhost:9001/inject?service=orders&rate=0.02

# Within 1h, the policy-slowdown alert should be pending
# then active; verify with amtool.
amtool alert query 'severity=policy-slowdown'
# active  OrdersBudgetSlowdown  summary: Orders budget: 25% consumed

For policy governance: confirm the policy document exists in the team’s runbook repo and is reviewed quarterly.

ls docs/runbooks/slo/orders/
# policy.md  fast-burn.md  slow-burn.md
git log --since='3 months ago' --oneline docs/runbooks/slo/orders/policy.md
# expected: at least one commit in the last 90 days (the quarterly review)

The validation is two-axis. The technical axis (recording rule produces 0-100, threshold alerts fire on synthetic consumption) confirms the metrics are right. The governance axis (policy document exists, reviewed quarterly, contains thresholds and authority) confirms the policy is operational.

How it can fail

1. Policy written but never enforced. The document exists; the on-call rotation mutes the threshold alerts because they were not consulted before the document was adopted. Symptom: alert fires, team ignores, no behaviour change. Fix: get explicit sign-off from the on-call rotation before adopting; review enforcement at the quarterly review.

2. Thresholds not calibrated to the team’s tempo. A team that ships five deploys a day treats “release freeze” as normal — the budget breaches every week and the policy is perma-engaged. A team that ships once a quarter treats “slowdown” as a catastrophe. Symptom: thresholds produce noise or are never reached. Fix: choose thresholds from observed burn rate, not from canonical values.

3. increase() over 30d against a counter that resets. The increase() function extrapolates the partial-data estimate at scrape boundaries; a counter reset within the window produces a wrong percentage. Symptom: budget percentage drops sharply after a process restart, then climbs. Fix: use a sub-window (rate() over 30d) and multiply by 30 days; or document the reset behaviour in the runbook.

4. Authority not delegated. The policy says release freeze applies at 100% but only the engineering director can invoke it, and the director is unavailable. Symptom: freeze is declared at 100% and lifted at 110%. Fix: delegate authority to the release captain on shift.

5. Policy applies to feature work but not to infrastructure changes. A team has paused feature deploys at 60% consumed but pushed a database migration that consumed another 40%. Symptom: budget is consumed while the team believed the freeze applied. Fix: the policy must cover all production changes, not only feature deploys.

6. Quarterly review meeting does not happen. The policy is adopted; the quarterly review meeting is on the calendar; the meeting is rescheduled three times and eventually cancelled. Symptom: the policy is treated as fixed doctrine, not as a document that the team regularly tunes. Fix: book the meeting in advance; mark attendance mandatory; rotate the meeting chair.

How to troubleshoot it

When the policy is not producing the intended behaviour:

1. Confirm the threshold alerts fired. Check amtool alert query for severity=policy-* over the relevant window. If no alert fired, the budget did not cross the threshold — the team’s belief is wrong, not the policy.
2. Inspect the budget percentage recording rule. Is the value what the team believes it should be? A wrong value means the metric is wrong, not the policy.
3. Inspect the Alertmanager route. The alert is firing; the receiver is wrong; nobody was notified.
4. Inspect the runbook link. The alert is firing; the runbook describes what to do; the recipient did not read the runbook.

When the team is ignoring the threshold alerts:

1. Talk to the on-call rotation. Why is the alert being muted? Is the policy wrong, or is the policy right but the alerting wrong?
2. Re-read the policy document. Is the policy still appropriate? The product or the team may have changed; the policy must be tuned.
3. Re-run the quarterly review meeting. The meeting is the corrective.

Security implications

The error budget percent consumed is a derived metric; it does not leak request content. The policy document itself is internal to the team. The release freeze authority granted to the release captain is operational, not security-relevant. The PII exposure is the same as the underlying SLI — verify that the recording rule does not aggregate away PII while keeping label cardinality.

Performance implications

One recording rule per SLO that computes the 30-day budget percentage is a moderate load: the increase() range vector spans 30 days at 30s scrape = ~86,400 samples per series at default scrape intervals. On Prometheus 2.55.x with default limits this is comfortable for ~100 SLOs; beyond that, shard the recording rule evaluation.

The dashboard cost is one panel per SLO reading the percentage metric. The recording rule is read directly; no additional computation.

Production guidance

• The policy is a document, not a config. Write it in the runbook repo, get it reviewed by the on-call rotation before publishing.
• Calibrate the thresholds from observed burn, not from a canonical reference. The reference values are good defaults but the team’s own data is better.
• Delegate authority explicitly. “The engineering manager can invoke slowdown” is operational; “the team” is not.
• Book the quarterly review in advance. A meeting on the calendar is a meeting that happens.
• A policy that is invoked once per year is an SLO that is too lax; a policy that is invoked once per sprint is an SLO that is too tight. The right cadence is roughly once per quarter — the policy is engaged but not perma-engaged.

Verification

You should now be able to answer:

• What are the three canonical thresholds of an error budget policy (slowdown, tighter scope, freeze), and what changes at each?
• Why is a written and signed policy necessary rather than a verbal agreement about reliability?
• What is the role of the quarterly budget review in calibrating the policy, and what decisions does it produce?
• What authority must be delegated, and to whom, so the policy can be invoked without escalation?

Quiz