Application SLOs

A checkout service is healthy. The RED panel is green. The USE panel is green. The dependencies are within budget. The on-call looks at the dashboard and asks: “is this service good enough for the user?” The RED panel reports the runtime; the SLO reports the commitment. The two are not the same. A service with 99.5% availability and 250 ms p95 latency is healthy by the runtime metric but failing the SLO if the user-visible commitment is 99.9% availability and 200 ms p95 latency.

The SLO is the contract between the service and the user. The RED metric is the measurement of the contract. The error budget is the currency of the contract. The alert is the check that the contract is being honoured.

What it is

A Service Level Objective (SLO) is a target value of a Service Level Indicator (SLI) over a window of time. The SLI is the metric; the SLO is the target; the window is the period over which the target is measured.

The canonical SLO for an HTTP service has two dimensions:

1. Availability SLO — the fraction of successful requests over the window. The SLI is 1 - (errors / total_requests). The SLO is typically 99.9% (three nines) or 99.95% (three and a half nines) over a 30-day window.
2. Latency SLO — the fraction of requests faster than a threshold. The SLI is requests_below_threshold / total_requests. The SLO is typically 99% of requests < 200 ms or 95% of requests < 500 ms over the same window.

The error budget is the inverse of the SLO. For a 99.9% SLO over 30 days, the budget is 0.1% of requests, or 43.2 minutes of downtime in the 30-day window. The budget is the currency the team spends when the service is slow or unavailable.

The multi-window burn-rate alert is the SLO’s alerting discipline. The alert fires when the budget is being consumed faster than the window allows. The canonical implementation uses two windows: a fast burn (5 minutes) and a slow burn (1 hour). The fast burn catches the acute incident; the slow burn catches the chronic regression.

The reference for the SLO discipline is the Google SRE workbook chapter on SLOs and the multi-window burn-rate alerting pattern. The two together form the operational discipline of the SLO.

Why a sysadmin cares

The single biggest difference between a team that has SLOs and a team that does not is the conversation about reliability. The team that does not have SLOs argues about whether the service is “good enough”. The team that has SLOs references the error budget: the team has consumed 12% of the 30-day budget in 30 minutes; the priority is high.

Three operational problems disappear when SLOs are in place:

1. The “is this incident a page or a ticket?” question. A page is for the SLO violation; a ticket is for the budget trend. The SLO distinguishes the two with the multi-window burn-rate alert.
2. The “how much capacity should we plan for?” question. The SLO sets the upper bound on the capacity. A 99.9% SLO means the team can plan for 0.1% downtime. The capacity plan is the budget, not the runtime.
3. The “should we deploy this week?” question. The error budget is the constraint. If the team has consumed 80% of the budget, the deploy is delayed. The SLO is the prioritisation.

How it works

The mental model is the budget. The team has a budget of failures per 30 days. The SLO is the target; the budget is the spend. The team spends the budget when the service is slow or unavailable.

  30-day window
  +--------------------------------------------+
  |                                            |
  |   budget = 0.1% of requests                |
  |                                            |
  |   consumed = sum(errors) over 30 days     |
  |                                            |
  |   remaining = budget - consumed             |
  |                                            |
  +--------------------------------------------+
        |
        v
  burn rate = consumed / window

The burn rate is the rate at which the budget is consumed. A burn rate of 1.0 means the budget will be exhausted at the end of the 30-day window. A burn rate of 24.5 means the budget will be exhausted in 30 days / 24.5 = 29 hours.

The alert is on the burn rate, not the budget. The alert fires when the burn rate is so high that the budget will be exhausted before the window ends. The canonical thresholds are 14.4x (1-hour burn) and 6x (6-hour burn).

The two SLOs are evaluated separately. The availability SLO is the rate of 5xx errors divided by total requests. The latency SLO is the rate of requests faster than the threshold divided by total requests. The two SLOs share the same error budget; the team spends the budget on whichever dimension is failing.

Under the hood

How to configure it

The SLO is configured in three places: the application SLI, the Prometheus recording rule, and the multi-window alert.

1. The application SLI

The SLI is derived from the RED histogram. The availability SLI is the fraction of HTTP requests with status code below 500. The latency SLI is the fraction of HTTP requests below the threshold.

The application does not emit the SLI directly; the SLI is computed in Prometheus from the RED histogram. The application emits the events; the platform computes the SLI.

2. The Prometheus recording rule

The SLO is computed as a recording rule:

# /etc/prometheus/rules/slo.rules
groups:
- name: slo
  interval: 30s
  rules:
  # Availability SLO burn rate (1h window)
  - record: slo:checkout_availability:burn_rate_1h
    expr: |
      (1 - (
        sum(rate(http_server_request_duration_seconds_count{
          service_name="checkout",
          http_response_status_code!~"5.."
        }[1h]))
        /
        sum(rate(http_server_request_duration_seconds_count{
          service_name="checkout"
        }[1h]))
      )) / (1 - 0.999)

  # Availability SLO burn rate (5m window)
  - record: slo:checkout_availability:burn_rate_5m
    expr: |
      (1 - (
        sum(rate(http_server_request_duration_seconds_count{
          service_name="checkout",
          http_response_status_code!~"5.."
        }[5m]))
        /
        sum(rate(http_server_request_duration_seconds_count{
          service_name="checkout"
        }[5m]))
      )) / (1 - 0.999)

  # Latency SLO burn rate (1h window)
  - record: slo:checkout_latency:burn_rate_1h
    expr: |
      (1 - (
        sum(rate(http_server_request_duration_seconds_bucket{
          service_name="checkout",
          le="0.5"
        }[1h]))
        /
        sum(rate(http_server_request_duration_seconds_count{
          service_name="checkout"
        }[1h]))
      )) / (1 - 0.95)

  # Latency SLO burn rate (5m window)
  - record: slo:checkout_latency:burn_rate_5m
    expr: |
      (1 - (
        sum(rate(http_server_request_duration_seconds_bucket{
          service_name="checkout",
          le="0.5"
        }[5m]))
        /
        sum(rate(http_server_request_duration_seconds_count{
          service_name="checkout"
        }[5m]))
      )) / (1 - 0.95)

The two SLOs are the availability (99.9%) and the latency (95% of requests below 500 ms). The two windows are 1h and 5m. The four recording rules are the input to the multi-window alert.

3. The multi-window alert

The multi-window alert is implemented as two alerts per SLO, OR-ed in Alertmanager:

# /etc/prometheus/rules/slo_alerts.rules
groups:
- name: slo_alerts
  rules:
  # Fast burn: page when budget will exhaust in 2 days
  - alert: CheckoutAvailabilitySLOFastBurn
    expr: |
      slo:checkout_availability:burn_rate_1h > 14.4
        and
      slo:checkout_availability:burn_rate_5m > 14.4
    for: 2m
    labels:
      severity: page
      team: checkout
      slo: availability
    annotations:
      summary: 'Checkout availability SLO: budget will exhaust in 2 days'
      runbook_url: 'https://runbooks.example.com/checkout/slo-availability'

  # Slow burn: ticket when budget will exhaust in 5 days
  - alert: CheckoutAvailabilitySLOSlowBurn
    expr: |
      slo:checkout_availability:burn_rate_6h > 6
        and
      slo:checkout_availability:burn_rate_30m > 6
    for: 5m
    labels:
      severity: ticket
      team: checkout
      slo: availability
    annotations:
      summary: 'Checkout availability SLO: budget will exhaust in 5 days'
      runbook_url: 'https://runbooks.example.com/checkout/slo-availability'

  # Latency fast burn
  - alert: CheckoutLatencySLOFastBurn
    expr: |
      slo:checkout_latency:burn_rate_1h > 14.4
        and
      slo:checkout_latency:burn_rate_5m > 14.4
    for: 2m
    labels:
      severity: page
      team: checkout
      slo: latency
    annotations:
      summary: 'Checkout latency SLO: budget will exhaust in 2 days'
      runbook_url: 'https://runbooks.example.com/checkout/slo-latency'

  # Latency slow burn
  - alert: CheckoutLatencySLOSlowBurn
    expr: |
      slo:checkout_latency:burn_rate_6h > 6
        and
      slo:checkout_latency:burn_rate_30m > 6
    for: 5m
    labels:
      severity: ticket
      team: checkout
      slo: latency
    annotations:
      summary: 'Checkout latency SLO: budget will exhaust in 5 days'
      runbook_url: 'https://runbooks.example.com/checkout/slo-latency'

The four alerts are the multi-window pattern. The fast burn is a page; the slow burn is a ticket. The two windows are AND-ed; the alert fires only when both windows are above the threshold.

The promtool validator confirms the rule syntax:

# SEVERITY: READ-ONLY
promtool check rules /etc/prometheus/rules/slo_alerts.rules

Expected output:

SUCCESS: /etc/prometheus/rules/slo_alerts.rules

How to validate it

Validate that the SLO is live with three checks.

1. The availability burn rate is recorded.

# SEVERITY: READ-ONLY
curl -s 'http://prometheus:9090/api/v1/query?query=slo:checkout_availability:burn_rate_1h'

Expected output:

{"value":[1755000030,"0.85"]}

A burn rate of 0.85 is below the 14.4x threshold; the alert is silent. The dashboard reads the burn rate as a percentage of the page threshold.

2. The latency burn rate is recorded.

# SEVERITY: READ-ONLY
curl -s 'http://prometheus:9090/api/v1/query?query=slo:checkout_latency:burn_rate_1h'

Expected output:

{"value":[1755000030,"1.20"]}

A burn rate of 1.20 is below the page threshold but above the budget index. The team is spending the budget at 1.2x the SLO rate; the budget will be exhausted in 25 days instead of 30.

3. The error budget is tracked.

# SEVERITY: READ-ONLY
curl -s 'http://prometheus:9090/api/v1/query?query=1%20-%20slo:checkout_availability:burn_rate_1h%20%2F%2024'

Expected output:

{"value":[1755000030,"0.965"]}

The 96.5% is the fraction of the 30-day budget remaining. The dashboard reads the remaining budget as a percentage.

How it can fail

Four specific failure shapes appear regularly in SLO deployments:

1. Mis-specified target. The team sets the availability SLO to 99.99% (four nines) for a service that has historically run at 99.5%. The SLO is violated every month. The team ignores the SLO. The alert is silenced. The SLO is dead. Symptom: the SLO alert fires every week; the on-call mutes the channel.
2. Wrong SLI. The team uses the application-side error rate as the SLI. The reverse proxy rewrites 5xx to 200. The SLI understates the error rate by an order of magnitude. The SLO is met every month regardless of the actual failure rate. Symptom: the SLO is met while customers report failures.
3. Single-window alert. The team is using a single-window alert on the 1h burn rate. The alert fires on transient spikes. The on-call mutes the channel. The next real incident is missed. Symptom: the alert fires 30 times a day; the page channel is muted by month-end.
4. Alert routing that ignores the budget. The team routes the fast-burn alert to a page and the slow-burn alert to a page. The on-call is paged for every budget trend. The alert routing ignores the severity of the budget spend. Symptom: the on-call is paged for transient spikes.

How to troubleshoot it

When an SLO alert says something the operator does not believe, the diagnostic order is:

1. Confirm the SLI is the right SLI. The SLI must be the user-visible metric, not the runtime metric. The application-side error rate may differ from the user-visible error rate; the SLI must be the user-visible one.
2. Confirm the target is the right target. The target must be the reachable commitment. A target above 99.99% for a service that runs at 99.5% is aspirational; the SLO is dead.
3. Confirm the multi-window pattern is in place. The single-window alert fires on transient spikes. The multi-window pattern is the filter that suppresses the noise.
4. Confirm the alert routing is on the budget. The fast burn is a page; the slow burn is a ticket. The alert routing must respect the budget spend.
5. Confirm the SLO is reviewed quarterly. The target must be recalibrated as the service improves. The SLO is the reachable commitment; the reachable target changes.

Security implications

The SLO metric is derived from the RED histogram. The same security implications apply: the SLI label set must not include user identifiers. The SLO query is not sensitive on its own, but the error budget spend is information that an attacker could use to time an attack. The error budget status should be restricted to operators with a recorded purpose.

The SLO alert is a page; the page channel is sensitive. The on-call rotation is a security boundary; the alert routing must be enforced at the Alertmanager, not at the team.

Performance implications

The SLO recording rule is cheap. The PromQL evaluation is bounded by the SLI’s underlying metric. The dashboard reads the recording rule.

The multi-window alert is bounded by the SLI. The alert evaluation is every 30 seconds. The cost is negligible.

The dashboard cost is the burn rate panel. The panel reads the recording rule; the query is cheap.

Production guidance

• The SLO is the reachable commitment, not the aspirational commitment. Set the target to the current performance minus the headroom. Recalibrate quarterly.
• The SLI is the user-visible metric, not the runtime metric. The application-side error rate may differ from the proxy-side error rate. The SLI must be the user-visible one.
• The multi-window pattern is the discipline. The single-window alert fires on transient spikes. The multi-window pattern is the filter that suppresses the noise.
• The alert routing respects the budget. The fast burn is a page; the slow burn is a ticket. The severity is the budget spend, not the current state.
• The SLO is reviewed quarterly. The target must be recalibrated as the service improves. The SLO is the reachable commitment; the reachable target changes.
• The SLO is paired with RED. The SLO is the commitment; the RED panel is the runtime. The two are read together.

Verification

You should now be able to answer:

• What is the SLI, the SLO, and the error budget, and how are the three related?
• What is the multi-window burn-rate alert pattern, and why is the single-window alert the failure shape?
• Why is the SLI the user-visible metric, not the runtime metric, and what does the wrong SLI look like?
• Why is the SLO target the reachable commitment, not the aspirational commitment, and what is the consequence of the aspirational target?
• What are the four standard SLOs for an HTTP service, and how are the four paired with the alert routing?

Quiz