SLO/SLI/SLA — the observability-driven targets

SLOs, SLIs, and SLAs are the observability-driven targets. The SLO is the target; the SLI is the measurement; the SLA is the contract. The error budget is the allowed downtime; the burn rate is the consumption rate. This lesson walks the concepts, the SLIs for cluster observability, the error budget, and the production patterns.

The three concepts

flowchart LR
    A[SLO] --> B[Target]
    C[SLI] --> D[Measurement]
    E[SLA] --> F[Contract]
    A --> G[Service]
    C --> G
    E --> G

The SLO is the target; the SLI is the measurement; the SLA is the contract.

The SLI

The SLI is the measurement:

# SLI: request success rate
sum(rate(http_requests_total{status=~"2.."}[5m]))
/
sum(rate(http_requests_total[5m]))

The SLI is a ratio of good events to total events:

SLI: 99.9% (i.e., 999 of 1000 requests succeed)

The SLI is the input for the SLO.

The SLO

The SLO is the target:

SLO: 99.9% availability over a 30-day window

The SLO is the threshold the SLI must meet.

flowchart LR
    A[SLI: 99.95%] --> B[SLO: 99.9%]
    A -->|OK| C[Meeting the SLO]
    A -->|Not OK| D[Violating the SLO]

The SLO is the threshold.

The SLA

The SLA is the contract:

SLA: 99.5% availability over a 30-day window
Refund: 10% of monthly fee if violated

The SLA is the external commitment; the SLO is the internal target. The SLO is stricter than the SLA; the SLA is the buffer.

The error budget

The error budget is the allowed downtime:

SLO: 99.9% availability over 30 days
30 days = 30 * 24 * 60 = 43,200 minutes
Budget: 0.1% of 43,200 = 43.2 minutes

The error budget is the allowed downtime within the SLO window.

The burn rate

The burn rate is the consumption rate of the error budget:

# Burn rate: how fast the budget is consumed
1 - (
  sum(rate(http_requests_total{status=~"2.."}[1h]))
  /
  sum(rate(http_requests_total[1h]))
)
/
(1 - 0.999)

The burn rate is the fraction of the SLO that is consumed per unit time.

Burn rate: 1.0 = consuming the budget at the SLO rate
Burn rate: 2.0 = consuming the budget at 2x the SLO rate
Burn rate: 14.4 = consuming the budget at 14.4x the SLO rate (will exhaust in 2 days)

The burn rate alerts

The burn rate alerts:

# Prometheus alert
- alert: HighErrorBudgetBurn
  expr: |
    sum(rate(http_requests_total{status=~"5.."}[5m]))
    /
    sum(rate(http_requests_total[5m]))
    > (1 - 0.999) * 14.4
  for: 2m
  labels:
    severity: critical
  annotations:
    summary: "Error budget burning at 14.4x; will exhaust in 2 days"

The alert fires when the burn rate is high.

The cluster SLOs

The cluster SLOs:

# API server SLO
SLI: successful API requests / total API requests
SLO: 99.95% over 30 days

# Kubelet SLO
SLI: successful pod starts / total pod starts
SLO: 99.9% over 30 days

# etcd SLO
SLI: successful writes / total writes
SLO: 99.99% over 30 days

The cluster SLOs are the targets for the cluster’s availability.

The error budget policy

The error budget policy:

# Budget consumption thresholds
- 25% consumption: warn the team
- 50% consumption: pause non-critical changes
- 75% consumption: emergency response
- 100% consumption: feature freeze

The policy is the discipline.

The cross-course references

• The Prometheus course (Part LXXXVIII) covers the recording rules.
• The Alertmanager course (Part XCII) covers the alerting.
• The Observability course covers the SLO/SLI/SLA patterns.

Quiz

Production discipline

• Define the SLIs. The measurement.
• Define the SLOs. The target.
• Define the SLAs. The contract.
• Compute the error budget. The allowed downtime.
• Configure the burn rate alerts. The early warning.
• Document the error budget policy. The consumption thresholds.

The SLO/SLI/SLA is the observability-driven discipline. Operating it well is defining the targets, computing the budget, and configuring the alerts.