The Operational SLO Practice

An SLO is not a static number; it is a practice. The team designs the SLO once, audits the measurement, observes the burn rate, investigates errors, and revises the target as the operational truth emerges. The discipline repeats weekly.

This lesson walks the SLO lifecycle end-to-end. It is the anchor for Parts XX (alert quality), XXII (SLO-based alerting), and CXIII (documentation/runbooks).

Phase 1: Design

The team chooses:

• A user-visible behaviour (login success, API latency, etc.).
• An SLI measurement (the metric, the formula, the window).
• An SLO target (the lowest number whose violation matters).
• The window (often 30 days; longer windows are difficult to reason about).

Useful design choices:

• Pick the lowest target whose violation matters. A 99.9% SLO is easier to operate than a 99.99% SLO; under-engineering is possible but over-engineering wastes engineering.
• Pair every availability SLO with a latency SLO. The two are independent signals.
• Document the SLO with examples of “this is what a violation looks like.” The on-call engineer should be able to predict the violation.
• Use a recording rule for the SLI. The SLO computed from the metric is stable; the dashboard reads the recording rule.

Phase 2: Implement

The team implements:

• The instrumentation (RED primitives in the service).
• The recording rule.
• The burn-rate alert (Part XXII covers alerts).
• The dashboard (the SLO trend over the window).
• The runbook (what to do when the budget burns).

A canonical recording rule:

groups:
  - name: slo
    interval: 30s
    rules:
      - record: slo:checkout:success:ratio_5m
        expr: |
          sum(rate(http_requests_total{route="/checkout", status!~"5xx"}[5m]))
          /
          sum(rate(http_requests_total{route="/checkout"}[5m]))

The recording rule is the SLI. The recording rule fires alerts. The dashboard reads from the recording rule.

Phase 3: Audit

The team audits:

• Is the metric correct? A load balancer’s request count may not match the application’s; an upstream rewrite may change what gets counted.
• Is the formula right? Off-by-one in the SLO target can burn the budget too fast.
• Are the labels complete? If route=/checkout does not match all paths to the route, the SLO is partial.
• Is the data ingested? A scrape failure on a critical service can produce a false pass or false fail.

The audit runs weekly. The audit is documented. The audit produces a pass/fail with actions.

Phase 4: Operate

The team operates the SLO through:

• Daily review. A 30-second check: is the SLO green? is the budget intact?
• Burn-rate pages. When the burn rate exceeds the threshold, page.
• Post-incident review. After an incident, review the SLO contribution and feed it into the budget.
• Burn-rate framing. Communicate the budget to leadership weekly: “we have X% of the budget intact for the next Y days; here are the open risks.”

The operation is the constant of the SLO discipline. The SLO becomes operational when the team asks and answers the question “is the budget intact?” every week.

Phase 5: Revise

The team revises the SLO when:

• The measurement is wrong. The formula or labels are incorrect; the SLO must be fixed or retired.
• The target is wrong. The team has learned the actual reliability and the original target is too lax or too strict. Revise after sign-off.
• The behaviour changed. The service behaviour changed (new endpoint, retired endpoint, new user flow). The SLO must follow.
• The team is over-/under-performing. If the team is burning less than 1% of the budget in a quarter, the target may be too lax; if it is burning >50%, the target may be too strict (or the team is under-investing in reliability).

A revised SLO is not a signal that the previous one was wrong. The first SLO is a placeholder until operational truth emerges.

When to retire an SLO

An SLO is retired when:

• The service is being deprecated.
• The user-visible behaviour is being replaced by another behaviour; a new SLO replaces it.
• The SLO has been green for years and the team has moved on to a more meaningful SLO.

Retirement is itself a discipline. An SLO without an owner is a wish. An SLO without a successor is a gap.

Production guidance

• Audit SLOs weekly. The audit is a 30-minute exercise that documents the SLO’s correctness, its burn rate, and any revisions.
• Use recording rules to make the SLI stable. The dashboard reads the recording rule; the alert fires on the recording rule.
• Pair every burn-rate alert with a runbook link. Burn-rate alerts that fire without a runbook are operations failures.
• Track the SLO trend over the window. The “is the budget intact?” question is the operational discipline.
• Retire SLOs deliberately. An SLO without an owner is a wish.

Verification

You should be able to answer:

• What are the five phases of the SLO lifecycle?
• Why is the audit cadence part of the discipline?
• When should an SLO be revised or retired?
• What is the operational heart of the SLO practice?

Quiz