SLO Iteration

A team adopted an SLO on the orders service in 2024. The target was 99.9%. By 2025, every quarter had ended with 100% budget consumed and a release freeze in place. The team believed the SLO was working. The team was wrong — the SLO was calibrated to a target the service could not meet, and the budget was being consumed by ordinary incidents that the team could not control. The SLO was neither motivating the team nor protecting the user.

SLOs that are not iterated become rituals. SLOs that are iterated become operational instruments.

What it is

SLO iteration is the discipline of revising the SLO as the service, the team, and the operational truth evolve. The SLO is not a static contract; it is a living document that the team updates when the underlying reality changes. The four phases are:

   Design          Instrument         Observe          Iterate
   ----            ----------         -------          --------
   Pick SLI        Add metric         Collect data     Revise
   Pick target     Add recording      Audit indicator  target
   Pick window     Add alert          Review budget    Retire
   Pick owner      Add dashboard      Talk to users    Tighten

Each phase produces artefacts that the next phase consumes:

• Design produces an SLO specification (SLI, target, window, owner).
• Instrument produces a recording rule, an alert rule, a dashboard, a runbook.
• Observe produces a quarterly review (budget, indicator audit, customer feedback).
• Iterate produces an updated SLO specification — same shape, revised values, or retirement.

A team that runs all four phases quarterly keeps the SLO aligned with the operational truth. A team that runs only Design and Observe produces an SLO that drifts.

Why a sysadmin cares

The SLO target is a measurement of what users care about. When the service changes — new features, new traffic, new dependencies — the target that was right six months ago is no longer right. The team that does not iterate the SLO either:

• Holds an SLO that the service routinely violates, which trains the team to ignore the dashboard and the alerts.
• Holds an SLO the service routinely exceeds, which trains the team to under-invest in reliability when it would be useful.

The middle is what the team wants: an SLO the service occasionally challenges, never breezes through, and always respects. The middle is what iteration produces.

The category of decision the team needs to make every quarter:

• The SLO target held at 99.9% for two years. The service has been at 99.95% for four consecutive quarters. Is the target appropriate?
• The SLO target held at 99.9% for two years. The service has been at 99.85% for four consecutive quarters. Is the target appropriate?
• The SLI is the 5xx rate. The product pivoted to a new business model where the user-facing failure is the abandoned-cart rate, not the 5xx rate. Is the SLI still measuring the right thing?

How it works

Iteration is a meeting, not a feature. The quarterly review brings together the SLI/SLO owner, the on-call rotation representative, the engineering manager, and (when the SLO touches customer experience) a product manager. The meeting runs through three review topics and produces one of four outcomes:

Quarterly review
  +-- 1. Budget consumption
  |     - What % of the 30-day budget was consumed in
  |       each of the last three months?
  |     - Which incidents drove the consumption?
  +-- 2. Indicator audit
  |     - Does the SLI still measure what the user
  |       experiences?
  |     - Did the metric definitions change?
  +-- 3. Customer feedback
        - What did users say in NPS?
        - What did support tickets cluster on?
        - What was the refund or churn rate?
        |
        v
   Outcome:
   - keep as is
   - tighten the target
   - loosen the target
   - retire the SLO

The outcome is recorded. The SLO specification file is updated. The recording rule and the alert rule are updated to match. The dashboard is updated. The runbook is updated. The team is briefed.

How to configure it

The iteration is a meeting; the documentation is a file. The template below is the SLO specification file the team maintains across iterations. The team’s SLO repo can be a standalone directory or a section of the runbook repo.

# Orders Service SLO — v5 (effective 2026-08-01)

## SLO target

- **SLI:** `non-5xx_responses / total_responses` over 30d
- **Target:** 99.9% availability
- **Window:** 30 days
- **Owner:** Orders team

## Iteration history

### v5 (2026-08-01) — current
- Previous: 99.9% target (held since 2024-04-01).
- Quarterly review 2026-Q2:
  - Budget consumed: 18% / 22% / 16% across the quarter.
  - Indicator audit: SLI matches the user experience
    (verified by sampling 200 5xx responses and confirming
    each was a real failure, not an artefact of the access
    log rewrite).
  - Customer feedback: NPS stable at 42; support tickets
    cluster on a different bug class (cart abandonment),
    not on availability.
- Decision: keep the target.

### v4 (2024-04-01)
- Previous: 99.95% target (held since 2022-Q4).
- Quarterly review 2024-Q1:
  - Budget consumed: 105% / 110% / 98% across the quarter.
  - Indicator audit: SLI matches the user experience.
  - Customer feedback: NPS stable at 44; no support-ticket
    clustering on availability.
- Decision: loosen the target from 99.95% to 99.9%. The
  service was reliably at 99.92%; the policy was engaging
  every quarter; the team was treating the alerts as
  noise. Loosening by 0.05% left headroom for ordinary
  incidents.

### v3 (2022-Q4)
- Previous: 99.9% target (held since 2021-Q2).
- Decision: tighten the target from 99.9% to 99.95%.
  Customer feedback showed an uptick in abandoned-cart
  complaints that correlated with 0.05% of additional
  availability loss. The team had headroom; tightening
  was meaningful.

### v2 (2021-Q2)
- Previous: 99% target (initial adoption).
- Decision: tighten the target from 99% to 99.9%. The
  service was reliably at 99.95%; the target was too lax
  to drive useful alerting.

### v1 (2020-Q4)
- Initial adoption. SLI, target, window, owner defined.

## Quarterly review cadence

- Review meeting: first Wednesday of each quarter, 14:00.
- Attendees: SLO owner, on-call rep, engineering manager,
  product manager (when customer-impact changes are
  considered).
- Output: a written decision (keep / tighten / loosen /
  retire) recorded in this file within 5 business days.

The recording rule is unchanged across versions; only the threshold and the target change. The version history is in the SLO specification file, not in the recording rule.

# /etc/prometheus/rules/slo-orders.yml
# Threshold parametrised by $(slo_target):
groups:
  - name: slo.orders.recording
    interval: 30s
    rules:
      - record: slo:orders:errors:ratio_rate1h
        expr: |
          sum(rate(http_requests_total{
            service="orders", code=~"5.."
          }[1h]))
          /
          sum(rate(http_requests_total{service="orders"}[1h]))

  - name: slo.orders.alerts
    interval: 30s
    rules:
      # Threshold scales with the SLO target.
      - alert: OrdersSLOFastBurn
        expr: |
          (
            slo:orders:errors:ratio_rate1h > (14.4 * (1 - 0.999))
            and
            slo:orders:errors:ratio_rate6h > (6 * (1 - 0.999))
          )
        for: 2m
        labels:
          severity: page
          slo: orders-availability
        annotations:
          summary: 'Orders SLO: fast burn (page)'
          runbook_url: 'https://runbooks/slo/orders'

How to validate it

The validation of an iteration is the validation that the new SLO specification produces the expected recording rule, alert rule, dashboard, and runbook. Each artefact is re-validated with its own tool.

# Step 1: SLO spec file structure.
grep -E '^### v[0-9]+' docs/runbooks/slo/orders/spec.md
# Should list all iteration versions since v1.

# Step 2: recording rule syntax.
promtool check rules /etc/prometheus/rules/slo-orders.yml
# expected: SUCCESS

# Step 3: alert fires when expected.
amtool alert query 'severity=page,slo=orders-availability'
# Should match the current threshold.

# Step 4: dashboard JSON.
jq '.panels[].targets[] | .expr' /etc/grafana/dashboards/orders-slo.json
# Each panel reads the recording rule, not the raw counter.

# Step 5: runbook resolves.
curl -sI 'https://runbooks/slo/orders' | head -1
# HTTP/1.1 200 OK (or 301/302 if the runbook redirects).

For the qualitative validation: the iteration meeting produced a written decision recorded in the spec file within 5 business days. The decision matched the data (budget consumption, indicator audit, customer feedback).

How it can fail

1. Quarterly review meeting does not happen. The meeting is on the calendar; the meeting is rescheduled three times; the meeting is eventually cancelled. The SLO drifts. Symptom: the SLO has not been reviewed in over a year. Fix: book the meeting in advance; mandatory attendance; rotating chair.

2. Decision is written but the rule files are not updated. The team agreed to tighten the SLO; the rule files still use the old target. Symptom: dashboard and alerts continue to compare against the old target; the team’s belief is wrong. Fix: the rule update is part of the iteration — pairing the spec change with the rule change in the same review.

3. Customer feedback is not wired. The team reviews the budget but never asks the customer. Symptom: the SLO is tightened against the team’s belief in the service, not against what users experience. Fix: pull NPS, support tickets, and refund or churn rate into the review agenda.

4. Retirement is the missing outcome. A team holds an SLO on a service that no longer exists, or that has been replaced by a different product, or whose SLO is not meaningful for the current product. Symptom: the SLO is on the dashboard but no longer corresponds to a user-visible behaviour. Fix: the review can produce “retire” as an outcome; the team is allowed to drop SLOs that are no longer useful.

5. Over-iteration. The team tightens the SLO every quarter by a few basis points without a customer justification. Symptom: the SLO target becomes unattainable; the team is exhausted; the SLO is ignored. Fix: tie every iteration to a measured signal (NPS drop, refund spike, support cluster) — never to the team’s wish to be more reliable.

6. Iteration changes break the on-call rotation. A target is tightened without briefing the rotation; the rotation is suddenly paged more often. Symptom: the rotation mutes the alerts. Fix: the rotation representative in the review meeting signs off on the new target before deploy.

7. History is not preserved. The team iterates but does not record the prior versions. Symptom: a year later, the team cannot explain why the target is what it is; the original customer signal is lost. Fix: the version history in the spec file is mandatory.

How to troubleshoot it

When the SLO appears to be mis-calibrated:

1. Check the last review date. git log --since='4 months ago' --oneline docs/runbooks/slo/<service>/spec.md. If the last commit is more than 4 months ago, the SLO has drifted.
2. Check the indicator audit. When was the SLI last traced end-to-end? If the answer is “never”, the indicator may be aspirational.
3. Check the customer feedback. What does NPS say? What do the support tickets cluster on? If the SLO is silently failing the customer, the customer feedback will say so.
4. Check the runbook. Does the runbook match the current target? If the runbook describes a 99.95% response and the target is 99.9%, the runbook is out of date.

When the iteration cycle is not being observed:

1. Re-book the meeting. Cancel non-essential meetings to make room for the quarterly review.
2. Delegate the meeting chair. A chair who rotates keeps the meeting alive when the original chair moves on.
3. Make the iteration visible. The SLO specification file is referenced in the on-call runbook; the next iteration date is in the team’s shared calendar.

Security implications

The SLO specification file is internal documentation; no secrets are required. The recording rule consumes existing metrics. The dashboard panels read the recording rule; the alert annotations read SLI data. None of these introduce a new attack surface.

The customer feedback signals (NPS, support tickets, refund rate) are themselves sensitive — they may include PII. The team’s review meeting must aggregate this data before discussing it in a shared document. Aggregate counts only; never raw feedback in the spec file.

Performance implications

SLO iteration does not add to Prometheus load; the recording and alert rules are revised, not added. The validation step adds a promtool check and an amtool query per quarter — sub-second on any production stack. The meeting itself is the dominant cost: one hour per quarter per service.

Production guidance

• Book the quarterly review in advance for the entire year. A meeting on the calendar at the start of Q1 is a meeting that happens in Q1, Q2, Q3, Q4.
• Tie every iteration to a measured signal: budget consumption, indicator audit, customer feedback. Never iterate on the team’s wish to be more reliable.
• Record the iteration history in the spec file. The history is the audit trail that explains why the SLO is what it is.
• Pair the spec change with the rule change in the same review meeting. A change to the target without a change to the rule is an inconsistency.
• Brief the on-call rotation before the new target is deployed. The rotation is the consumer; the consumer must agree.

Verification

You should now be able to answer:

• What are the four phases of the SLO lifecycle (design, instrument, observe, iterate) and which artefact does each produce?
• What three signals suggest the SLO should be tightened, loosened, or retired?
• How is the customer-feedback loop wired into the quarterly review, and why is it required for the SLO to remain operational?
• What is the role of the iteration history in the SLO spec file, and what does the absence of history indicate?

Quiz