Lab: SLO-Based Alerting

Objective

By the end of this lab you will have the complete alerting package for one SLO, and evidence that the part everybody skips is the part that matters.

The evidence is two fixtures. The first replays seven hours of traffic containing one 30-minute error spike, and shows the same input firing a single-window burn-rate rule while leaving the multi-window pair silent. The second replays three days of a slow 0.4 percent burn, and shows a ticket opening while the page never fires. Both run offline in seconds, because the alternative — waiting three days for the 72-hour window to fill — is not an experiment anyone runs twice.

Then you bring the same rule file up on a live Prometheus and watch the fast-burn pair go inactive to pending to firing, and find out what the burn-rate windows report on a server that has been running for four minutes.

Architecture

One host, three containers, and a shell loop standing in for a service that serves a thousand requests every five seconds:

  your shell
      |
      |  push http_requests_total{code="200"|"500"} every 5s
      v
  +---------------+       scrape        +--------------+
  | Pushgateway   | <------------------ |  Prometheus  |
  | :9091         |   honor_labels      |  :9090       |
  +---------------+                     +--------------+
                                              |
                                              |  5 recording rules
                                              |  4 alert rules
                                              v
                                        +--------------+
                                        | Alertmanager |
                                        | :9093        |
                                        +--------------+
                                          |     |     |
                                       page  ticket  policy
                                        (three receivers,
                                         no integrations)

Pushgateway lets you choose the error ratio by typing a number. It is the wrong tool for a real service — it holds the last value you pushed forever and has no notion of a target being down — and exactly the right tool here, because the rules under test cannot tell a pushed series from a scraped one.

Alertmanager carries three receivers with no integrations, matching the three severities the error budget policy uses. Nothing is delivered anywhere; what you are proving is that each alert leaves Prometheus with the labels its route matches on.

Requirements

Linux or macOS with a shell, Docker Engine 28.x and Docker Compose v2.
TCP ports 9090, 9091 and 9093 free, and roughly 1 GB of disk for three images.
curl and jq. promtool runs out of the Prometheus image, so there is nothing to install.
90 minutes. About 20 of those are spent watching timers you must not shorten, and about 40 are spent on fixtures that finish in seconds.
Nothing here touches host networking, firewall rules or SSH. The worst outcome is three stopped containers and a directory to delete.

Scenario

The orders team adopted a 99.9 percent availability SLO and wired up the alert everyone writes first: burn rate over a one-hour window, page above 14.4x. For three months it fired most Friday afternoons, when the traffic dip made a handful of errors look like a large ratio. The rotation learned the alert, then muted it. In month four a memory leak burned about 0.4 percent of requests for three days straight — never sharp enough to trip the one-hour window, easily enough to consume the whole monthly budget — and nobody was told until the budget dashboard went red.

One rule was noisy and blind at the same time. This lab builds the rule set that is neither, and proves both properties before the rules go anywhere near a rotation.

Tasks

Task 1: Record the starting state and lay out the directory

If any of these print a line, stop and free the port or pick another host. The lab publishes on all three.

Read-only / Safehost— Nothing should be listening on 9090, 9091 or 9093

$ ss -ltnp 2>/dev/null | grep -E ':(9090|9091|9093)\b' || echo 'ports free'

On macOS, lsof -nP -iTCP:9090 -sTCP:LISTEN is the equivalent.

Create the tree, and define the promtool helper the rest of the lab uses. Re-declare it if you open a new terminal:

WORKDIR="$HOME/obs-slo-lab"
mkdir -p "$WORKDIR"/rules/test
cd "$WORKDIR"

promtool_run() {
  docker run --rm -v "$PWD:/work" -w /work \
    --entrypoint /bin/promtool prom/prometheus:v2.55.1 "$@"
}

The container runs as uid 65534, so it reads only world-readable files. A default umask produces 0644 and is fine; a restrictive umask needs chmod a+r on the directory contents first.

Task 2: Write the SLO down as numbers before writing any YAML

Every threshold in this lab is derived from two values. Write them at the top of your notes and keep them there:

SLO target: 99.9 percent of orders requests return a non-5xx status, measured over a rolling 30 days.
Error budget: 1 - 0.999 = 0.001. One request in a thousand may fail. That fraction is the denominator of every burn rate that follows.

Burn rate is the observed error ratio divided by the budget, so a threshold of “burn at 14.4x” is written in a rule as an error ratio of 14.4 * 0.001. The four canonical windows and what each one buys:

Window	Burn rate	Rule threshold	Budget gone in	Severity
1h	14.4x	`14.4 * 0.001`	2.08 days	page (with 6h)
6h	6x	`6 * 0.001`	5 days	page (with 1h)
24h	3x	`3 * 0.001`	10 days	ticket (with 72h)
72h	1x	`1 * 0.001`	30 days	ticket (with 24h)

The 0.001 in every one of those is the orders budget. Copying these lines to a service with a 99.5 percent SLO without changing 0.001 to 0.005 produces a rule that pages five times sooner than intended, and it will look completely normal in review. Write the multiplication out rather than pre-computing 0.0144, so that the target is visible in the file.

Task 3: Write the SLO package

One file, two groups: the recording rules that materialise the SLI over each window, and the alerts that compare them.

# rules/slo-orders.rules.yml
groups:
  - name: slo.orders.recording
    interval: 30s
    rules:
      - record: slo:orders:errors:ratio_rate1h
        expr: |
          sum(rate(http_requests_total{service="orders", code=~"5.."}[1h]))
          /
          sum(rate(http_requests_total{service="orders"}[1h]))

      - record: slo:orders:errors:ratio_rate6h
        expr: |
          sum(rate(http_requests_total{service="orders", code=~"5.."}[6h]))
          /
          sum(rate(http_requests_total{service="orders"}[6h]))

      - record: slo:orders:errors:ratio_rate24h
        expr: |
          sum(rate(http_requests_total{service="orders", code=~"5.."}[24h]))
          /
          sum(rate(http_requests_total{service="orders"}[24h]))

      - record: slo:orders:errors:ratio_rate72h
        expr: |
          sum(rate(http_requests_total{service="orders", code=~"5.."}[72h]))
          /
          sum(rate(http_requests_total{service="orders"}[72h]))

      - record: slo:orders:errors:budget_30d_consumed_pct
        expr: |
          (
            sum(increase(http_requests_total{service="orders", code=~"5.."}[30d]))
            /
            (0.001 * sum(increase(http_requests_total{service="orders"}[30d])))
          ) * 100

  - name: slo.orders.alerts
    interval: 30s
    rules:
      - alert: OrdersSLOFastBurnPage
        expr: |
          (
            slo:orders:errors:ratio_rate1h > (14.4 * 0.001)
            and
            slo:orders:errors:ratio_rate6h > (6 * 0.001)
          )
        for: 2m
        labels:
          severity: page
          slo: orders-availability
        annotations:
          summary: 'orders SLO fast burn: 1h and 6h windows both over threshold'
          description: 'The 1h burn rate is {{ $value | humanize }} as an error ratio. At this rate the 30-day budget is gone in two to five days.'
          runbook_url: 'https://runbooks.example.com/slo/orders-fast-burn'

      - alert: OrdersSLOSlowBurnTicket
        expr: |
          (
            slo:orders:errors:ratio_rate24h > (3 * 0.001)
            and
            slo:orders:errors:ratio_rate72h > (1 * 0.001)
          )
        for: 1h
        labels:
          severity: ticket
          slo: orders-availability
          ticket_queue: slo-quarterly-review
        annotations:
          summary: 'orders SLO slow burn: 24h and 72h windows both over threshold'
          runbook_url: 'https://runbooks.example.com/slo/orders-slow-burn'

      - alert: OrdersBudgetPolicy25
        expr: slo:orders:errors:budget_30d_consumed_pct > 25
        for: 1h
        labels:
          severity: policy-slowdown
          slo: orders-availability
        annotations:
          summary: 'orders 30-day error budget more than 25 percent consumed'
          runbook_url: 'https://runbooks.example.com/slo/orders-policy'

      - alert: OrdersBudgetPolicy100
        expr: slo:orders:errors:budget_30d_consumed_pct > 100
        for: 30m
        labels:
          severity: policy-freeze
          slo: orders-availability
        annotations:
          summary: 'orders 30-day error budget exhausted: release freeze applies'
          runbook_url: 'https://runbooks.example.com/slo/orders-policy'

Now the rule the scenario started with, in its own file so that you can load it into the fixture and into the live server and then throw it away:

# rules/single-window.rules.yml
groups:
  - name: slo.orders.singlewindow
    interval: 30s
    rules:
      - alert: OrdersSLOFastBurn1hOnly
        expr: slo:orders:errors:ratio_rate1h > (14.4 * 0.001)
        for: 2m
        labels:
          severity: page
          slo: orders-availability
        annotations:
          summary: 'orders SLO fast burn on the 1h window alone'
          runbook_url: 'https://runbooks.example.com/slo/orders-fast-burn'

Check that both files load:

Read-only / Safehost— Static check only: does Prometheus accept these files?

$ promtool_run check rules rules/slo-orders.rules.yml rules/single-window.rules.yml

Checking rules/slo-orders.rules.yml
SUCCESS: 9 rules found

Checking rules/single-window.rules.yml
SUCCESS: 1 rules found

Illustrative output

Both files pass. So would a version with 0.005 in place of 0.001, and so would the single-window rule that a rotation muted for three months. A static check answers “will this load”, which is the right first question and a bad last one.

Under the hood

The and between the two legs is a vector matching operator, not a boolean. It returns the samples on the left that have a matching label set on the right. Both sum(...) expressions aggregate away every label, so each recording rule produces a single labelless series and the match is trivially satisfied — which is exactly why aggregating both legs to the same label set matters. If one leg carried service and the other did not, the and would return nothing and the alert could never fire, with no error anywhere.

There is a second consequence: if the right-hand recording rule fails to evaluate or has gone stale, the conjunction is silently false. A missing recording rule turns a page into silence, which is why meta-monitoring on rule health is a separate concern from the SLO itself.

Task 4: The fixture that separates the two rules

This is the centre of the lab. The fixture describes seven hours of traffic: a service serving 1,000 successful requests every five minutes, with one 30-minute burst of errors near the end.

# rules/test/spike_test.yml
rule_files:
  - ../slo-orders.rules.yml
  - ../single-window.rules.yml

evaluation_interval: 5m

tests:
  - interval: 5m
    name: a short spike must page the single-window rule and not the AND
    input_series:
      - series: 'http_requests_total{service="orders", code="200"}'
        values: '0+1000x110'
      - series: 'http_requests_total{service="orders", code="500"}'
        values: '0+0x78 0+40x6 240+0x24'
    alert_rule_test:
      - eval_time: 7h20m
        alertname: OrdersSLOFastBurnPage
        exp_alerts: []
      - eval_time: 7h20m
        alertname: OrdersSLOFastBurn1hOnly
        exp_alerts:
          - exp_labels:
              alertname: OrdersSLOFastBurn1hOnly
              severity: page
              slo: orders-availability
            exp_annotations:
              summary: 'orders SLO fast burn on the 1h window alone'
              runbook_url: 'https://runbooks.example.com/slo/orders-fast-burn'

Read the error series before you run it, because the three segments are the whole scenario and the syntax hides them. 0+0x78 is 79 samples of zero, covering the first six and a half hours. 0+40x6 adds 40 errors per five-minute step for half an hour, so 240 errors accumulate. 240+0x24 holds the counter flat afterwards — a counter may never go down, so the third segment has to start at the value the second one reached.

Run it:

Read-only / Safehost— Offline. Seven hours of synthetic traffic, no network, no server.

$ promtool_run test rules rules/test/spike_test.yml

Unit Testing:  rules/test/spike_test.yml
SUCCESS

Illustrative output

Both assertions passed against the same input, and they disagree. Do the arithmetic, because the numbers are the lesson:

The 1h leg at 7h20m looks at a window holding roughly 11,000 successes and all 240 errors. That is an error ratio near 0.021, which is about 21x the budget — comfortably over the 14.4x threshold. The single-window rule pages.
The 6h leg at the same instant looks at a window holding roughly 71,000 successes and the same 240 errors. That is a ratio near 0.0034, about 3.4x the budget, under the 6x threshold. The conjunction is false, so the multi-window rule produces nothing at all — not a pending alert, nothing.

The spike was real. It was also 240 errors out of about 87,000 requests, which is 0.28 percent of the traffic and a rounding error against a monthly budget. The AND is what encodes “real but not worth waking someone for”.

Task 5: The fixture for the burn nobody notices

The second failure mode from the scenario: a leak burning 0.4 percent of requests, sustained. Below the fast thresholds, above the slow ones.

# rules/test/slow-burn_test.yml
rule_files:
  - ../slo-orders.rules.yml
  - ../single-window.rules.yml

evaluation_interval: 5m

tests:
  - interval: 5m
    name: a sustained 0.4 percent burn tickets and never pages
    input_series:
      - series: 'http_requests_total{service="orders", code="200"}'
        values: '0+996x936'
      - series: 'http_requests_total{service="orders", code="500"}'
        values: '0+4x936'
    alert_rule_test:
      - eval_time: 75h
        alertname: OrdersSLOFastBurnPage
        exp_alerts: []
      - eval_time: 75h
        alertname: OrdersSLOFastBurn1hOnly
        exp_alerts: []
      - eval_time: 75h
        alertname: OrdersSLOSlowBurnTicket
        exp_alerts:
          - exp_labels:
              alertname: OrdersSLOSlowBurnTicket
              severity: ticket
              slo: orders-availability
              ticket_queue: slo-quarterly-review
            exp_annotations:
              summary: 'orders SLO slow burn: 24h and 72h windows both over threshold'
              runbook_url: 'https://runbooks.example.com/slo/orders-slow-burn'

996 successes and 4 errors per step is exactly 0.4 percent, held for 78 hours so that the 72h window is genuinely full at the 75h evaluation. The four thresholds, against a constant ratio of 0.004:

Leg	Threshold	Ratio	Over?
1h	0.0144	0.004	no
6h	0.006	0.004	no
24h	0.003	0.004	yes
72h	0.001	0.004	yes

Run it. This fixture drives 900 evaluations and reads a 72-hour range vector on each one, so it takes noticeably longer than the first — still seconds, not the three days the same experiment costs in production.

promtool_run test rules rules/test/slow-burn_test.yml

Neither page rule fires. Both page rules are correct not to: at 4x the budget the service has ten days before it runs out, which is a ticket during working hours, not a phone call. The ticket rule’s for: 1h says the same thing in a different way — an hour of dwell on a condition you have days to act on costs nothing.

Task 6: Bring up the stack

Three files. Write them all before starting anything.

# docker-compose.yml
services:
  prometheus:
    image: prom/prometheus:v2.55.1
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.path=/prometheus
      - --web.enable-lifecycle
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./rules:/etc/prometheus/rules:ro
      - prom-data:/prometheus
    ports:
      - "9090:9090"

  pushgateway:
    image: prom/pushgateway:v1.10.0
    ports:
      - "9091:9091"

  alertmanager:
    image: prom/alertmanager:v0.28.1
    command:
      - --config.file=/etc/alertmanager/alertmanager.yml
      - --storage.path=/alertmanager
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
      - am-data:/alertmanager
    ports:
      - "9093:9093"

volumes:
  prom-data:
  am-data:

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - /etc/prometheus/rules/*.yml

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']

  - job_name: pushgateway
    honor_labels: true
    static_configs:
      - targets: ['pushgateway:9091']

honor_labels: true keeps the job and instance labels the push supplied instead of overwriting them with the gateway’s own. The rule_files glob is one level deep and matches .yml, so the fixtures in rules/test/ are not loaded as rules.

# alertmanager.yml
route:
  receiver: 'default'
  group_by: ['alertname', 'slo']
  group_wait: 10s
  group_interval: 1m
  repeat_interval: 4h
  routes:
    - matchers: [ 'severity = "page"' ]
      receiver: 'orders-oncall-page'
    - matchers: [ 'severity = "ticket"' ]
      receiver: 'slo-ticket-queue'
    - matchers: [ 'severity =~ "policy-.*"' ]
      receiver: 'orders-eng-manager'

receivers:
  - name: 'default'
  - name: 'orders-oncall-page'
  - name: 'slo-ticket-queue'
  - name: 'orders-eng-manager'

Four receivers with names and no integrations. They are valid, they deliver nothing, and they let you prove routing without configuring a pager. Start the stack:

Configuration changehost— Pulls three images on first run, then starts them

$ docker compose up -d && docker compose ps

Confirm the server loaded every rule, rather than assuming it:

Read-only / Safehost— Rule health as the running server reports it

$ curl -s http://localhost:9090/api/v1/rules | jq -r '.data.groups[].rules[] | "\(.name) health=\(.health) type=\(.type)"'

slo:orders:errors:ratio_rate1h health=ok type=recording
slo:orders:errors:ratio_rate6h health=ok type=recording
slo:orders:errors:ratio_rate24h health=ok type=recording
slo:orders:errors:ratio_rate72h health=ok type=recording
slo:orders:errors:budget_30d_consumed_pct health=ok type=recording
OrdersSLOFastBurnPage health=ok type=alerting
OrdersSLOSlowBurnTicket health=ok type=alerting
OrdersBudgetPolicy25 health=ok type=alerting
OrdersBudgetPolicy100 health=ok type=alerting
OrdersSLOFastBurn1hOnly health=ok type=alerting

Illustrative output

Ten rules, all healthy, and no series behind any of them yet. Nothing has pushed a metric, so every recording rule currently evaluates to nothing — which is why no alert is pending.

Task 7: Drive a healthy service

Save this as slo-drive.sh and make it executable. It pushes a counter pair every tick, always both series: a POST to Pushgateway replaces every series sharing a metric name, so a tick that omitted code="200" would delete it rather than leave it alone.

#!/usr/bin/env bash
# slo-drive.sh - push a synthetic orders SLI to Pushgateway.
# Usage: ./slo-drive.sh ERROR_PERCENT TICKS [SECONDS_BETWEEN_TICKS]
set -euo pipefail

PGW="${PGW:-http://localhost:9091}"
ERROR_PERCENT="${1:-0}"
TICKS="${2:-24}"
SLEEP_SECONDS="${3:-5}"
REQUESTS_PER_TICK=1000

ok_total=0
err_total=0

for ((i = 1; i <= TICKS; i++)); do
  err=$(( REQUESTS_PER_TICK * ERROR_PERCENT / 100 ))
  ok_total=$(( ok_total + REQUESTS_PER_TICK - err ))
  err_total=$(( err_total + err ))

  printf '%s\n' \
    '# TYPE http_requests_total counter' \
    "http_requests_total{service=\"orders\",code=\"200\"} ${ok_total}" \
    "http_requests_total{service=\"orders\",code=\"500\"} ${err_total}" \
    | curl -sf --data-binary @- \
        "${PGW}/metrics/job/orders/instance/orders-1"

  printf 'tick %2d  ok=%-8d err=%-7d error_percent=%s\n' \
    "$i" "$ok_total" "$err_total" "$ERROR_PERCENT"
  sleep "${SLEEP_SECONDS}"
done

Run three minutes of a healthy service:

chmod +x slo-drive.sh
./slo-drive.sh 0 36 5

In a second terminal, read all four windows at once:

Read-only / Safehost (second terminal)— All four window recording rules, by name

$ curl -sG http://localhost:9090/api/v1/query --data-urlencode 'query={__name__=~"slo:orders:errors:ratio_rate.*"}' | jq -r '.data.result[] | "\(.metric.__name__) \(.value[1])"'

slo:orders:errors:ratio_rate1h 0
slo:orders:errors:ratio_rate6h 0
slo:orders:errors:ratio_rate24h 0
slo:orders:errors:ratio_rate72h 0

Illustrative output

Four series, all zero, and every alert stays inactive. Divide the same selector by 0.001 and you get the burn rate as a multiple of the budget — and lose the names, because arithmetic on a vector drops __name__. That is worth meeting here rather than in a dashboard panel whose legend has silently become four identical entries.

Task 8: Drive the incident and time the transitions

Clear the pushed group first, so the next phase starts a fresh counter instead of appearing to reset one:

Destructivehost— Deletes only the synthetic orders group from Pushgateway

$ curl -sf -X DELETE http://localhost:9091/metrics/job/orders/instance/orders-1 && echo deleted

Wait 30 seconds so Prometheus scrapes the disappearance and marks the series stale. Restarting the push immediately would put a counter decrease inside the rate windows.

Now four percent errors — 40 of every 1,000 requests — and note the wall-clock time you start:

date -u +%H:%M:%S
./slo-drive.sh 4 90 5

In the second terminal, poll every 15 seconds until the state stops changing, then stop with Ctrl-C:

while true; do
  printf '%s  ' "$(date -u +%H:%M:%S)"
  curl -s http://localhost:9090/api/v1/alerts \
    | jq -r '.data.alerts[] | "\(.labels.alertname)=\(.state)"' \
    | sort | tr '\n' ' '
  echo
  sleep 15
done

Four percent is forty times the budget, and no leg jumps straight there. Both windows still contain the healthy phase from Task 7 — and because deleting the Pushgateway group restarted the counter, rate() detected the reset and carried the pre-reset total forward, so the denominator holds every request you drove earlier. Each ratio therefore climbs as the incident’s errors accumulate against that history. The 6h leg crosses first, because its threshold is the lower of the two; the 1h leg follows about a minute behind it; only when both are true does the dwell start.

Alert	Dwell	State a few minutes in
`OrdersSLOFastBurn1hOnly`	2m	firing
`OrdersSLOFastBurnPage`	2m	firing
`OrdersSLOSlowBurnTicket`	1h	pending
`OrdersBudgetPolicy25`	1h	pending
`OrdersBudgetPolicy100`	30m	pending

Record the gap between starting the driver and the first firing line. Four things make it up: up to 15 seconds waiting for a scrape, up to 15 seconds waiting for a rule evaluation, the time each leg needs for the incident to outweigh the clean traffic already inside its window, and exactly two minutes of for: dwell. Only the last of the four is written down in the rule, which is why a team that has never watched this accumulate cannot say why an incident starting at 02:58 paged at 03:02.

Then confirm the alert arrived with the labels its route matches on, and in the right receiver:

Read-only / Safehost— What Alertmanager holds, and which receiver it selected

$ curl -s http://localhost:9093/api/v2/alerts | jq -r '.[] | "\(.labels.alertname) severity=\(.labels.severity) slo=\(.labels.slo) receiver=\(.receivers[0].name)"'

OrdersSLOFastBurn1hOnly severity=page slo=orders-availability receiver=orders-oncall-page
OrdersSLOFastBurnPage severity=page slo=orders-availability receiver=orders-oncall-page

Illustrative output

Two page-severity alerts, identical labels, same receiver. On this server, with this history, the naive rule and the tested one are indistinguishable — which is exactly why the fixtures had to exist. A live stack four minutes old can prove that a rule fires. It cannot prove that a rule declines to fire on the input that should not fire it.

Finally, look at the budget metric:

Read-only / Safehost— The error budget policy metric, on a server with no 30-day window

$ curl -sG http://localhost:9090/api/v1/query --data-urlencode 'query=slo:orders:errors:budget_30d_consumed_pct' | jq -r '.data.result[].value[1]'

Illustrative output

Four thousand percent of the monthly budget consumed, by a service that has been failing for four minutes. The rule is arithmetically right and operationally meaningless: increase(...[30d]) measured over minutes reports the burn rate multiplied by 100, not a budget fraction. It only becomes the number the error budget policy is written against once the window is genuinely full — which is why the policy thresholds carry for: values of an hour and belong on a dashboard a human reads, not on a pager.

Validation

Each of these is checkable, and each fails loudly if a step was skipped.

promtool_run check rules rules/slo-orders.rules.yml rules/single-window.rules.yml exits 0 for both files.
promtool_run test rules rules/test/spike_test.yml prints SUCCESS.
Deleting the leading 0+0x78 segment from the spike fixture makes the multi-window assertion fail. Put it back and it passes again — a fixture you have never seen fail is a fixture you have not tested.
Changing 6 * 0.001 to 3 * 0.001 in the fast-burn rule also makes the spike fixture fail. Change it back.
promtool_run test rules rules/test/slow-burn_test.yml prints SUCCESS, with the ticket firing and neither page rule firing.
/api/v1/rules reports ten rules, all health=ok with an empty lastError.
Your timeline shows a gap of a few minutes between the driver starting and OrdersSLOFastBurnPage reaching firing, and you can attribute each component of it — scrape, evaluation, ratio climb, dwell — to something you configured.
/api/v2/alerts returned both page alerts with severity, slo and the orders-oncall-page receiver.
You can state the value the 6h leg held during the spike fixture, and why it was under threshold while the 1h leg was over it.

Expected Outcome

obs-slo-lab/
├── alertmanager.yml
├── docker-compose.yml
├── prometheus.yml
├── slo-drive.sh
└── rules/
    ├── single-window.rules.yml
    ├── slo-orders.rules.yml
    └── test/
        ├── slow-burn_test.yml
        └── spike_test.yml

One rule file carrying the full package for a single SLO: four window recording rules, a budget metric, a page pair, a ticket pair and two policy thresholds, every threshold written as a multiplication so the SLO target stays visible. Two fixtures that pin the behaviour in both directions, and a saved diff showing the rule that would have paged.

Troubleshooting

promtool_run says permission denied. The container is uid 65534 and your files are not world-readable. chmod a+r rules rules/*.yml rules/test/*.yml.

The slow-burn fixture takes minutes or seems to hang. It evaluates 900 ticks against a 72-hour range vector. Reduce evaluation_interval work by keeping it at 5m — a fixture at 30s does twelve times the evaluations for no extra fidelity.

A fixture fails with “no rules found” or an empty alert list for a rule you can see in the file. The rule_files: paths are relative to the fixture, not to the working directory. From rules/test/, the package is ../slo-orders.rules.yml.

An alert assertion fails on labels you did not write. promtool prints the alert it got with every label. Copy the label set from the diff rather than reconstructing it — alertname in particular is easy to forget in exp_labels.

A recording rule shows health=err on the live server. The file loaded and the expression failed at evaluation. The lastError field in /api/v1/rules names the problem; a division whose legs carry different label sets is the usual cause.

No series at all while the driver is clearly pushing. Query http_requests_total on its own. If it comes back with job="pushgateway", honor_labels: true is missing from the scrape config and the pushed identity was overwritten at scrape time.

The burn rate spikes absurdly when you restart the driver. You did not delete the Pushgateway group first, so the counter went backwards and rate() treated it as a reset. Delete the group, wait 30 seconds, start again.

Alertmanager holds nothing while Prometheus says firing. Ask Prometheus what Alertmanagers it knows about: curl -s http://localhost:9090/api/v1/alertmanagers | jq. An empty activeAlertmanagers list means the two containers are not on the same Compose network or the target name is wrong.

Cleanup

The lab created three containers, two named volumes, one Compose network and one directory. All of it comes back.

Step 1. Stop the containers and remove the volumes:

Destructivehost— Removes the containers, the network, and both named volumes

$ cd "$HOME/obs-slo-lab" && docker compose down -v

Step 2. Confirm nothing of the lab’s is still running and the ports you recorded in Task 1 are free again:

docker compose ps
ss -ltnp 2>/dev/null | grep -E ':(9090|9091|9093)\b' || echo 'ports free'

rm -rf takes the whole directory, notes included

The block below deletes $HOME/obs-slo-lab recursively, without confirmation. The copy in front of it preserves the rules and the fixtures; it knows nothing about a file you added. List the directory before you delete it:

ls -la "$HOME/obs-slo-lab"

The path is written in full and anchored to $HOME deliberately. A relative rm -rf run from the wrong directory is how people delete work they meant to keep.

Data-loss riskhost— Copies the deliverables out, then deletes the lab directory

$ mkdir -p "$HOME/obs-lab-deliverables" && cp -a "$HOME/obs-slo-lab/rules" "$HOME/obs-lab-deliverables/slo-lab-rules" && rm -rf "$HOME/obs-slo-lab"

Step 3. The three images stay in the local cache. Leave them if you are going on to another Prometheus lab; otherwise:

docker image rm prom/prometheus:v2.55.1 prom/pushgateway:v1.10.0 prom/alertmanager:v0.28.1

Production notes

Mapping this exercise onto a real change window:

Delete single-window.rules.yml before this goes anywhere. It exists to make a fixture argue with itself. Shipping it alongside the tested rule gives the rotation two pages for one incident and teaches them to mute both.
The rules and their fixtures ship in one commit. A reviewer looking at a threshold change with no fixture change cannot tell a tuning from a semantics change. Both promtool check rules and promtool test rules belong in CI; the second is the one that has an opinion.
Parametrise on the SLO target, do not copy the file. Every 0.001 in this lab is the orders budget. The lessons cover Sloth, which generates this whole bundle from a short specification and removes the class of bug where one of six thresholds was left at the old target.
A new SLO’s alerts start as tickets. Route them somewhere that wakes nobody for the first weeks, watch what they do against real traffic, and promote to page once the firing pattern matches intent. The for: values in particular can only be set from production data.
The slow pair is blind after a restart until the windows refill. A Prometheus rebuilt from an empty TSDB reports a 72h burn rate over whatever it has. If the slow pair matters to you, either retain long enough to cover it or alert on the recording rule’s own absence.
The policy alerts need an owner, not a pager. severity: policy-slowdown routes to whoever can actually slow feature work down; a threshold nobody has authority to act on is a dashboard panel with a notification attached.

What You Learned

Every threshold in the package is a multiplication of the error budget. Writing 14.4 * 0.001 rather than 0.0144 keeps the SLO target visible in review, where a wrong budget is otherwise invisible.
The AND of two windows encodes “real but not page-worthy”. The spike fixture fired one rule and not the other on identical input, and the difference was 240 errors diluted across six hours instead of one.
A sustained burn below every page threshold still empties the budget. The 0.4 percent fixture never paged and always ticketed, which is the correct handling for something you have ten days to fix.
A burn-rate window reports a number long before it holds that much data. The ratio of two rates over a partial window is the ratio over the data that exists, so a young server answers every window confidently and scopes none of them correctly.
A live stack proves firing; only a fixture proves not-firing. Both page rules were indistinguishable on the running server and were separated in milliseconds offline.
for: is what turns one threshold set into three severities. The same four percent of errors produced a page in two minutes, a pending ticket for an hour, and a policy alert that a human reads on a dashboard.

Objectives

Prerequisites

Objective

Architecture

Requirements

Scenario

Tasks

Task 1: Record the starting state and lay out the directory

Task 2: Write the SLO down as numbers before writing any YAML

Task 3: Write the SLO package

Task 4: The fixture that separates the two rules

Task 5: The fixture for the burn nobody notices

Task 6: Bring up the stack

Task 7: Drive a healthy service

Task 8: Drive the incident and time the transitions

Validation

Expected Outcome

Troubleshooting

Cleanup

Production notes

What You Learned

Deliverables

Verification status