Lab: Build an Alert Rule

Objective

By the end of this lab you will have one alerting rule that fires for the right reason, and evidence for every part of that claim: a fixture that fails on the wrong version of the rule and passes on the right one, a live Prometheus that walks the rule from inactive to pending to firing, and an Alertmanager that received it with the labels a route can match on.

The part that matters most is the middle. You will write a rule that promtool check rules approves and that is nevertheless wrong, and you will find out it is wrong from a fixture rather than from a page at 03:00.

Architecture

One host, three containers, and a shell loop that pretends to be an application:

  your shell
      |
      |  push a counter pair every 5s
      v
  +---------------+       scrape        +--------------+
  | Pushgateway   | <------------------ |  Prometheus  |
  | :9091         |   honor_labels      |  :9090       |
  +---------------+                     +--------------+
                                              |
                                              |  rule evaluation
                                              |  every 15s
                                              v
                                        +--------------+
                                        | Alertmanager |
                                        | :9093        |
                                        +--------------+
                                              |
                                              v
                                        receiver "sink"
                                        (deliberately no
                                         integrations)

Pushgateway stands in for the application. It is the wrong tool for a long-running service in production — it holds the last value you pushed forever, and it has no notion of a target being down — but it is exactly the right tool here, because it lets you decide the error ratio by typing a number, and the rule under test cannot tell the difference between a pushed series and a scraped one.

Alertmanager is present so you can prove the alert left Prometheus with its labels intact. It has a receiver with no integrations, so nothing is sent anywhere. Routing and delivery are the subject of the Alertmanager lab, not this one.

Requirements

• Linux or macOS with a shell, Docker Engine 28.x and Docker Compose v2.
• Roughly 1 GB of free disk for the three images, and TCP ports 9090, 9091 and 9093 free on the host.
• curl and jq. Everything else runs in a container.
• 90 minutes, of which about 15 are spent waiting for timers you must not shortcut — the waiting is the measurement.
• No out-of-band access requirement. This lab touches no host networking, no firewall and no SSH configuration; the worst outcome is three stopped containers and a directory to delete.

Scenario

The checkout team wants to be paged when orders-api starts returning server errors. Somebody writes the rule in ten minutes, runs promtool check rules, sees SUCCESS, and merges it. Two days later the rule has paged the rota four times on a service that was returning eleven errors in an hour out of a hundred thousand requests.

The rule was not broken in any way a parser can see. It compared an error rate against a threshold that was chosen for a ratio. Both are numbers, both parse, and only one of them means what the team meant. This lab reproduces that mistake deliberately, so that you meet it in a fixture instead of in a rota.

Tasks

Task 1: Record the starting state and lay out the directory

Confirm the ports are free. If any of these print a line, stop and pick a different host or free the port — the lab publishes on all three.

$ ss -ltnp 2>/dev/null | grep -E ':(9090|9091|9093)\b' || echo 'ports free'

On macOS, lsof -nP -iTCP:9090 -sTCP:LISTEN is the equivalent check.

Create the tree the rest of the lab writes into:

WORKDIR="$HOME/obs-alert-rule-lab"
mkdir -p "$WORKDIR"/rules/test
cd "$WORKDIR"

Everything in this lab that needs promtool runs it from the Prometheus image, so there is nothing to install. Define the helper once — later tasks assume it is in your shell, so re-declare it if you open a new terminal:

promtool_run() {
  docker run --rm -v "$PWD:/work" -w /work \
    --entrypoint /bin/promtool prom/prometheus:v2.55.1 "$@"
}

The container runs as uid 65534, so it can only read files that are world-readable. The default umask on a normal account produces 0644 files, which is fine; if you have a restrictive umask, chmod a+r the directory contents before the first promtool_run.

Task 2: Write the draft rule, the one that looks right

This is the rule as somebody writes it under time pressure. Save it and read it before you run anything — the mistake is visible if you know to look for it.

# rules/orders-api.rules.yml
groups:
  - name: orders-api.slo
    interval: 15s
    rules:
      - alert: OrdersApiHighErrorRate
        expr: |
          sum by (service) (
            rate(http_requests_total{service="orders-api", status=~"5.."}[1m])
          )
          > 0.05
        for: 2m
        keep_firing_for: 5m
        labels:
          severity: critical
          team: checkout
          service: orders-api
        annotations:
          summary: '{{ $labels.service }} 5xx ratio above 5 percent for 2 minutes'
          description: 'FILL THIS IN FROM THE PROMTOOL DIFF'
          runbook_url: 'https://runbooks.example.com/checkout/orders-api-5xx'

Now run the check that the merge request would have run:

$ promtool_run check rules rules/orders-api.rules.yml

Checking rules/orders-api.rules.yml
SUCCESS: found 1 rules, 1 alerts

SUCCESS. The YAML is well formed, the expression parses, the labels and annotations are present, the for: is a valid duration. Every static property of this rule is correct.

The expression says: the per-second rate of 5xx responses, summed across statuses, is above 0.05. That is five hundredths of a request per second — about three errors a minute. The threshold was chosen for a ratio, where 0.05 means five percent. Nothing in the file records which of those two meanings was intended, and promtool check rules has no opinion, because both are arithmetically valid PromQL.

Task 3: Write the fixture that disagrees

The fixture declares a healthy service: 100 successful requests and 1 server error per 15-second interval, sustained for ten minutes. That is an error ratio just under one percent — a service nobody should be paged for.

# rules/test/orders-api_test.yml
rule_files:
  - ../orders-api.rules.yml

evaluation_interval: 15s

tests:
  - interval: 15s
    name: a one percent error ratio must never fire
    input_series:
      - series: 'http_requests_total{service="orders-api", status="200"}'
        values: '0+100x40'
      - series: 'http_requests_total{service="orders-api", status="500"}'
        values: '0+1x40'
    alert_rule_test:
      - eval_time: 8m
        alertname: OrdersApiHighErrorRate
        exp_alerts: []

0+100x40 is the shorthand from the unit-test format: start at 0 and add 100 forty times, one sample per interval. The two series together describe 101 requests every 15 seconds, one of which fails.

Run it:

$ promtool_run test rules rules/test/orders-api_test.yml

Unit Testing:  rules/test/orders-api_test.yml
FAILED:
  alertname: OrdersApiHighErrorRate, time: 8m0s,
      exp:[],
      got:[
          0:
            Labels:{alertname="OrdersApiHighErrorRate", service="orders-api", team="checkout", ...}
            Annotations:{...}
      ]

The fixture asserted no alert and got one. Do the arithmetic before you change anything, because the number is the whole point: one error per 15-second interval is 1/15, or 0.0667 errors per second, and 0.0667 is greater than 0.05. The rule fires on a service with a 0.99 percent error ratio, and it would keep firing forever.

Save this output. It is a lab deliverable, and it is the cheapest possible demonstration that a green static check is not a tested rule.

Task 4: Fix the expression at the level the mistake was made

The fix is not a bigger threshold. A threshold tuned against an absolute rate has to be re-tuned every time traffic changes, and it says nothing about the service’s health at three in the morning when traffic is a tenth of daytime. The fix is to compare like with like: divide the error rate by the total rate, so the number on the right of the comparison really is a ratio.

Replace the expr: block with the divided form and leave everything else alone:

# rules/orders-api.rules.yml — expr only, the rest is unchanged
        expr: |
          sum by (service) (
            rate(http_requests_total{service="orders-api", status=~"5.."}[1m])
          )
          /
          sum by (service) (
            rate(http_requests_total{service="orders-api"}[1m])
          )
          > 0.05

The sum by (service) on both sides is what makes the division work. Without it, the numerator carries status="500" and the denominator carries every status, so the two sides have no matching label set and PromQL returns nothing — a rule that can never fire and never errors. Aggregating both sides to service alone leaves one series on each side with identical labels, and the division binds them.

Re-run the static check and then the fixture:

$ promtool_run check rules rules/orders-api.rules.yml && promtool_run test rules rules/test/orders-api_test.yml

Checking rules/orders-api.rules.yml
SUCCESS: found 1 rules, 1 alerts

Unit Testing:  rules/test/orders-api_test.yml
SUCCESS

One error in 101 requests is 0.99 percent, which is below five percent, so no alert. The rule now means what the threshold always claimed it meant.

Task 5: Add the case that proves the rule still works

A rule that never fires passes the no-fire test perfectly. Add the scenario that must fire: ten errors per interval against a hundred successes, an error ratio of about 9.1 percent.

Append to rules/test/orders-api_test.yml:

  - interval: 15s
    name: a nine percent error ratio is not firing at 1m and is firing at 4m
    input_series:
      - series: 'http_requests_total{service="orders-api", status="200"}'
        values: '0+100x40'
      - series: 'http_requests_total{service="orders-api", status="500"}'
        values: '0+10x40'
    alert_rule_test:
      - eval_time: 1m
        alertname: OrdersApiHighErrorRate
        exp_alerts: []
      - eval_time: 4m
        alertname: OrdersApiHighErrorRate
        exp_alerts:
          - exp_labels:
              alertname: OrdersApiHighErrorRate
              service: orders-api
              severity: critical
              team: checkout
            exp_annotations:
              summary: 'orders-api 5xx ratio above 5 percent for 2 minutes'
              description: 'FILL THIS IN FROM THE PROMTOOL DIFF'
              runbook_url: 'https://runbooks.example.com/checkout/orders-api-5xx'

The first assertion is the one people leave out. At one minute the ratio has been over threshold for well under the two-minute dwell, so the alert exists in pending and no firing alert is produced. Asserting emptiness there is what pins the for: value: change for: 2m to for: 30s and this assertion goes red.

Run it. It will fail, on purpose, because the description annotation still says FILL THIS IN FROM THE PROMTOOL DIFF in both the rule and the fixture:

promtool_run test rules rules/test/orders-api_test.yml

The failure prints the expected alert and the alert it actually got, with every label and every annotation of each. That output is the authoritative statement of what your rule produces — more reliable than reading the Go template and predicting the render, which is how annotation assertions usually end up wrong.

Now make the annotation say something useful. In the rule file:

          description: 'orders-api returned {{ $value | humanizePercentage }} 5xx over the last minute. Triage order is in the runbook.'

Run the fixture again, read the rendered description out of the diff, and paste that exact string into exp_annotations.description in the fixture. Then run it a third time and expect SUCCESS.

Task 6: Bring up the stack

Four files. Write them all before starting anything.

# docker-compose.yml
services:
  prometheus:
    image: prom/prometheus:v2.55.1
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.path=/prometheus
      - --web.enable-lifecycle
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./rules:/etc/prometheus/rules:ro
      - prom-data:/prometheus
    ports:
      - "9090:9090"

  pushgateway:
    image: prom/pushgateway:v1.10.0
    ports:
      - "9091:9091"

  alertmanager:
    image: prom/alertmanager:v0.28.1
    command:
      - --config.file=/etc/alertmanager/alertmanager.yml
      - --storage.path=/alertmanager
    volumes:
      - ./alertmanager.yml:/etc/alertmanager/alertmanager.yml:ro
      - am-data:/alertmanager
    ports:
      - "9093:9093"

volumes:
  prom-data:
  am-data:

# prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - /etc/prometheus/rules/*.yml

alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']

  - job_name: pushgateway
    honor_labels: true
    static_configs:
      - targets: ['pushgateway:9091']

honor_labels: true is load-bearing. Without it Prometheus overwrites the job and instance labels that the push supplied with the gateway’s own, and every pushed series ends up attributed to Pushgateway rather than to the thing that pushed.

# alertmanager.yml
route:
  receiver: 'sink'
  group_by: ['alertname']
  group_wait: 10s
  group_interval: 1m
  repeat_interval: 1h

receivers:
  - name: 'sink'

A receiver with a name and no integrations is valid and sends nothing. That is deliberate: this lab proves delivery to Alertmanager, and stops there.

Note that rules/test/ is inside the directory mounted at /etc/prometheus/rules, and the rule_files glob is *.yml — one level only, so the fixture is not loaded as a rule file. Start the stack:

$ docker compose up -d && docker compose ps

Confirm Prometheus loaded the rule, rather than assuming it:

$ curl -s http://localhost:9090/api/v1/rules | jq -r '.data.groups[].rules[] | "\(.name) health=\(.health) state=\(.state) lastError=\(.lastError)"'

OrdersApiHighErrorRate health=ok state=inactive lastError=

state=inactive is correct and expected: nothing has pushed a metric yet, so the expression returns no series. An empty response instead of a line means the file was not picked up by the glob — check the mount and the filename extension before anything else.

Task 7: Drive a healthy service and confirm silence

Save this as drive.sh and make it executable. It pushes a counter pair on every tick, always including both series. curl --data-binary sends a POST, and a POST to Pushgateway replaces every series sharing a metric name with what you pushed — so a tick that omitted the status="200" series would delete it rather than leave it alone:

#!/usr/bin/env bash
# drive.sh — push a synthetic http_requests_total pair to Pushgateway.
# Usage: ./drive.sh ERRORS_PER_TICK TICKS [SECONDS_BETWEEN_TICKS]
set -euo pipefail

PGW="${PGW:-http://localhost:9091}"
ERRORS_PER_TICK="${1:-0}"
TICKS="${2:-24}"
SLEEP_SECONDS="${3:-5}"

ok_total=0
err_total=0

for ((i = 1; i <= TICKS; i++)); do
  ok_total=$(( ok_total + 100 ))
  err_total=$(( err_total + ERRORS_PER_TICK ))

  printf '%s\n' \
    '# TYPE http_requests_total counter' \
    "http_requests_total{service=\"orders-api\",status=\"200\"} ${ok_total}" \
    "http_requests_total{service=\"orders-api\",status=\"500\"} ${err_total}" \
    | curl -sf --data-binary @- \
        "${PGW}/metrics/job/orders-api/instance/orders-api-1"

  printf 'tick %2d  ok=%-6d err=%-6d ratio=%d%%\n' \
    "$i" "$ok_total" "$err_total" \
    "$(( ERRORS_PER_TICK * 100 / (ERRORS_PER_TICK + 100) ))"
  sleep "${SLEEP_SECONDS}"
done

Run the healthy phase: one error per tick against a hundred successes, for two minutes.

chmod +x drive.sh
./drive.sh 1 24 5

While it runs, in a second terminal, watch what Prometheus computes:

$ curl -sG http://localhost:9090/api/v1/query --data-urlencode 'query=sum by (service) (rate(http_requests_total{service="orders-api", status=~"5.."}[1m])) / sum by (service) (rate(http_requests_total{service="orders-api"}[1m]))' | jq -r '.data.result[] | "\(.metric.service) \(.value[1])"'

orders-api 0.009900990099009901

Just under one percent. Confirm the alert is still inactive:

curl -s http://localhost:9090/api/v1/alerts | jq '.data.alerts'

An empty array is the correct answer. A healthy service produced no alert state at all — not a pending one, not a suppressed one, nothing.

Task 8: Drive an unhealthy service and time the transitions

First clear the pushed group, so the next phase starts a fresh counter rather than appearing to reset one:

$ curl -sf -X DELETE http://localhost:9091/metrics/job/orders-api/instance/orders-api-1 && echo deleted

Wait 30 seconds. Prometheus needs one scrape to observe the series disappear and mark it stale; restarting the push immediately would put a counter decrease inside the one-minute rate window and produce a meaningless spike.

Now the unhealthy phase — ten errors per tick, for five minutes — and note the wall-clock time when you start it:

date -u +%H:%M:%S
./drive.sh 10 60 5

In the second terminal, poll the alert state every 15 seconds. Leave it running until the transition happens, then stop it with Ctrl-C:

while true; do
  printf '%s  ' "$(date -u +%H:%M:%S)"
  curl -s http://localhost:9090/api/v1/alerts \
    | jq -r '.data.alerts[] | "\(.labels.alertname) \(.state) activeAt=\(.activeAt)"' \
    | tr '\n' ' '
  echo
  sleep 15
done

Record three timestamps: when you started the driver, when the first pending line appears, and when it becomes firing. Three delays separate them, and each one is a design decision somebody made:

The activeAt field is the moment the alert entered pending. Subtracting it from the timestamp of the first firing line should give you the for: value, and if it does not, the ratio dipped below threshold at some point and reset the timer.

Task 9: Confirm the alert left Prometheus intact

A rule that fires and does not arrive is a rule that did not work. Ask Alertmanager what it holds:

$ curl -s http://localhost:9093/api/v2/alerts | jq -r '.[] | "\(.labels.alertname) severity=\(.labels.severity) team=\(.labels.team) service=\(.labels.service)"'

OrdersApiHighErrorRate severity=critical team=checkout service=orders-api

Three labels, all present, all spelled the way a route would expect. This is the handover point between the two halves of the alerting pipeline: everything up to here is a Prometheus problem, everything after here is an Alertmanager problem, and knowing which side you are on is most of a diagnosis.

Now stop the driver with Ctrl-C and watch what happens. The pushed counters stop advancing, so within a minute rate() returns zero for both sides of the division, and the expression evaluates 0/0.

Poll the alert state again. It stays firing for five more minutes, then resolves — that is keep_firing_for: 5m doing its job.

Validation

Each of these is checkable, and each fails loudly if you skipped a step.

1. promtool_run check rules rules/orders-api.rules.yml exits 0 and reports one alert.
2. promtool_run test rules rules/test/orders-api_test.yml prints SUCCESS for both scenarios.
3. Reverting the expr: to the Task 2 draft makes scenario one fail, and restoring the divided form makes it pass again. Try it; a fixture you have never seen fail is a fixture you have not tested.
4. Changing for: 2m to for: 30s makes the eval_time: 1m assertion fail. Change it back.
5. curl -s http://localhost:9090/api/v1/rules reports health=ok and an empty lastError for the rule.
6. Your timeline shows a gap between the driver start and the firing transition of roughly three and a half minutes, and you can attribute each component of it to a line of configuration.
7. curl -s http://localhost:9093/api/v2/alerts returned the alert with severity, team and service all present.
8. After the driver stops, the alert stays firing for about five minutes before resolving, and you can name the key responsible.

Expected Outcome

obs-alert-rule-lab/
├── alertmanager.yml
├── docker-compose.yml
├── drive.sh
├── prometheus.yml
└── rules/
    ├── orders-api.rules.yml
    └── test/
        └── orders-api_test.yml

One rule file whose expression compares a ratio against a ratio threshold, carrying severity, team and service labels and three annotations of which one is a working runbook link. One fixture with a no-fire case and a fire case, both green, and the fire case pinning both the label set and the for: value. A saved diff from the failing draft. A timeline you can read out loud.

Troubleshooting

promtool_run says “permission denied” reading the rule file. The container is uid 65534 and your files are not world-readable. ls -l the directory; chmod a+r rules rules/*.yml rules/test/*.yml fixes it.

The rule does not appear in /api/v1/rules. The glob is /etc/prometheus/rules/*.yml, one level deep, matching .yml only. A file named .yaml, or a file left in a subdirectory, is silently not loaded. Check with docker compose exec prometheus ls /etc/prometheus/rules.

health is err and lastError is populated. The file loaded and the expression failed at evaluation. The message names the problem; a division whose two sides have different label sets is the usual cause here.

The expression returns nothing while the driver is clearly pushing. Query http_requests_total on its own. If it comes back with job="pushgateway" rather than job="orders-api", honor_labels: true is missing from the scrape config — the labels were overwritten at scrape time, and service may have survived while job did not.

The ratio spikes wildly at the start of a phase. You restarted the driver without deleting the Pushgateway group, so the counter went backwards and rate() treated it as a counter reset. Delete the group, wait 30 seconds, start again.

The alert never leaves pending. Poll the raw expression. If the ratio oscillates around 0.05 the for: timer resets on every dip; a single empty evaluation is enough to send it back to inactive.

Alertmanager holds nothing while Prometheus says firing. Ask Prometheus what it thinks its Alertmanagers are: curl -s http://localhost:9090/api/v1/alertmanagers | jq. An empty activeAlertmanagers list means the alerting: block did not resolve — the two containers are not on the same Compose network, or the target name is wrong.

Cleanup

The lab created three containers, two named volumes, one Compose network and one directory. All of it comes back.

Step 1. Stop the containers and remove the volumes:

$ cd "$HOME/obs-alert-rule-lab" && docker compose down -v

Step 2. Confirm nothing of the lab’s is left running, and that the ports you recorded in Task 1 are free again:

docker compose ps
ss -ltnp 2>/dev/null | grep -E ':(9090|9091|9093)\b' || echo 'ports free'

Step 3. Keep the deliverables, then remove the directory.

$ mkdir -p "$HOME/obs-lab-deliverables" && cp -a "$HOME/obs-alert-rule-lab/rules" "$HOME/obs-lab-deliverables/alert-rule-lab-rules" && rm -rf "$HOME/obs-alert-rule-lab"

Step 4. The three images remain in the local cache. Leave them if you are going on to the Alertmanager lab, which uses two of them. Otherwise:

docker image rm prom/prometheus:v2.55.1 prom/pushgateway:v1.10.0 prom/alertmanager:v0.28.1

Production notes

Mapping this exercise onto a real change window:

• The rule change is a code change. It goes through review, and the fixture goes with it in the same commit. A reviewer who cannot see the fixture cannot tell a threshold change from a semantics change.
• Both gates belong in CI. promtool check rules on every rule file, then promtool test rules on every fixture. The first is fast enough to run on every push; the second is fast enough that there is no argument for skipping it.
• Reloading is separate from merging. Prometheus does not watch the filesystem. A deploy sends SIGHUP or posts to /-/reload — which requires --web.enable-lifecycle, as in this lab’s Compose file — and the deploy step should re-run check rules against the file it just wrote, because the file that reaches the server is not always the file that passed CI.
• A new rule starts at warning. Route it somewhere that does not wake anybody for its first week, watch what it does against real traffic, and promote it to critical only once the firing pattern matches the intent. The for: value in particular is an empirical setting, and the only data that can set it is production data.
• Renaming an alert is a breaking change. alertname is the key Alertmanager groups on, dashboards filter on, and silences match on. A rename orphans every silence that referenced the old name.

What You Learned

• A green promtool check rules says the file loads, and nothing else. The draft rule passed every static gate and was wrong by a factor that depended on traffic volume.
• The fixture is where the meaning is asserted. exp_alerts: [] on a healthy input is the single highest-value assertion you can write, and it is the one most often omitted.
• Assert the pending window as well as the firing one. The eval_time: 1m assertion is what pins for: 2m; without it the dwell is undocumented and any value passes.
• Read the rendered annotation out of the diff. promtool prints exactly what the rule produced, which is more reliable than predicting a Go template’s output.
• Four separate delays sit between a breach and a page. Scrape interval, evaluation interval, rate window and for: dwell — and you measured them accumulating instead of reading about them.
• keep_firing_for holds an alert open after the expression goes quiet, which is protection against a flapping recovery and a delay on a genuine one.
• A ratio alert goes quiet when traffic stops. 0/0 is NaN, NaN fails every comparison, and the alert resolves at precisely the moment the service is most broken.