Lab: Build a Recording Rule

Objective

A recording rule is a cache with a contract. The cache part is easy to justify and easy to measure, and this lab measures it: you will run one expression, read what it cost, record it, and read what the recorded form costs instead.

The contract is the part that goes wrong. A rule’s name is a promise about the label set its consumers will receive, and Prometheus does not check that promise — it will happily load a rule called job:something:rate5m that emits one series per CPU core per host, evaluate it every fifteen seconds, and report it as healthy. So will a rule whose source metric was renamed last month and now matches nothing.

By the end you will have caused all three of those failures deliberately, confirmed that none of them produces an error anywhere an operator would look, and written the one artefact that does catch them: a unit test that asserts a value.

Architecture

One Prometheus with a mounted rules directory, scraping three node_exporter containers and itself. The self-scrape is not decoration — it is the only way to see the rule engine’s own health metrics, and half this lab reads them.

  +------------------------------------------------------+
  |  prometheus 2.55  :9090                               |
  |                                                       |
  |   scrape --> exp-a, exp-b, exp-c   (job="node", 15s)  |
  |   scrape --> itself                (job="prometheus") |
  |                                                       |
  |   rule_files: /etc/prometheus/rules/*.rules.yml       |
  |        |                                              |
  |        +-- group cpu-base    interval 15s             |
  |        +-- group cpu-rollup  interval 5m   <- Task 5  |
  +------------------------------------------------------+
                    ^
                    | ./rules  (bind mount, edited on the host)

The three exporters exist to give the recorded expression something wide to aggregate over. node_cpu_seconds_total carries one series per CPU per mode per instance, so on an ordinary lab host the raw expression touches a few hundred series and the recorded form touches one. That gap is the measurement.

Requirements

• A Linux host with Docker Engine 28.x and Docker Compose v2, plus network access on first run to pull two images (roughly 300 MiB).
• curl and jq on the host. Every measurement is an HTTP API call.
• Free TCP port 9090. No exporter ports are published; nothing outside the compose network needs to reach them.
• Roughly 250 MiB of memory and a few hundred MiB of disk.
• Time. The rate windows are five minutes wide and one rule group ticks every five minutes. Several steps say “wait”; they mean it.
• A text editor. Files under ./rules are edited on the host and read by the container through a bind mount.
• No out-of-band access requirement, and nothing outside the lab directory is modified.

Scenario

At 03:14 an on-call engineer opens the CPU dashboard during an unrelated incident. Every panel takes eleven seconds to draw. Prometheus is pinned at one core. The dashboard has fourteen panels, three teams have copies of it, and each copy recomputes the same wide aggregation on every refresh — plus once more per evaluation interval for each of the four alerts built on the same expression.

The obvious fix is a recording rule, and the team writes one that afternoon. Two weeks later the same dashboard is showing one row per CPU core instead of one row per job, an alert that used to page once a quarter has gone silent, and nobody can find an error message about either. The recording rule is reported as healthy. It is healthy. It is also wrong in two different ways that Prometheus has no opinion about.

Your job is to build the rule properly, measure what it actually bought, and then reproduce both silent failures so you can recognise them.

Tasks

Task 1: Build the stack

LABDIR="$HOME/rb-obs-rules"
mkdir -p "$LABDIR/rules"
cd "$LABDIR"

prometheus.yml. Note the rule_files glob: it matches *.rules.yml only, so the unit-test fixture you write in Task 8 can live in the same directory without Prometheus trying to load it as a rule file.

global:
  scrape_interval: 15s
  evaluation_interval: 15s

rule_files:
  - /etc/prometheus/rules/*.rules.yml

scrape_configs:
  - job_name: node
    static_configs:
      - targets: ['exp-a:9100', 'exp-b:9100', 'exp-c:9100']

  # Self-scrape. Without this there is no prometheus_rule_* metric to read,
  # and the rule engine is invisible to the thing it runs inside.
  - job_name: prometheus
    static_configs:
      - targets: ['localhost:9090']

compose.yaml:

name: rb-obs-rules

x-exporter: &exporter
  image: prom/node-exporter:v1.8.2
  restart: unless-stopped

services:
  exp-a:
    <<: *exporter
    container_name: rb-rules-exp-a
  exp-b:
    <<: *exporter
    container_name: rb-rules-exp-b
  exp-c:
    <<: *exporter
    container_name: rb-rules-exp-c

  prometheus:
    image: prom/prometheus:v2.55.1
    container_name: rb-rules-prom
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=3h'
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - ./rules:/etc/prometheus/rules:ro
      - prom-data:/prometheus
    ports:
      - '9090:9090'
    depends_on:
      - exp-a
      - exp-b
      - exp-c

volumes:
  prom-data:

$ docker compose up -d

An empty rules directory is fine: a glob that matches nothing is not an error. Confirm four targets and then wait six minutes, because every expression below is a five-minute rate and a partly-filled window produces numbers that move while you read them.

curl -sG http://localhost:9090/api/v1/query --data-urlencode 'query=up' \
| jq -r '.data.result[] | "\(.metric.job)\t\(.metric.instance)\tup=\(.value[1])"' | sort

Task 2: Measure the expression before you record it

This is the expression from the scenario — the per-job rate of non-idle CPU time, which is a real and useful thing to put on a dashboard and a genuinely wide one to compute:

sum by (job) (rate(node_cpu_seconds_total{mode!="idle"}[5m]))

Ask Prometheus what it costs. Look at the whole stats object once, so you know what fields are on offer before you start extracting them:

Q='sum by (job) (rate(node_cpu_seconds_total{mode!="idle"}[5m]))'
curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode "query=$Q" --data-urlencode 'stats=all' \
| jq '.data.stats'

Two numbers matter. samples.totalQueryableSamples is how many stored samples the engine had to read to answer the question. timings.evalTotalTime is how long that took. Record both:

Q='sum by (job) (rate(node_cpu_seconds_total{mode!="idle"}[5m]))'
curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode "query=$Q" --data-urlencode 'stats=all' \
| jq -r '"BEFORE  series=\(.data.result | length)  samples=\(.data.stats.samples.totalQueryableSamples)  eval=\(.data.stats.timings.evalTotalTime)s"'

The series count is small — one per job. The sample count is not: it is (cores × non-idle modes × instances) series, each contributing about twenty samples for the five-minute window. On an eight-core host that is several thousand samples read, condensed into two numbers, on every single evaluation.

Count the input width yourself so the number has a shape you can reason about:

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=count(node_cpu_seconds_total{mode!="idle"})' \
| jq -r '"input series: \(.data.result[0].value[1])"'

Task 3: Write the rule, check it, load it, verify it

Create rules/cpu.rules.yml. The name is the part to be deliberate about: the level is job because that is what the sum by keeps, the metric is the source metric, and the operation names the function and its window.

groups:
  - name: cpu-base
    # Matches the global evaluation_interval. Stated explicitly because
    # consumers read freshness off this number, and an implicit default
    # changes when somebody edits the global section.
    interval: 15s
    rules:
      # level : metric : operations
      #  job  : node_cpu_seconds : rate5m
      # The sum by (job) below is what makes the "job" level true.
      - record: job:node_cpu_seconds:rate5m
        expr: sum by (job) (rate(node_cpu_seconds_total{mode!="idle"}[5m]))

Validate the file before it goes anywhere near the server. promtool ships inside the Prometheus image, so no host install is needed:

docker compose exec prometheus promtool check rules /etc/prometheus/rules/cpu.rules.yml

promtool checks two things: that the YAML is well formed, and that every expr parses as PromQL. It does not and cannot check that the expression means what you intended — that is Task 8.

Now load it. A SIGHUP re-reads the configuration and the rule files without restarting the process, so the TSDB head and every in-flight scrape survive:

$ docker compose kill -s SIGHUP prometheus

Three verifications, and all three are needed, because each one can pass while the next fails.

Is the rule loaded, and is it healthy?

curl -s http://localhost:9090/api/v1/rules \
| jq -r '.data.groups[] | .name as $g | .interval as $i | .rules[]
         | "\($g)\tinterval=\($i)s\t\(.name)\thealth=\(.health)\tlastError=\"\(.lastError)\""'

Did it write series? A healthy rule that emits nothing is the Task 7 failure, and this is the query that separates the two:

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=job:node_cpu_seconds:rate5m' \
| jq -r '.data.result[] | "\(.metric | tostring)\t\(.value[1])"'

Does the recorded series agree with the expression it replaced? This is the check nobody runs, and it is one subtraction:

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=job:node_cpu_seconds:rate5m - sum by (job) (rate(node_cpu_seconds_total{mode!="idle"}[5m]))' \
| jq -r '.data.result[] | "drift for job=\(.metric.job): \(.value[1])"'

A small non-zero number is expected and is not drift: the rule was evaluated up to fifteen seconds ago and the live expression is being evaluated now, over a window that has moved. A number of the same order as the value itself is a real disagreement.

Task 4: Measure what recording bought, and what it moved

Run the same stats query against the recorded series:

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=job:node_cpu_seconds:rate5m' --data-urlencode 'stats=all' \
| jq -r '"AFTER   series=\(.data.result | length)  samples=\(.data.stats.samples.totalQueryableSamples)  eval=\(.data.stats.timings.evalTotalTime)s"'

The sample count collapses to the number of output series — one stored sample per series, read directly. The reduction is two or three orders of magnitude and it is not a micro-optimisation: it is the difference between a panel that draws and a panel that times out.

Now be honest about the other half. The expression still runs; it runs in the rule engine, once per group interval, whether or not anybody looks at the dashboard. Read what that costs:

curl -s http://localhost:9090/api/v1/rules \
| jq -r '.data.groups[] | "group=\(.name)  last evaluation took \(.evaluationTime)s, interval \(.interval)s"'

The trade you made: a cost proportional to the number of queries became a cost proportional to time. That is a win when the expression is read many times per interval and a loss when it is read once a week. A rule that feeds one panel that one person opens each Monday is pure overhead, evaluated 40,320 times between readings at a fifteen-second interval.

Task 5: Layer a second rule, and measure the lag you introduce

Add a rollup that consumes the first rule, and put it deliberately in the wrong place — a separate group with a much longer interval. Append to rules/cpu.rules.yml:

  # Deliberately a separate group on a slow interval. Task 5 measures
  # what that does to the freshness of everything downstream.
  - name: cpu-rollup
    interval: 5m
    rules:
      - record: cluster:node_cpu_seconds:rate5m
        expr: sum(job:node_cpu_seconds:rate5m)

docker compose exec prometheus promtool check rules /etc/prometheus/rules/cpu.rules.yml
docker compose kill -s SIGHUP prometheus

Wait five minutes for the slow group to tick at least once. The rule engine publishes its own health, and the self-scrape from Task 1 is what makes it readable. Find the family first rather than trusting a remembered metric name:

curl -s http://localhost:9090/api/v1/label/__name__/values \
| jq -r '.data[] | select(startswith("prometheus_rule"))'

The one that answers this task is the last-evaluation timestamp, one series per rule group. Subtract it from time() and you have each group’s staleness in seconds:

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=time() - prometheus_rule_group_last_evaluation_timestamp_seconds' \
| jq -r '.data.result[] | "\(.metric.rule_group)\tlast evaluated \(.value[1]) seconds ago"'

curl -s http://localhost:9090/api/v1/rules \
| jq -r '.data.groups[] | "\(.name)\tinterval=\(.interval)s\tlast eval took \(.evaluationTime)s"'

cpu-base is never more than fifteen seconds stale. cpu-rollup is anything up to five minutes stale, which means the cluster-level number on a dashboard can be five minutes behind the per-job numbers sitting next to it — from the same source data, on the same page, with nothing to indicate the difference.

Confirm the two layers disagree by roughly that lag:

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=cluster:node_cpu_seconds:rate5m - sum(job:node_cpu_seconds:rate5m)' \
| jq -r '"rollup minus live sum of its own input: \(.data.result[0].value[1])"'

Run that subtraction a few times over a couple of minutes. It is near zero immediately after the slow group ticks and drifts further from zero the longer you wait, which is the shape of a stale cache and not the shape of a bug.

Two more metrics belong on a meta-monitoring dashboard: the failure counter and the missed-iteration counter. A group whose evaluation takes longer than its own interval skips ticks, silently:

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=prometheus_rule_evaluation_failures_total' \
| jq -r '.data.result[] | "failures \(.metric.rule_group // "all"): \(.value[1])"'

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=prometheus_rule_group_iterations_missed_total' \
| jq -r '.data.result[] | "missed iterations: \(.value[1])"'

Both should be zero here. Neither is zero on a server where somebody has added an expensive rule to a fast group, and neither raises an alert unless you write one.

Task 6: The name is a contract, and nothing enforces it

Add a rule whose name says one level and whose expression produces another — which is precisely the change the team in the scenario made two weeks later, and which passes code review whenever the reviewer reads the name and not the sum by. Add it to the end of the cpu-base group’s rules: list — that is now in the middle of the file, above the cpu-rollup group you added in Task 5:

      # WRONG ON PURPOSE. The name promises the job level; the expression
      # keeps instance and cpu. Prometheus has no opinion about this.
      - record: job:node_cpu_seconds:rate5m:percpu
        expr: sum by (job, instance, cpu) (rate(node_cpu_seconds_total{mode!="idle"}[5m]))

docker compose exec prometheus promtool check rules /etc/prometheus/rules/cpu.rules.yml
docker compose kill -s SIGHUP prometheus

Wait thirty seconds, then count the damage:

for R in job:node_cpu_seconds:rate5m job:node_cpu_seconds:rate5m:percpu; do
  printf '%-42s ' "$R"
  curl -sG http://localhost:9090/api/v1/query --data-urlencode "query=count($R)" \
  | jq -r '.data.result[0].value[1] // "0"'
done

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=job:node_cpu_seconds:rate5m:percpu' \
| jq -r '.data.result[0].metric | keys | join(", ")' | sed 's/^/labels on the output: /'

One series against dozens, from two rules that differ by one sum by clause. promtool check rules passed. /api/v1/rules reports health=ok. There is no log line, no counter, no API field anywhere that says the name and the expression disagree — the check is a human reading two lines of YAML next to each other, which is the entire reason the naming convention exists.

Task 7: Two failures that report themselves as healthy

The rule whose source does not exist. Add this to the end of the cpu-base group’s rules: list, alongside the one from Task 6:

      # The source metric does not exist in this TSDB. Watch what
      # Prometheus reports about it.
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))

docker compose kill -s SIGHUP prometheus
sleep 30

curl -s http://localhost:9090/api/v1/rules \
| jq -r '.data.groups[].rules[] | select(.name | startswith("job:http")) | "name=\(.name) health=\(.health) lastError=\"\(.lastError)\""'

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=job:http_requests:rate5m' \
| jq -r '"series produced: \(.data.result | length)"'

health=ok, lastError empty, zero series. This is correct behaviour and it is indistinguishable from a rule that is working on a metric that happens to be quiet. Every renamed exporter metric, every decommissioned job, every typo in a source metric name lands here — a rule that evaluates perfectly and produces nothing, feeding a panel that reads “No data” and an alert that cannot fire.

The rule file that will not load. Now break the file. Add one more rule to the same list, with an expression that does not parse — note the missing closing parenthesis:

      - record: job:broken:rate5m
        expr: sum by (job) (rate(node_cpu_seconds_total[5m])

Catch it the way CI would, before it reaches the server:

docker compose exec prometheus promtool check rules /etc/prometheus/rules/cpu.rules.yml \
  || echo "promtool rejected the file, exit $?"

Then send the reload anyway, and watch what the server does with it:

docker compose kill -s SIGHUP prometheus
sleep 5

docker compose logs --tail=15 prometheus

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=prometheus_config_last_reload_successful' \
| jq -r '"config reload successful: \(.data.result[0].value[1])"'

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=job:node_cpu_seconds:rate5m' \
| jq -r '"the good rule is still producing: \(.data.result | length) series"'

prometheus_config_last_reload_successful is 0, the log names the file and the parse error — and the previously loaded rules keep running, because a failed reload is rejected wholesale and the server continues on the last good configuration.

Repair the file before continuing — delete the job:broken:rate5m rule, then confirm the reload succeeds:

docker compose exec prometheus promtool check rules /etc/prometheus/rules/cpu.rules.yml
docker compose kill -s SIGHUP prometheus
sleep 5
curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=prometheus_config_last_reload_successful' \
| jq -r '"config reload successful: \(.data.result[0].value[1])"'

Task 8: Assert the value, not the existence

Everything in Task 6 and Task 7 was invisible to promtool check rules, because that command asks whether the file parses. The command that asks whether the rule is right is promtool test rules, and its fixtures run against synthetic input with no server involved.

Create rules/cpu.test.yml. Prometheus will not load it — the rule_files glob matches *.rules.yml only — and promtool resolves the rule_files path relative to the fixture’s own directory.

Write the assertion wrong on purpose. The point of the first run is to see what the runner tells you:

rule_files:
  - cpu.rules.yml

evaluation_interval: 1m

tests:
  - interval: 1m
    name: per-job non-idle CPU rate excludes idle and aggregates to the job
    input_series:
      # One core burning exactly 30 CPU-seconds per minute in user mode.
      - series: 'node_cpu_seconds_total{job="node",instance="exp-a:9100",cpu="0",mode="user"}'
        values: '0+30x10'
      # Idle on the same core. The rule must not count this.
      - series: 'node_cpu_seconds_total{job="node",instance="exp-a:9100",cpu="0",mode="idle"}'
        values: '0+30x10'
    promql_expr_test:
      - expr: job:node_cpu_seconds:rate5m
        eval_time: 10m
        exp_samples:
          - labels: 'job:node_cpu_seconds:rate5m{job="node"}'
            value: 1        # WRONG ON PURPOSE - see what promtool says

  - interval: 1m
    name: a counter that does not move produces a rate of zero
    input_series:
      - series: 'node_cpu_seconds_total{job="node",instance="exp-a:9100",cpu="0",mode="user"}'
        values: '100+0x10'
    promql_expr_test:
      - expr: job:node_cpu_seconds:rate5m
        eval_time: 10m
        exp_samples:
          - labels: 'job:node_cpu_seconds:rate5m{job="node"}'
            value: 0

docker compose exec prometheus promtool test rules /etc/prometheus/rules/cpu.test.yml \
  || echo "exit $?"

The first test fails and the failure names the expression, the evaluation time, the expected sample and the sample it actually got. Read the number it got, then derive it independently before you trust it:

• The user-mode counter rises by 30 every minute, so its true rate is 0.5 CPU-seconds per second.
• The idle series rises identically and is excluded by mode!="idle", so it contributes nothing. If the matcher were wrong the answer would be 1, which is exactly the wrong value you asserted — a fixture that catches a real mistake.
• sum by (job) has one series to sum, so the job-level answer is 0.5.

Change value: 1 to value: 0.5, re-run, and both tests pass.

Validation

Four checks. Each one proves a claim rather than repeating a step.

1. Recording measurably reduced the read cost of the expression. Both measurements, side by side:

Q='sum by (job) (rate(node_cpu_seconds_total{mode!="idle"}[5m]))'
for E in "$Q" 'job:node_cpu_seconds:rate5m'; do
  curl -sG http://localhost:9090/api/v1/query \
    --data-urlencode "query=$E" --data-urlencode 'stats=all' \
  | jq -r '"samples=\(.data.stats.samples.totalQueryableSamples)\teval=\(.data.stats.timings.evalTotalTime)s"'
done

The second line must show a sample count smaller by orders of magnitude.

2. Every loaded rule is healthy, and you can say which ones produce nothing.

curl -s http://localhost:9090/api/v1/rules \
| jq -r '.data.groups[].rules[] | "\(.name)\thealth=\(.health)"'

for R in job:node_cpu_seconds:rate5m cluster:node_cpu_seconds:rate5m job:http_requests:rate5m; do
  printf '%-42s ' "$R"
  curl -sG http://localhost:9090/api/v1/query --data-urlencode "query=count($R)" \
  | jq -r '.data.result[0].value[1] // "0 series - healthy and empty"'
done

Every rule healthy; job:http_requests:rate5m healthy and empty. That combination is the finding, not a fault in the check.

3. The naming violation is countable. The two rules that differ only by a sum by clause differ enormously in output width:

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=count(job:node_cpu_seconds:rate5m:percpu) - count(job:node_cpu_seconds:rate5m)' \
| jq -r '"extra series produced by the mis-levelled rule: \(.data.result[0].value[1])"'

4. The unit tests pass, and they assert values. A zero exit status, plus the evidence that the fixture is not a presence check — set value: 0.5 back to 1 briefly and confirm the run fails:

docker compose exec prometheus promtool test rules /etc/prometheus/rules/cpu.test.yml \
  && echo "unit tests PASS"

Expected Outcome

• Four containers running; one Prometheus scraping three exporters and itself, with a bind-mounted rules directory.
• Four recording rules loaded across two groups: one correct, one deliberately mis-levelled, one whose source metric does not exist, and one rollup in a slow group.
• A before-and-after cost measurement for one expression, in queryable samples and evaluation time, with a sentence about which cost was removed and which was converted into a per-interval cost.
• A measured evaluation lag between two rule groups on different intervals, and the subtraction that shows the rollup disagreeing with its own input.
• A recorded observation that a healthy rule can produce zero series, and that a broken rule file is ignored on reload and fatal at startup.
• A passing promtool test rules fixture that asserts values, plus the failure output from the run where the asserted value was wrong.

Troubleshooting

• promtool check rules says the file does not exist. The path is the container’s path, not the host’s: /etc/prometheus/rules/cpu.rules.yml. The bind mount makes the host directory visible there.
• The SIGHUP reload appears to do nothing. Check prometheus_config_last_reload_successful. A value of 0 means the reload was rejected and the old configuration is still live; docker compose logs prometheus names the file and line.
• A new rule reports health=unknown. It has not been evaluated yet. Wait one group interval — up to five minutes for cpu-rollup.
• prometheus_rule_group_last_evaluation_timestamp_seconds returns nothing. Either the self-scrape is missing from prometheus.yml or no rule group has evaluated yet. Confirm up{job="prometheus"} is 1, then re-run the __name__ listing to see which prometheus_rule_* metrics this build exposes.
• The stats fields are missing from the response. stats=all must be sent as a parameter alongside query. Dump .data.stats on its own first to confirm the shape before extracting fields from it.
• totalQueryableSamples is much smaller than expected. The TSDB has less than five minutes of data, so the rate windows are not full. Wait six minutes after up -d before taking any measurement.
• The unit test fails on the second case with “no samples”. 100+0x10 is the notation for a value that repeats; a bare 100x10 behaves differently across versions. Use the explicit +0 form.
• Prometheus exits at startup after an edit. A rule file that fails to parse is fatal at process start. Run promtool check rules before docker compose up, and remember that the same file would only have been ignored on reload.

Cleanup

Everything the lab created is one directory, one compose project and one named volume.

$ cd ~/rb-obs-rules && docker compose down -v

docker volume ls | grep rb-obs-rules || echo "volumes gone"
rm -rf "$HOME/rb-obs-rules"

The rule files lived only in the lab directory and were mounted read-only, so nothing outside it was written. No host networking, firewall or systemd state was touched.

Production notes

Record what is read many times, not what is expensive once. The trade is a per-query cost for a per-interval cost. Before adding a rule, count the readers: dashboards times panels times refresh rate, plus every alert on the same expression. A rule feeding one weekly panel is evaluated tens of thousands of times between readings.

The recorded metric’s name and label set are a published interface. Changing the sum by clause of a live rule is a breaking change to every dashboard and alert downstream, and it produces no error anywhere. Change the name at the same time, run both rules for a deprecation window, and delete the old one once nothing references it.

Put dependent rules in the same group as their input. Rules inside a group evaluate in declaration order against the same tick, so a rule can read the output another rule just wrote. Across groups there is no ordering guarantee and, when the intervals differ, a guaranteed lag — which is what you measured in Task 5. Split groups by cost and interval, never by topic, and keep a dependency chain inside one group.

Alert on the rule engine itself. These three cover the failures in this lab that produce no error:

groups:
  - name: rule-engine-health
    interval: 1m
    rules:
      - alert: PrometheusConfigReloadFailed
        expr: prometheus_config_last_reload_successful == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: 'Prometheus is running on a stale config; the next restart will fail'

      - alert: PrometheusRuleEvaluationFailures
        expr: increase(prometheus_rule_evaluation_failures_total[10m]) > 0
        for: 5m
        labels:
          severity: warning

      - alert: PrometheusRuleGroupMissedIterations
        expr: increase(prometheus_rule_group_iterations_missed_total[10m]) > 0
        for: 5m
        labels:
          severity: warning
          summary: 'A rule group evaluates slower than its own interval'

Gate rule files in CI on both commands. promtool check rules catches the file that would kill the process at its next restart. promtool test rules catches the refactor that keeps the name and changes the number. The first is non-negotiable; the second is what stops a recording rule from becoming a confidently-wrong dashboard.

Every rule needs a test that asserts a value, and one that asserts the quiet case. Presence assertions pass for every failure in this lab. A fixture that pins both the busy value and the zero value catches unit errors, window changes and dropped matchers, and it costs about ten lines per rule.

Write the level in the name and make reviewers read the sum by. The convention level:metric:operations only works if the review question is “does the sum by clause produce the level in the name”. Nothing in the toolchain asks that question, which makes it a checklist item rather than a preference.

What You Learned

• The cost a dashboard pays is the sample count, not the series count. You measured several thousand queryable samples for the raw expression and a handful for the recorded one.
• Recording converts a per-query cost into a per-interval cost. That is a win when the expression is read often and pure overhead when it is not, and the break-even is countable before you write the rule.
• A rule’s name is a promise nothing enforces. Two rules differing by one sum by clause both loaded, both reported health=ok, and produced output differing by orders of magnitude in width.
• health=ok and zero series is a normal, silent, permanent state. It is what a renamed source metric looks like, and it is indistinguishable from a quiet service on every panel and in every alert.
• A broken rule file is ignored on reload and fatal at startup, so it can sit merged and inert until an unrelated restart turns it into an outage. prometheus_config_last_reload_successful is the metric that sees it coming.
• promtool check rules asks whether the file parses; promtool test rules asks whether the rule is right. Only the second one catches the refactor that keeps the name and changes the number.