Recording Rule Tests

An engineer refactors a recording rule from sum by (job) (rate(http_requests_total[5m])) to sum by (job) (rate(http_requests_total[1m])). The local promtool test rules run passes. The change merges. The production latency dashboard spikes: every panel that read the recorded metric now reads a much noisier number because the rate window changed from five minutes to one minute. The test passed because the fixture asserted only that the recorded metric existed, not that the rate window produced the number the dashboard expected.

This is the failure shape that the recording-rule lesson exists to prevent. A recording rule test that asserts only “the metric is there” is the same as no test at all. The unit-test fixture for a recording rule must assert on the metric value, the label set, and the timestamp.

What it is

A recording rule test is a fixture entry whose assertion block is unit_test: (not alert_rule_test:). The block contains exp_series: which is a list of expected metric outputs. Each output is a metrics: entry that declares:

• name: — the recorded metric name (the value of the rule’s record:).
• labels: — the label set the rule produces for the output series.
• value: — the exact numeric value the rule produces at the eval_time for the synthetic input.

The runner compares the rule’s actual output against the exp_series: list at every eval_time. Any divergence in metric name, label set, or value fails the test. The discipline of asserting on value, not just presence, is the core of recording-rule testing.

The contrast with alerting-rule tests is sharp. An alerting rule fires or does not fire; the assertion is a list of labels and annotations. A recording rule produces a number; the assertion is the number itself. Both use the same fixture infrastructure (rule_files:, evaluation_interval:, tests:, input_series:), but the assertion block is different, and the discipline is different.

Why a sysadmin cares

Recording rules are caches. Every dashboard panel, every alert, every SLO computation that reads the recorded metric consumes the number it produces. A wrong number poisons the downstream chain. A test that asserts only presence is indistinguishable from no test.

The failure shapes are specific and recurrent:

• Off-by-factor bugs. A refactor changes a constant from seconds to milliseconds, or divides by 60 instead of multiplying. The metric exists, the labels are right, the number is wrong by a factor of 1000. A presence assertion passes; a value assertion catches it.
• Aggregation-level drift. A rule’s record: name says job:... but the expr: retains instance. The metric exists with extra labels; dashboards that grouped by job see one panel per instance. A label-set assertion catches it; a presence assertion does not.
• Rate-window miscalibration. A rate over [1m] versus [5m] produces wildly different numbers under bursty load. A presence assertion passes; a value assertion catches the window mismatch.

For each shape, the test that catches it asserts on the specific property that changed. The value assertion is the single tool that catches all three.

How it works

The runner’s path through a recording-rule test is the same as for alerting rules up to the assertion:

  rule_files: [recording.yml]
        |
        v
  in-memory storage seeded with input_series
        |
        v
  rule manager starts with evaluation_interval
        |
        v
  for each test:
        |   tick the clock to eval_time
        |   evaluate the recording rule
        |   capture the output vector
        |   compare against unit_test.exp_series
        v
  report PASS or FAIL with metric name, labels, value diff

The difference is the comparison. For alerting rules, the runner compares the alert state vector (which alerts are pending, which are firing, with what labels and annotations). For recording rules, the runner compares the metric output vector: which metrics exist, with what labels, and with what numeric values at the eval_time.

The runner does not approximate. A value of 0.03333333333333333 must match exactly. A label set of job="orders-api" must match exactly. A metric name of job:http_requests:rate5m must match exactly. This is deliberate: a recording rule’s contract is precise, and a test that approximates the contract lets bugs through.

How to configure it

A worked fixture for two recording rules. The first asserts a per-job request rate; the second asserts a histogram quantile that consumes the first.

# rules/test/recording_test.yml
rule_files:
  - ../recording.yml

evaluation_interval: 30s

tests:
  # ---- Recording rule: per-job request rate over 5m ----
  # The rule under test:
  #   record: job:http_requests:rate5m
  #   expr:   sum by (job) (rate(http_requests_total[5m]))
  - interval: 30s
    name: per-job rate is the count of increments over the window
    input_series:
      - series: 'http_requests_total{job="orders-api",status="200"}'
        # 1 increment per 30s for 5 minutes = 10 samples
        values: '0+1x10'
      - series: 'http_requests_total{job="checkout-api",status="200"}'
        # 2 increments per 30s for 5 minutes = 10 samples
        values: '0+2x10'
    unit_test:
      - eval_time: 5m
        exp_series:
          - metrics:
              - name: job:http_requests:rate5m
                labels: 'job="orders-api"'
                # 10 increments / 300s = 0.0333... per second
                value: 0.03333333333333333
              - name: job:http_requests:rate5m
                labels: 'job="checkout-api"'
                # 20 increments / 300s = 0.0666... per second
                value: 0.06666666666666667

  # ---- Recording rule: sub-threshold stays sub-threshold ----
  - interval: 30s
    name: per-job rate is zero when no increments occur
    input_series:
      - series: 'http_requests_total{job="orders-api",status="200"}'
        # constant for 10 samples, no increments
        values: '100x10'
    unit_test:
      - eval_time: 5m
        exp_series:
          - metrics:
              - name: job:http_requests:rate5m
                labels: 'job="orders-api"'
                value: 0

  # ---- Recording rule: counter reset ----
  # A counter that drops mid-window should produce a high rate
  # because rate() detects the reset.
  - interval: 30s
    name: counter reset produces a non-zero rate
    input_series:
      - series: 'http_requests_total{job="orders-api",status="200"}'
        # 5 samples growing, then a reset, then 5 more samples
        values: '0+1x5 0+1x5'
    unit_test:
      - eval_time: 5m
        exp_series:
          - metrics:
              - name: job:http_requests:rate5m
                labels: 'job="orders-api"'
                # The reset is detected; rate is non-zero across
                # the whole 5m window.
                value: 0.016666666666666666

  # ---- Recording rule: histogram quantile that consumes the rate ----
  # The rule under test:
  #   record: job:http_request_duration:p99
  #   expr:   histogram_quantile(0.99,
  #             sum by (job, le) (rate(http_request_duration_seconds_bucket[5m])))
  - interval: 30s
    name: p99 latency quantiles are produced per job
    input_series:
      - series: 'http_request_duration_seconds_bucket{job="orders-api",le="0.1"}'
        values: '0+10x10'
      - series: 'http_request_duration_seconds_bucket{job="orders-api",le="0.5"}'
        values: '0+50x10'
      - series: 'http_request_duration_seconds_bucket{job="orders-api",le="+Inf"}'
        values: '0+50x10'
      - series: 'http_request_duration_seconds_bucket{job="orders-api",le="1.0"}'
        values: '0+50x10'
    unit_test:
      - eval_time: 5m
        exp_series:
          - metrics:
              - name: job:http_request_duration:p99
                labels: 'job="orders-api"'
                # 50 observations all in the 0.5 bucket; p99 = 0.5
                value: 0.5

Four scenarios. Each asserts a specific value the rule produces for a specific input shape. The presence-only failure mode is impossible here: a wrong value produces a numeric mismatch and the test fails.

Notice the precision of the value: fields. promtool compares floats exactly. Use the value the rule produces for the synthetic input. For rate() over a constant-rate counter, the value is increments / seconds. For a histogram_quantile(), the value is the bucket boundary that contains the quantile.

How to validate it

Three checks.

# 1. The rule file parses.
promtool check rules rules/recording.yml

Expected output:

Checking rules/recording.yml
  SUCCESS: found 3 rules

# 2. The fixture passes.
promtool test rules rules/test/recording_test.yml

Expected output:

SUCCESS

A failure names the metric, the labels, and the value mismatch:

FAILED
  test: per-job rate is the count of increments over the window
  eval_time: 5m
    metric: job:http_requests:rate5m{job="orders-api"}
    expected: 0.03333333333333333
    actual:   0.03333333333333334

A floating-point mismatch like the above is real: the rule produces a slightly different float than the fixture expects because of internal sample ordering. Compute the expected value with the same arithmetic the rule uses (rate() against the synthetic series) and copy the value exactly.

# 3. The recorded metric is queryable in production.
curl -s 'http://localhost:9090/api/v1/query?query=job:http_requests:rate5m' \
  | jq '.data.result[] | {metric, value}'

Expected output (illustrative):

{ "metric": {"job": "orders-api"}, "value": [1755043200, "0.03333333333333333"] }

A value matching the fixture confirms the rule produced the same number in production as it did in the test. A mismatch means the production input series differs from the synthetic one; investigate the source metric.

How it can fail

Six specific failure modes for recording-rule tests.

1. Presence-only assertion. Symptom: the fixture passes even when the rule produces a wrong number. Cause: the exp_series: entry omits value:. Fix: assert the exact value the rule produces for the synthetic input.
2. Wrong metric name. Symptom: missing series failure that names the exp_series: entry but no rule output. Cause: the name: field in exp_series: does not match the rule’s record: (typo, refactor renamed the rule). Fix: read the rule’s record: and copy it verbatim.
3. Label set drift. Symptom: unexpected output failure with extra labels on the rule output. Cause: the rule’s expr: retains labels the record: name implies it should aggregate away. Fix: align the sum by clause with the record: level, or update exp_series: labels to match the actual output.
4. Rate window too short for eval_time. Symptom: the fixture reports no samples for series at eval_time. Cause: rate(http_requests_total[5m]) needs at least five minutes of data before eval_time. Fix: extend values: or move eval_time later.
5. Floating-point mismatch. Symptom: value mismatch with expected and actual differing by a few ULPs. Cause: rate() against the synthetic series produces a float with slightly different ordering than the hand- computed fixture value. Fix: run the rule once, copy the actual value, paste into exp_series:.
6. Missing eval_time after the for: dwell of an alerting rule that consumes this recording rule. Symptom: the recording rule test passes but the alerting rule that consumes it fails. Cause: the alerting rule needs the recorded metric to exist at every tick across the dwell; a single-eval_time fixture leaves the alerting rule untested for the dwell.

How to troubleshoot it

In order:

1. Run the fixture with --debug if available. The runner names the failing test, the metric, the labels, and the expected versus actual value.
2. Compare metric names. Read the rule’s record: and the exp_series: name: side by side. A typo here is the most common cause of missing series.
3. Compare label sets. List the labels the rule produces (the result of the sum by clause plus any static labels) and compare against the exp_series: labels: string. A sum by (job) rule produces a label of job only; the exp_series: must say labels: 'job="orders-api"'.
4. Compute the expected value by hand. For sum by (job) (rate(http_requests_total[5m])) against 0+1x10 (one increment per 30s for 5m), the value is 10 / 300 = 0.0333.... If the fixture value disagrees, the fixture is wrong.
5. Confirm the synthetic series has data at eval_time. The last sample in values: must be at or after eval_time. A series that ends early produces an out-of-range evaluation.
6. Run the rule once in production, copy the actual value, and paste it. A floating-point mismatch is easier to fix by adopting the actual value than by recomputing it by hand.

Security implications

Recording-rule fixtures carry the same risks as alerting-rule fixtures: committed secrets, internal topology in series names, and rule expressions that reveal internal naming conventions. The risks multiply because recording-rule tests tend to use more series per scenario (the rule aggregates), which means more strings that could leak internal information.

Two additional risks specific to value assertions:

1. Fixture values reveal thresholds. A value: 0.03333333333333333 in a fixture for a request rate rule reveals the input rate the test was written against. If the test uses production-shaped numbers, the fixture reveals the production rate. Use rounded or anonymised values for fixtures that reach public repositories.
2. histogram_quantile() fixtures leak bucket boundaries. A fixture that asserts a specific quantile value against specific le boundaries reveals the application’s bucket design. Treat that information as internal.

Performance implications

A recording-rule fixture runs in milliseconds. The cost is in CI minutes, not in production CPU. The cost grows linearly with the number of scenarios and the size of the synthetic series. A fixture with twenty scenarios and hundred-sample series runs in hundreds of milliseconds; still fast.

The hidden cost is in the test author’s time. Computing the expected value for a rate() against a synthetic series requires running the arithmetic by hand or running the rule once and copying the actual value. Plan for the authoring time when budgeting recording-rule test work.

Production guidance

• Assert value, label set, and timestamp. A presence assertion is the failure shape in disguise. Every exp_series: entry must include value:.
• Use the rule’s actual value when in doubt. A floating-point mismatch is easier to fix by copying the rule’s output than by recomputing it by hand. The cost of a slightly off expected value is a CI failure; the cost of a hand-computed wrong value is a wrong test.
• One fixture per rule file. A fixture that asserts every recording rule in a multi-megabyte rules.yml is hard to maintain. Split by domain.
• Re-author fixtures after every recording-rule refactor. A change to the expr: or record: is a change to the output; the fixture must change with it. The PR that changes the rule should change the fixture in the same commit.
• Wire the fixture into CI. A test that does not run is not a test. The CI step that runs promtool test rules against every fixture under rules/test/ is the enforcement mechanism.

Verification

You should now be able to answer:

• What three fields must a metrics: entry in exp_series: declare, and why is each one necessary?
• What is the difference between a presence assertion and a value assertion, and which one catches an off-by-factor refactor?
• Why does a rate() over [5m] need at least five minutes of synthetic data before the fixture’s eval_time?
• What is the recommended approach for a floating-point mismatch between expected and actual value?
• Why does re-authoring fixtures after every recording-rule refactor matter, and what is the consequence of skipping it?

Quiz