Recording Rule Purpose · Observability

At 03:14 an on-call engineer opens the latency dashboard. Every panel that touches p99 hangs for twelve seconds. Prometheus is at 99 percent CPU. The dashboard uses histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))) on every panel load, across forty instances, in six clusters. The query has no record. Every dashboard refresh re-runs it. The cluster runs four dashboards with the same panel; Alertmanager also evaluates the same expression every thirty seconds. The cost is paid over and over.

This is the operational shape that recording rules exist to prevent. A recording rule stores the result of a PromQL expression as a new time series. The expression is evaluated at a fixed interval. The result is written to the local TSDB under the name given by record:. Dashboards, alerts, and other rules read the cached series instead of recomputing the expression on every query.

A team without recording rules pays the query cost on every alert evaluation and every dashboard load. A team with the right recording rules pays a constant cost per metric per evaluation interval. The difference is operational.

What it is

A recording rule is a YAML-stored PromQL expression with a name. The Prometheus server evaluates the expression at a fixed interval (configurable per group, default evaluation_interval from the global section of prometheus.yml). The result is written to the local TSDB as one or more new series. The new series is queryable immediately after evaluation completes. From the query engine’s perspective, the rule output is just another time series: it obeys the same retention rules, the same label semantics, and the same PromQL grammar as any raw metric.

The shape of a rule is:

groups:
  - name: <group_name>
    interval: 30s          # optional, defaults to global evaluation_interval
    rules:
      - record: <new_metric_name>
        expr: <PromQL expression>
        labels:             # optional, static labels added to every output series
          team: payments

Three pieces matter:

record: is the name the new metric is stored under. The convention is level:metric_name:operations (lesson 02 covers the convention in depth).
expr: is the PromQL expression to evaluate. It can reference raw metrics and other recording rules.
labels: is an optional map of static labels applied to every output series. Useful for ownership (team, service, tier).

Recording rules and alerting rules share the same YAML format and the same evaluation engine. The difference is the output:

Recording rule produces a new time series. Consumers are dashboards, alerts, and other rules.
Alerting rule produces a pending or firing alert. Consumers are Alertmanager and the on-call rotation.

A team that uses recording rules for SLO inputs and alerting rules for SLO breaches gets the right division of labour. The recording rule produces the SLI. The alerting rule produces the page. The two are coupled through the metric name.

Why a sysadmin cares

Three operational pain points disappear when the right recording rules are in place.

Dashboard load time becomes independent of the query cost. A panel that reads from a recording rule evaluates a single metric lookup. A panel that recomputes histogram_quantile() per render pays the cost every time the dashboard loads.
Alert evaluation cost stays bounded. An alert that reads job:http_requests:rate5m evaluates against a precomputed series. An alert that computes the rate itself pays the rate cost per evaluation; if there are ten copies of the same alert on different teams, the cost multiplies.
SLO computations become stable. The SLI is computed once per minute and reused across dashboards and alerts. Both consumers read the same number; the alert cannot disagree with the dashboard because they read the same source.

The trade-off is also real: a recording rule costs CPU every evaluation interval regardless of whether the result is consumed. A rule that feeds one dashboard loaded once per hour by one user is wasted work. The lesson is to record what is computed many times, not what is computed once.

How it works

The flow is the same as a regular PromQL query, but the result is persisted.

   Raw metric (e.g. http_requests_total)
       |
       |  PromQL expression evaluated at the group's interval
       v
   Recording rule output
   (e.g. job:http_requests:rate5m)
       |
       +--- Dashboard panel A
       +--- Dashboard panel B
       +--- Alerting rule
       +--- Other recording rules

The evaluator is a goroutine per group, started by the rule manager in rules/manager.go. Each goroutine ticks at the group’s interval, evaluates its rules in declaration order, and writes the results to the local TSDB. The goroutine is independent of other groups; a slow group does not block a fast group.

When recording rules pay off

Three patterns appear repeatedly in production rule sets.

Rate-of-rate. A five-minute rate of a one-minute rate is expensive. The outer rate requires the inner rate to be evaluated first. A recording rule stores the inner rate; the outer expression becomes a single rate call against a precomputed series.
Top-k on a wide input. topk(10, sum by (job, instance) (rate(...))) requires a fan-out over every label set on every query. A rule precomputes the per-(job, instance) rate; the top-k is a cheap selection against the precomputed series.
A large join. A recording rule can hold the join key stable (a label derived from another metric, a topology lookup). The downstream panel reads the join result without recomputing it. The rule is the cache.

When a rule does not pay off:

The expression is trivial and evaluated by one panel. The rule adds overhead with no benefit.
The expression is computed once per dashboard load by a panel nobody opens. Same reasoning.
The rule output is consumed by a single alert that fires infrequently. Recomputing per alert evaluation is cheap.

Under the hood: rule evaluation

Rule files are referenced from rule_files: in prometheus.yml. At startup, and on every SIGHUP, the rule manager reads each file and constructs an in-memory model of the rule groups. Each group is dispatched to its own goroutine. The goroutine ticks at the group’s interval, evaluates the rules in declaration order, and writes the results back to the TSDB through the same appender path used by scraped samples.

The evaluation pipeline:

Tick. The group’s ticker fires.
Snapshot. The query engine reads the relevant series from the TSDB head block at the tick timestamp.
Evaluate. Each rule’s expression runs against the snapshot.
Write. The result is converted to a ref/data write and appended to the head block.
Refresh. The query engine re-reads the rule’s output series for any later rule in the same group that depends on it.

Evaluation failures are tracked per-rule in prometheus_rule_evaluation_failures_total and logged with the rule name and the underlying error. A rule that fails to evaluate because its source metric is absent produces no output series for that tick; the rule’s previous value remains in the TSDB until the next successful evaluation or until staleness markers expire it.

How to configure it

A typical recording-rules file in /etc/prometheus/rules/api.recording.yml:

# Per-job request rate. Consumed by:
#   - dashboard latency-overview
#   - alert HighRequestRate
#   - SLO rule job:api:request_rate (lesson 06)
groups:
  - name: api-recording
    interval: 30s
    rules:
      - record: job:http_requests_total:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))

      - record: job:http_requests_total:irate1m
        expr: sum by (job) (irate(http_requests_total[1m]))

      - record: job:http_request_duration_seconds:p99
        expr: |
          histogram_quantile(
            0.99,
            sum by (job, le) (
              rate(http_request_duration_seconds_bucket[5m])
            )
          )

Three things to notice:

Group interval: 30s. Faster than the global default of one minute. The cost is doubled per evaluation; the benefit is fresher data for the alerts that consume it.
sum by (job). The level is job. The output series have one label: job. Lesson 02 covers why the level is in the metric name.
histogram_quantile() over sum by (job, le). The le label is mandatory for the inner aggregation; lesson 02-histogram-quantile covers the failure mode in depth.

The prometheus.yml reference:

rule_files:
  - /etc/prometheus/rules/*.yml

global:
  evaluation_interval: 1m

evaluation_interval is the default for any group that does not override interval. A group that sets interval: 30s runs twice as often; a group that sets interval: 5m runs five times less often. Use per-group intervals to match the SLO or alert sensitivity the rule feeds.

How to validate it

Three checks, each catching a different failure shape.

1. promtool validates the YAML and the PromQL.

promtool check rules /etc/prometheus/rules/api.recording.yml

Expected output:

SUCCESS: rules are valid

A non-zero exit means the file has either a YAML syntax error or a PromQL expression that does not parse. promtool reports the file and line.

2. The rule is loaded by Prometheus.

curl -s http://localhost:9090/api/v1/rules \
  | jq '.data.groups[] | select(.name=="api-recording") | .rules[] | {name, health, lastError, evaluationTime}'

Expected output (illustrative):

{
  "name": "job:http_requests_total:rate5m",
  "health": "ok",
  "lastError": "",
  "evaluationTime": 0.0123
}

health: ok and an empty lastError are the green light. health: err and a populated lastError mean the rule loaded but fails to evaluate.

3. The rule produced series.

curl -s 'http://localhost:9090/api/v1/query?query=job:http_requests_total:rate5m' \
  | jq '.data.result | length'

Expected: a positive integer (one per job label set). Zero means the rule is loaded and evaluating but producing nothing — usually a wrong source metric, a typo in expr:, or an aggregation that drops every series.

How it can fail

Six failure modes, in the order they appear in real incidents.

Missing source metric. A rule references http_requests_total but the exporter was removed or renamed. The rule produces no series for that evaluation. The dashboard panel reads “no data”. The alert reads “no data” and does not fire. The platform logs an evaluation failure; nobody watches the log.
Wrong aggregation level. A rule’s expression uses sum by (job, instance) (...) but the rule’s name says job:.... The output series carry an instance label that consumers do not expect. Dashboards that grouped by job now show one panel per instance — a cardinality explosion that looks like a metrics outage.
Recursive rule without backstop. Rule A depends on rule B; rule B depends on rule A. Prometheus does not detect cycles; one of the rules reads the previous tick’s value and the result lags by one interval. The lag is silent.
Label churn. The source metric carries a label whose cardinality is unbounded (e.g. email, request_id, trace_id). The rule passes the label through. The TSDB series count grows without bound. OOM in days.
Rule interval shorter than the rule can complete. A rule takes ninety seconds to evaluate. The group’s interval is thirty seconds. Prometheus logs rule_group_iterations_missed_total > 0. The rule never catches up; downstream panels read stale data.
Rule file fails to load. A YAML syntax error or invalid PromQL in one file prevents all rules in that file from loading. Other files continue. The team sees a startup message that looks fatal but is in fact scoped. Rules in the broken file never produce data; alerts they were meant to feed never fire.

How to troubleshoot it

When a dashboard reads “no data” or a recording rule’s output disagrees with what the same expression returns inline, the diagnosis order matters.

Is the rule loaded? curl -s http://localhost:9090/api/v1/rules | jq '.data.groups[].file' — the rule’s file should appear in the list.
Is the rule evaluating successfully? Same endpoint, drill into the rule: .data.groups[].rules[] | {name, health, lastError, evaluationTime}. health: ok and empty lastError are required.
Is the rule producing series? curl -s 'http://localhost:9090/api/v1/query?query=RECORD_NAME' — a positive result count. Zero series means the expression returns nothing against the live TSDB.
Is the source metric present? Evaluate the same expr: in the expression browser. If the raw expression returns data and the rule does not, the difference is in the record: name or the group’s interval (lesson 03 covers interval interactions).
Is the rule’s interval too short for its workload? Check prometheus_rule_group_iterations_missed_total for the group. A non-zero value means the group missed at least one tick.
Has cardinality exploded? Check prometheus_tsdb_head_series. A sudden rise after a rule change points to a label passed through by mistake.

What could go wrong?

The silent failure is the missing source metric. The rule produces no series; the dashboard reads “no data”; the alert does not fire because there is no value to compare. The team is told the system is healthy when it is not. The investigation starts with a wrong assumption. The fix is at the source: the rule must reference a metric that exists, or it must be removed, or it must be made defensive (an absent() overlay that produces a synthetic zero so the alert still fires).

The loud failure is a YAML syntax error in one rule file. The file fails to load. The error is logged at startup and on every SIGHUP. Other rule files load normally. The team sees an error that looks fatal but is in fact scoped. They assume it is cosmetic. The rules in the broken file never produce data; the alerts they were meant to feed never fire. Months later, a genuine incident shows up that the rules would have caught.

Security implications

Rule files are loaded from the local filesystem. There is no network authentication on the rule parser itself. Three exposures matter.

/api/v1/rules exposes rule content to anyone with HTTP access to the Prometheus HTTP server. Production deployments restrict the API surface with --web.listen-address, network ACLs, or a reverse proxy with authentication. The /api/v1/rules endpoint returns the rule expressions, including any hardcoded secrets or internal hostnames that the rules reference.
Rule output inherits the source metric’s labels. A rule with a typo that pulls a sensitive label (e.g. customer_email) into a new metric exposes the label on the /api/v1/query endpoint and to every dashboard. Treat rule output as carefully as the source metric.
Recording rules do not introduce a new write attack surface. They write to the local TSDB with the same permissions as a scrape. The risk is the same as for any other metric ingestion path.

The platform-security part of the course covers Prometheus HTTP auth, network isolation, and the labelling guide.

Performance implications

A recording rule’s cost is paid every evaluation interval, regardless of consumption.

CPU: proportional to the number of series the expression produces. A rule that reads rate() over ten thousand series costs more per evaluation than one that reads a hundred.
TSDB write: proportional to the number of output series. Each output series is one sample per evaluation.
TSDB storage: proportional to the number of output series times the retention. A rule with ten thousand output series evaluated every thirty seconds over thirty days of retention produces roughly 10,000 * 2 * 86,400 * 30 = 51.8 billion samples. The TSDB compresses them; wall-clock storage is in the tens of gigabytes.

The trade-off is constant evaluation cost against repeated query cost. The rule wins when the expression is computed more than a handful of times per minute. The rule loses when it is computed once per dashboard load — the rule adds overhead with no benefit.

A useful rule of thumb: a rule that takes more than ten percent of its group’s interval to evaluate is a candidate for splitting into a separate group with a longer interval (lesson 03 covers group mechanics).

Production guidance

Use the level:metric:operations naming convention so consumers can infer the aggregation level from the metric name. Lesson 02 covers the convention in depth.
Prefer sum by (...) to sum without (...). The by clause is explicit about what is kept. without hides the intent and breaks when the source metric grows new labels.
Document the rule’s inputs, outputs, and consumers in a header comment. A rule named job:api:requests:rate5m with the comment “consumed by alert HighRequestRate and dashboard latency-overview” is self-describing.
Validate with promtool check rules before every reload. CI runs promtool on every pull request that touches a rule file.
Add the rule’s group and interval to the team’s runbook. The interval is part of the contract between the rule and its consumers; changing it without telling consumers is a silent behaviour change.
Audit the rule set quarterly. Sort rules by prometheus_rule_evaluation_duration_seconds_sum divided by the count to get the mean per-rule duration. The top decile is where the cost lives.

Verification

You should now be able to answer:

What does a recording rule precompute, and where is the result stored?
What three patterns make a recording rule pay off?
What is the difference between a recording rule and an alerting rule, and where does each belong in an SLO pipeline?
Where do recording rules live in the Prometheus configuration, and how do you confirm a rule is loaded and producing data?