Naming Recording Rules

At 09:30 the on-call engineer opens the per-job dashboard. A panel that should be one series per job reads one series per instance — a thousand series where there should be twelve. The dashboard freezes for thirty seconds. The cause is a recording rule whose name says job:... but whose sum by retains instance. The naming convention would have caught this in code review. The team does not have a convention.

A recording rule’s name is a contract with its consumers. The name tells the consumer what level of aggregation to expect, what metric to look for, and what operations have been applied. A team that documents the convention and enforces it in code review gets the right division of labour for free. A team without one accumulates a graveyard of rules named metric_a_v2 that no one can decode at 03:00.

What it is

The convention used by every production Prometheus deployment that survives more than two teams is level:metric_name:operations. The Prometheus documentation calls this the level-metric-label form. The RunBook Academy version extends it to four components with a fixed rate-window suffix; the mnemonic is the 1-5-15 rule:

job : http_requests_total : rate5m : 1m
^       ^                   ^        ^
|       |                   |        |
1 level 5 metric name       15 ops   rate window

• 1 level — the highest label in the sum by clause, or the original metric when no aggregation is performed. Common values: job, instance, pod, node, cluster, tenant.
• 5 metric name — the underlying metric the rule reads. The number 5 is a mnemonic, not a length limit; the metric name may be longer. The intent is: one source metric per rule.
• 15 operations — the aggregation functions and their windows, in canonical order. Common suffixes: rate5m, irate1m, sum, count, max, p99, sum_rate5m, count_over_time_5m. The number 15 is again a mnemonic; the suffix is bounded by what consumers can read.
• rate window — the trailing :1m is optional but recommended. It is the time window used by the rule’s outer aggregation. Useful when the rule is one layer in a recursive stack; the consumer can read the freshness from the name.

The canonical example:

job:http_requests_total:rate5m

Means: the per-job rate, over a five-minute window, of the http_requests_total counter.

A rule whose expression keeps the instance label would be named:

instance:http_requests_total:rate5m

The naming and the expression agree. A consumer that reads job:http_requests_total:rate5m and gets an instance label back has hit the convention violation.

Why a sysadmin cares

The naming convention is the contract between the rule and its consumers. Three operational consequences follow.

1. Recursive rule behaviour is legible. A rule that consumes another rule’s output is named by adding a layer to the front of the metric name. The stack cluster:job:instance:http_requests_total:rate5m reads as: “the cluster-level sum of the per-instance rate of http_requests_total”. A reviewer can read the stack and verify the layers match the expressions.
2. Stack visibility in dashboards. Grafana’s panel query inspector lists the recorded metric names in use. A team that uses the convention reads the dashboard and knows what aggregation level each panel is using. A team without the convention reads a wall of metric names and has to click through to find out.
3. Code review catches aggregation drift. A PR that adds a rule named job:http_requests_total:rate5m with the expression sum by (job, instance) (...) is rejected on review because the sum by retains a label the name does not advertise. Without the convention, the PR passes and the dashboards break.

A team that adopts the convention spends an hour documenting it and saves a year of incident-time confusion.

How it works

The convention has four parts. Each part corresponds to a specific piece of the rule’s expression.

   job : http_requests_total : rate5m : 1m
   |        |                  |        |
   |        |                  |        +-- rate window (optional)
   |        |                  +-- operations (aggregation + window)
   |        +-- source metric name
   +-- aggregation level (the highest label in `sum by`)

The level maps to the sum by (...) clause (or, when there is no aggregation, to the original metric’s label set):

The metric name is the underlying metric, unmodified. Do not abbreviate, do not singularise. The operations are listed in canonical order: outer function first, then inner functions and their windows.

Examples

groups:
  - name: api-recording
    interval: 30s
    rules:
      # Per-job request rate over 5 minutes
      - record: job:http_requests_total:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))

      # Per-instance request rate over 5 minutes
      - record: instance:http_requests_total:rate5m
        expr: sum by (job, instance) (rate(http_requests_total[5m]))

      # Cluster-level aggregation of per-job rate
      - record: cluster:job:http_requests_total:rate5m:sum
        expr: sum by (cluster) (job:http_requests_total:rate5m)

      # Per-job p99 latency over 5 minutes
      - record: job:http_request_duration_seconds:p99
        expr: |
          histogram_quantile(
            0.99,
            sum by (job, le) (
              rate(http_request_duration_seconds_bucket[5m])
            )
          )

      # Per-route error ratio over 5 minutes
      - record: job:http_requests:errors:ratio_rate5m
        expr: |
          sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
          /
          sum by (job) (rate(http_requests_total[5m]))

Each rule’s name tells the consumer what aggregation level to expect, what metric to look for, and what operations have been applied. A consumer can read the stack cluster:job:http_requests_total:rate5m:sum and verify the expression does what the name says.

The level-explosion warning

The convention encourages recursion: a rule that aggregates the output of another rule by adding a layer to the front of the metric name. The recursion has a ceiling. Each layer that adds a new label dimension multiplies the output series count by the cardinality of that label. A rule stack that goes cluster:zone:node:pod:container:http_requests_total:rate5m produces series per (cluster, zone, node, pod, container) tuple. At twenty clusters, ten zones per cluster, fifty nodes per zone, twenty pods per node, and three containers per pod, the stack produces sixty thousand series for a single metric.

The level-explosion failure is silent. The stack is well-named; each layer is a valid rule; the expressions are correct. The TSDB absorbs the series. Weeks later, the team hits an OOM and the recording rules are the last place anyone looks because they were “obviously fine”.

How to configure it

The configuration is the rule file. The convention is enforced in review, not in the file.

# /etc/prometheus/rules/api.recording.yml
#
# Naming convention: level:metric:operations:rate_window
#
# Owners: team-payments (this file)
# Source metric docs: https://runbook.example.com/metrics/http
# Consumers:
#   - dashboard: latency-overview (panels 3, 5)
#   - alert: HighErrorRate
#   - SLO rule: job:api:errors:ratio_rate5m (lesson 06)
groups:
  - name: api-recording
    interval: 30s
    rules:
      - record: job:http_requests_total:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))

      - record: job:http_request_duration_seconds:p99
        expr: |
          histogram_quantile(
            0.99,
            sum by (job, le) (
              rate(http_request_duration_seconds_bucket[5m])
            )
          )

      - record: job:http_requests:errors:ratio_rate5m
        expr: |
          sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
          /
          sum by (job) (rate(http_requests_total[5m]))

A header comment at the top of the file documents:

• The convention used (level:metric:operations:rate_window).
• The team that owns the file.
• The source metric contract (which metrics this file expects to exist and how they should be labelled).
• The consumers (which dashboards, alerts, and SLO rules depend on this file).

Lesson 05 covers the file-organisation discipline in depth.

How to validate it

Three checks.

1. promtool validates the file.

promtool check rules /etc/prometheus/rules/api.recording.yml

A passing check confirms the YAML parses and every expression is valid PromQL. It does not check the naming convention; that is the linter’s job.

2. The convention is enforced by a linter in CI.

A simple check that catches the most common violation (name claims one level, expression retains another):

# Example shell-based linter. Replace with a proper parser in
# production. (CONFIGURATION)
for f in /etc/prometheus/rules/*.yml; do
  python3 - "$f" <<'PY'
import sys, yaml, re
path = sys.argv[1]
with open(path) as fh:
    doc = yaml.safe_load(fh)
for group in doc.get("groups", []):
    for rule in group.get("rules", []):
        if "record" not in rule:
            continue
        name = rule["record"]
        expr = rule["expr"]
        # Extract the level prefix (first colon-separated token).
        claimed = name.split(":")[0]
        # Extract the labels retained in `sum by (...)`.
        m = re.search(r"sum\s+by\s*\(([^)]*)\)", expr)
        kept = [l.strip() for l in (m.group(1) if m else "").split(",") if l.strip()]
        highest = kept[-1] if kept else "<untouched>"
        if claimed != highest and not (claimed == "instance" and "instance" not in kept):
            print(f"{path}: {name} claims level {claimed} but expression keeps {kept}")
PY
done

A clean run produces no output. Any line that prints is a convention violation.

3. The rule appears under the expected name.

curl -s http://localhost:9090/api/v1/rules \
  | jq -r '.data.groups[].rules[] | select(.type=="recording") | .name' \
  | grep '^job:http_requests_total:rate5m$'

A matching line confirms the rule loaded under the expected name. A missing line means the file failed validation or the rule crashed on load.

How it can fail

Six failure modes, in the order they appear in code review.

1. Name claims a coarser level than the expression keeps. A rule named job:http_requests_total:rate5m has the expression sum by (job, instance) (...). The dashboards that expect a per-job panel now see one series per (job, instance) tuple. Symptom: a dashboard that worked yesterday renders a thousand panels today.
2. Name claims a finer level than the expression has. A rule named instance:http_requests_total:rate5m has the expression sum by (job) (...). The recorded metric collapses all instances into the job dimension. Symptom: per-instance dashboards show one line per job — a flat line that hides the bad instance.
3. Level explosion through deep recursion. A team builds a stack region:zone:cluster:job:instance:http_requests_total:rate5m:sum:sum:sum:sum that produces one series per (region, zone, cluster, job, instance) tuple. Symptom: the TSDB series count climbs without bound; OOM in weeks.
4. Name collision between teams. Two teams add a rule named job:http_requests_total:rate5m in different files. Only one survives; Prometheus logs a duplicate-name error. The losing team’s dashboards and alerts read the winner’s expression — possibly with the wrong semantics.
5. Rate window missing. A rule named job:http_requests:rate has the expression rate(http_requests_total[1m]). The window is not in the name. Consumers cannot tell from the name what window the rate covers; changing the window in the expression is a silent behaviour change.
6. Operations out of canonical order. A rule named job:http_requests:5m_rate puts the window before the function. The name still parses; the convention does not match. Reviewers spot it; grep for it later is harder.

How to troubleshoot it

When a recorded metric does not match what the consumer expects, the diagnosis order matters.

1. Confirm the name and expression agree. Open the rule file. Read the name. Read the sum by clause. The highest label in sum by should equal the level prefix in the name.
2. Confirm the output labels match the name’s level.

   curl -s 'http://localhost:9090/api/v1/query?query=job_http_requests_total_rate5m' \
     | jq '.data.result[].metric | keys'

   Expected: ["job"]. Anything else is a violation.
3. Confirm the rule’s stack is bounded. List the recorded metrics with their prefix lengths:

   curl -s http://localhost:9090/api/v1/rules \
     | jq -r '.data.groups[].rules[] | select(.type=="recording") | .name' \
     | awk -F: '{print NF-1, $0}' | sort -rn | head

   A rule with more than five colon-separated layers is a review candidate.
4. Confirm the rate window is in the name. For rate-based rules, the window should appear in the operations suffix.
5. Confirm no two rules share a name. The linter in the validation step above catches this; a manual check is grep -rh '^ - record:' /etc/prometheus/rules/ | sort | uniq -d.
6. Confirm cardinality is bounded. Check prometheus_tsdb_head_series after the rule is loaded; a sudden jump is a level explosion.

Security implications

The naming convention does not introduce a new attack surface. The rule name is stored in the TSDB alongside the series and exposed through /api/v1/rules and /api/v1/series. Three considerations apply.

1. Names leak through /api/v1/rules. Anyone with HTTP access to the Prometheus API can read the rule names and expressions. A naming convention that uses internal codenames (internal_payment_v3) leaks more than a convention that uses the source metric (http_requests_total). Use the source metric name.
2. Names leak through /api/v1/label/{name}/values. A query for __name__ values returns every metric name in the TSDB, including recording rule outputs. The convention does not change this; it makes the leak more uniform.
3. Naming consistency aids review. A team that adopts the convention can review a PR for naming violations quickly. A team without one cannot, and security-relevant label changes slip through.

Performance implications

The naming convention is free at evaluation time; the cost is at TSDB-write time and storage time.

• TSDB write: one write per output series per evaluation. Naming does not change the count; it changes only the index key.
• TSDB storage: proportional to series count and retention. A level-exploded stack with sixty thousand series writes sixty thousand samples per evaluation.
• Query performance: a recorded metric whose name reflects its aggregation level is easier to grep for in dashboards and rules, which means review cycles are faster. Naming itself does not change query cost.

The level-explosion failure mode is the performance hazard. The convention does not prevent it; it makes the stack visible enough that a reviewer can catch it.

Production guidance

• Document the convention in the team’s instrumentation guide. The first paragraph should be the convention; the rest should be examples.
• Enforce it with a linter in CI. A shell script that compares the level prefix to the sum by clause is enough to catch the most common violation. Replace it with a proper PromQL parser if the rule set grows.
• Bound the level depth at review. A rule stack with more than four or five colon-separated layers is a candidate for splitting into a separate metric. The convention makes the depth visible.
• Use the source metric name unmodified. Do not abbreviate, do not singularise. The metric http_requests_total is a contract with the exporter; renaming it in the recorded metric breaks the contract.
• Include the rate window in the operations suffix. A name that includes the window makes it visible to reviewers and consumers.
• Document consumers in the file header. A rule whose consumers are listed is easier to deprecate when the rule is no longer needed.

Verification

You should now be able to answer:

• What are the four components of the level:metric:operations convention, and what does each component mean?
• How does the level prefix relate to the sum by clause in the expression?
• What is the level-explosion failure mode, and how does the convention make it visible?
• How does the naming convention help reviewers spot recursive dependencies between rules?

Quiz