Rule Files

A dashboard panel shows the 95th-percentile latency of an internal API. The panel takes 40 seconds to load because the query recomputes a histogram quantile over two million series, on every refresh, for every viewer. One engineer fixes it with a three-line recording rule; a second engineer adds an alert on the recorded series; a third deletes the expensive dashboard query entirely. That is the whole value proposition of rule files: compute once, store the result, let alerts and dashboards read cheap series instead of re-deriving them.

Rule files are also where pages are born. The same YAML that precomputes dashboards defines every alerting rule Prometheus evaluates. Getting the layout, the group semantics and the validation discipline right is the difference between a rules directory you can operate and one that pages you for its own mistakes.

What a rule file is

Prometheus loads rule files listed in rule_files, a list of globs:

rule_files:
  - rules/platform-*.yml
  - rules/app-*.yml
  - /etc/prometheus/rules.d/*.yml

Relative paths resolve against the directory containing prometheus.yml. Two properties to internalise:

1. The glob is expanded at config load. Dropping a brand-new file into rules/ does nothing until the next reload, even if it matches an existing glob. Rules are code: they ship on the reload path.
2. One bad file fails the whole load. Rule files are parsed as part of the configuration. A YAML error in any matched file rejects the entire reload — every other change in the batch waits behind the typo.

Each file holds a list of groups:

groups:
  - name: platform-node-availability     # unique within the file
    interval: 30s                        # default: global evaluation_interval
    limit: 1000                          # per-rule output cap; 0 = no limit
    rules:
      - record: job:up:avg5m
        expr: avg by (job) (up)
        labels:
          team: platform

      - alert: InstanceDown
        expr: up == 0
        for: 5m
        keep_firing_for: 10m
        labels:
          severity: critical
          team: platform
        annotations:
          summary: 'Target {{ $labels.instance }} is down'
          description: >-
            The {{ $labels.job }} target {{ $labels.instance }} has been
            unreachable for more than 5 minutes.
          runbook_url: 'https://runbooks.example.com/InstanceDown'

Recording rules carry record, expr and optional labels. Alerting rules carry alert, expr, optional for (how long the condition must hold before firing), optional keep_firing_for (flap damping on the way out), labels that become the alert’s identity, and annotations — templated free text, where $labels and $value plug evaluation results into the notification a human reads.

Organisation by team and domain

The layout that survives contact with a real organisation:

rules/
  platform-node.yml         # team: platform, domain: hosts
  platform-prometheus.yml   # self-monitoring
  platform-blackbox.yml     # synthetic probes
  app-checkout.yml          # team: checkout owns these pages
  app-payments.yml

File name is team-domain.yml; group names inside follow the same pairing (platform-node-availability, platform-node-capacity). One concern per group, so the group’s interval reflects the cost of its expressions: cheap health checks at 30s, heavy aggregation over millions of series in their own group at 5m. One expensive group never slows another — but the rules inside a group share its cadence, so grouping by cost is grouping by latency.

Conventions worth enforcing in review:

• Recording rule names follow level:metric:operation (job:up:avg5m, instance:node_cpu:rate5m). The name alone tells a reader what granularity they are looking at.
• Every alert has severity and an owning team label, and a runbook_url annotation. No runbook, no merge.
• Annotations use $labels and $value; keep template logic trivial. Clever templates break at 03:00.

Validating rules before they run

promtool check config already parses every file matched by rule_files, but the dedicated checker goes further and is the right CI gate:

promtool check rules /etc/prometheus/rules/*.yml
# Checking /etc/prometheus/rules/app-checkout.yml
#   SUCCESS: 6 rules found
#
# Checking /etc/prometheus/rules/platform-node.yml
#   SUCCESS: 4 rules found

# And the failure shape, exit code 1:
# Checking /etc/prometheus/rules/platform-node.yml
#   FAILED: parsing YAML file /etc/prometheus/rules/platform-node.yml:
#   yaml: line 14: did not find expected key

The checker parses expressions and validates annotation template syntax. Add --lint-fatal in CI so findings such as duplicate rule names fail the build rather than printing politely. For rules that page people, go one step further: promtool test rules runs unit tests against fixture series — the alert-testing lesson covers the format. Rules that wake humans earn the same test discipline as code that moves money.

On the running server, the rules API and the rule metrics are the live view:

curl -s http://localhost:9090/api/v1/rules | \
  jq -r '.data.groups[] | [.name, (.rules | length)] | @tsv'
# platform-node-availability   4
# app-checkout-latency         6

Each rule there carries a health field (ok, err). The metrics to watch long-term: prometheus_rule_evaluation_failures_total, prometheus_rule_group_duration_seconds against the group’s interval, and prometheus_rule_group_iterations_missed_total — the group falling behind its own cadence.

How rule files fail

1. YAML error in one file rejects the whole reload. Symptom: prometheus_config_last_reload_successful drops to 0, the log names the file and line, and every rules change in the same batch — including the urgent one — is stuck behind the typo.
2. Glob that never matches. A new file named node.yaml while the glob says *.yml. Symptom: total silence. No error anywhere; the alert simply does not exist. Found months later during an incident review.
3. Duplicate record names across groups. Two groups write the same series at the same evaluation timestamp. Symptom: duplicate sample errors, evaluation failures climbing, and dashboards that disagree with themselves depending on which group wrote last.
4. Expensive expression in a fast group. A million-series join at 15s. Symptom: prometheus_rule_group_duration_seconds exceeds the interval, iterations are missed, evaluation effectively stretches, and alerts depending on the group fire late.
5. Template typo in annotations. {{ $labels.instanc }} parses fine but fails to expand at evaluation time. Symptom: rule health flips to err, the alert never reaches Alertmanager, and the first anyone hears of it is the user report.
6. limit breached after cardinality growth. A recording rule producing 900 series under limit: 500. Symptom: the evaluation fails with an exceeded-limit error, no samples from that rule are written, and downstream alerts read staleness as “all quiet”.

Troubleshooting, in order

1. Is the group loaded? /api/v1/rules — absent group means glob or load failure; present group with health: err means evaluation failure.
2. Is the failure at load or at evaluation? Load failures show in the reload metric and logs; evaluation failures show in prometheus_rule_evaluation_failures_total and per-rule health.
3. What does the log say? Error evaluating rule lines carry the expression and the cause — template expansion, vector matching, exceeded limit.
4. Is the group keeping up? Compare group duration to interval; check missed iterations. A group behind schedule delays every alert it owns.
5. Is the output there? Query the recorded series directly. Present but stale points upstream (rule or ingestion); absent entirely points at the rule never producing.

Security implications

Rule files are code that runs inside your monitoring server. PromQL cannot exfiltrate data by itself, but an expensive expression is a denial-of-service with extra steps, and annotations copy label values into notifications — any secret that leaks into a label value rides straight into Slack, e-mail, or a paging vendor. Treat the rules directory as deployment-gated content: written via CI, reviewed like application code, 0640 on disk. The people allowed to merge alerting rules are, effectively, the people allowed to page anyone at any time.

Performance implications

Rule cost is expression cost divided by interval, summed over groups. The same engine answers dashboard queries, so a rule fleet that eats 40% of CPU is taxing every Grafana refresh too. Recording rules convert repeated query cost into storage cost — usually a superb trade; a five-minute aggregation stored once replaces thousands of identical dashboard evaluations. The failure side is cardinality: a recording rule with a careless by (...) clause mints series you store forever. Watch prometheus_rule_group_duration_seconds, cap blast radius with limit, and keep heavy groups on slow intervals.

Production guidance

• One file per team-domain pair; one concern per group; group interval matched to expression cost.
• Enforce the naming contract (level:metric:operation) and the alert contract (severity, team, runbook_url) in review, then in CI with --lint-fatal.
• Unit-test rules that page with promtool test rules.
• Set limit on groups whose expressions can grow with cardinality.
• Alert on evaluation failures and missed iterations. A monitoring system that cannot watch its own rules is half a monitoring system.

Verification

You should now be able to answer:

• When are rule_files globs expanded, and why does a new file need a reload even when it matches?
• What does one malformed file do to the rest of a reload batch?
• Which fields distinguish a recording rule from an alerting rule?
• What happens when a rule’s output exceeds its group’s limit?
• Which two metrics tell you a group is failing versus falling behind?

Quiz