Rule Organisation

At 02:14 a recording rule stops producing data. The on-call engineer opens /api/v1/rules, finds the rule by name, and sees the file: /etc/prometheus/rules/legacy_api.yml. The file has no header comment. The engineer searches Slack for the rule name, finds nothing. The rule’s original author left the company eight months ago. The rule is owned by nobody.

The team fixes the rule in forty minutes — but only because someone happened to remember the file path. The post-mortem names the failure as “ownership vacuum in legacy rule files” and recommends the file-per-team convention with mandatory owner metadata.

This is the operational shape of rule organisation. Rules are production code. They live in files, they have owners, they go through review. A team that treats rules as configuration debris discovers the ownership vacuum at 02:14 under an incident. A team that treats rules as code gets the right answer in forty-five seconds: open the file, read the header, page the owner.

What it is

Rule organisation is the discipline of structuring the directory of rule files so that ownership, dependency, and validation are visible at a glance. Three conventions apply.

1. File-per-team or file-per-domain. Each team owns its own rule file; the file is named after the team or the domain the rules belong to (api.recording.yml, payments.recording.yml, platform.recording.yml).
2. File header metadata. Every rule file starts with a comment block that records the owning team, the on-call rotation, the source-metric contract, and the consumers.
3. Git as source of truth. Rule files live in a git repo; changes go through pull requests; CI validates every PR with promtool check rules and the naming linter.

The combination is the difference between rules-as-config and rules-as-code. A team that applies all three conventions answers “who owns this rule?” in five seconds by reading the file header. A team that applies none answers the question in forty minutes by searching Slack.

Why a sysadmin cares

Three operational consequences follow.

1. Ownership is visible at the file level. The on-call engineer opens the file, reads the header, and knows who to page. Without the header, the engineer has to search the codebase, the wiki, and Slack. The mean time to find the owner drops from minutes to seconds.
2. PR review catches mistakes. A PR that adds a rule whose name does not match the expression is rejected on review. A PR that uses a non-existent source metric is rejected by promtool test rules. A PR that introduces a circular dependency is rejected by the linter. The review is the discipline; the CI is the enforcement.
3. The directory structure documents the dependency graph. A layered pipeline — curated metrics, SLI rules, SLO rules, alert rules — is a directory structure. Reading the directory tells the reader which rules depend on which.

A team that adopts the conventions spends an afternoon documenting them and a day moving files around. The operational payoff appears the first time a rule breaks under an incident.

How it works

The directory layout is the source of truth. A typical production layout:

/etc/prometheus/rules/
├── _headers.yml              # template + ownership comment
├── curated/
│   ├── api.raw.recording.yml      # raw metric -> 1m/5m rate
│   ├── payments.raw.recording.yml
│   └── platform.raw.recording.yml
├── sli/
│   ├── api.sli.recording.yml      # SLI rules per service
│   ├── payments.sli.recording.yml
│   └── platform.sli.recording.yml
├── slo/
│   ├── api.slo.recording.yml      # SLO + burn-rate rules
│   ├── payments.slo.recording.yml
│   └── platform.slo.recording.yml
└── alerts/
    ├── api.alerts.yml            # alerting rules
    ├── payments.alerts.yml
    └── platform.alerts.yml

Three patterns to notice.

• Layered pipeline. The curated/ directory holds rules that produce the per-job, per-instance, per-route aggregations from raw metrics. The sli/ directory holds rules that compute the SLI from the curated rules. The slo/ directory holds SLO and burn-rate rules that read the SLI. The alerts/ directory holds alerting rules that read the SLO. Each layer depends on the layer below it; no layer depends on a layer above.
• File-per-team within each layer. Each service or domain has its own file at each layer. The api.sli.recording.yml file holds the SLI rules for the API; the payments.sli.recording.yml file holds the SLI rules for payments. The owner of the API owns the file.
• Git is the source of truth. The directory is a checkout of a git repository. Changes go through PRs. CI runs promtool check rules on every PR and rejects any that fail. A merge to main deploys the rules to the Prometheus instances.

The file header convention

Every rule file starts with a header comment that records the metadata the on-call engineer needs during an incident.

# File:        api.recording.yml
# Owner:       team-payments
# On-call:     payments-oncall (PagerDuty schedule: payments)
# Source:      https://github.com/example/prometheus-rules
# Consumers:   dashboard latency-overview, alert HighErrorRate,
#              SLO rule job:api:errors:ratio_rate5m
# Last review: 2026-07-15 by alice (PR #1234)
#
# Naming convention: level:metric:operations:rate_window
# Validate locally:  promtool check rules api.recording.yml
#
# Source metric contract:
#   - http_requests_total{job, instance, status} is emitted by
#     the API at a 15s scrape interval.
#   - http_request_duration_seconds_bucket{job, instance, route, le}
#     is emitted by the API at a 15s scrape interval with the
#     standard Prometheus default buckets plus 0.15 and 0.2.
groups:
  - name: api-recording
    interval: 30s
    rules:
      - record: job:http_requests_total:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))

The header captures six pieces of information that are not in the rule expressions:

• Owner — the team that owns the file.
• On-call — the rotation to page during an incident.
• Source — the git repo where the file lives.
• Consumers — the dashboards, alerts, and SLO rules that read this file.
• Last review — the date of the last review PR.
• Source metric contract — the metrics the file expects to exist, and how they should be labelled.

The header is the difference between “we have a rule file” and “we have a rule file we can operate”.

Git as the source of truth

Rule files are stored in a git repository. The Prometheus configuration is a checkout of that repository. Changes go through pull requests. CI runs:

1. promtool check rules on every changed file.
2. The naming linter (lesson 02) on every changed rule.
3. promtool test rules against the unit-test fixture.
4. The dependency-direction linter (curated → SLI → SLO → alerts) on every cross-file reference.

A merge to main deploys the rules to the Prometheus instances via the team’s standard deployment pipeline. The deployment runs promtool check rules once more and then triggers a Prometheus SIGHUP (or --web.enable-lifecycle-driven reload) to pick up the changes.

How to configure it

The configuration is the directory layout and the prometheus.yml reference.

# /etc/prometheus/prometheus.yml
global:
  evaluation_interval: 1m

rule_files:
  - /etc/prometheus/rules/curated/*.yml
  - /etc/prometheus/rules/sli/*.yml
  - /etc/prometheus/rules/slo/*.yml
  - /etc/prometheus/rules/alerts/*.yml

The four globs load the four layers in dependency order. curated/ first, sli/ next, slo/ third, alerts/ last. A rule in sli/ can reference a rule in curated/; a rule in alerts/ can reference a rule in slo/ or sli/.

Example curated/api.raw.recording.yml:

# File:        curated/api.raw.recording.yml
# Owner:       team-payments
# On-call:     payments-oncall (PagerDuty schedule: payments)
# Source:      https://github.com/example/prometheus-rules
# Consumers:   sli/api.sli.recording.yml, alerts/api.alerts.yml
# Last review: 2026-07-15 by alice (PR #1234)
#
# Source metric contract:
#   - http_requests_total{job, instance, status} scraped at 15s
#   - http_request_duration_seconds_bucket{job, instance, route, le}
#     scraped at 15s
groups:
  - name: api-curated
    interval: 1m
    rules:
      - record: job:http_requests_total:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))

      - record: job:http_request_duration_seconds:p99
        expr: |
          histogram_quantile(
            0.99,
            sum by (job, le) (
              rate(http_request_duration_seconds_bucket[5m])
            )
          )

Example sli/api.sli.recording.yml:

# File:        sli/api.sli.recording.yml
# Owner:       team-payments
# On-call:     payments-oncall (PagerDuty schedule: payments)
# Source:      https://github.com/example/prometheus-rules
# Consumers:   slo/api.slo.recording.yml, alerts/api.alerts.yml
# Last review: 2026-07-15 by alice (PR #1234)
#
# Depends on:  curated/api.raw.recording.yml
groups:
  - name: api-sli
    interval: 1m
    rules:
      - record: job:api:requests:error_ratio:rate5m
        expr: |
          sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
          /
          sum by (job) (rate(http_requests_total[5m]))

Example slo/api.slo.recording.yml:

# File:        slo/api.slo.recording.yml
# Owner:       team-payments
# On-call:     payments-oncall (PagerDuty schedule: payments)
# Source:      https://github.com/example/prometheus-rules
# Consumers:   alerts/api.alerts.yml
# Last review: 2026-07-15 by alice (PR #1234)
#
# Depends on:  sli/api.sli.recording.yml
groups:
  - name: api-slo
    interval: 1m
    rules:
      # Error budget remaining (lesson 06).
      - record: job:api:error_budget:remaining:ratio
        expr: 1 - job:api:requests:error_ratio:rate5m

      # 1-hour burn rate (lesson 06).
      - record: job:api:slo:burn_rate:5m
        expr: job:api:requests:error_ratio:rate5m / 0.001

The three files together document the layered pipeline for the API: curated metrics feed SLI rules, SLI rules feed SLO rules, SLO rules feed alerts. The directory structure documents the dependency graph; the file headers document the ownership.

How to validate it

Four checks.

1. promtool validates every file.

find /etc/prometheus/rules -name '*.yml' \
  | xargs -I{} promtool check rules {}

A passing run produces no output. A failing run produces a per-file error. CI runs this on every PR.

2. Confirm the file’s group loaded with its source file.

curl -s http://localhost:9090/api/v1/rules \
  | jq '.data.groups[] | {name, file}'

The file field should match the path on disk. A mismatch suggests the file was renamed on disk but the rule_files glob in prometheus.yml was not updated.

3. Confirm the file header contains the owner metadata.

for f in /etc/prometheus/rules/**/*.yml; do
  if ! grep -q '^# Owner:' "$f"; then
    echo "$f: missing # Owner:"
  fi
  if ! grep -q '^# On-call:' "$f"; then
    echo "$f: missing # On-call:"
  fi
done

A clean run produces no output. The header is the ownership contract; missing fields are review failures.

4. Confirm the last reload succeeded.

prometheus_config_last_reload_successful == 1

The metric is 1 if the last reload succeeded, 0 otherwise. The on-call dashboard should alert on a value of 0.

How it can fail

Six failure modes.

1. Ownership vacuum. A rule file has no # Owner: header. The on-call engineer cannot find the owner during an incident. Symptom: the engineer pages the platform team instead of the right team; the platform team does not know the rule; the incident lasts longer than it should.
2. Rule shadowed across files. Two files declare the same rule name. One loads; the other fails with a duplicate-name error. Symptom: a PR that adds a rule to one file shadows the same-named rule in another file. Consumers read the winning file’s expression, which may have different semantics.
3. PR merged without promtool validation. A PR bypasses CI and lands in main. The file fails to load on the next reload. Symptom: the reload fails; the rule is missing; prometheus_config_last_reload_successful reads 0.
4. Circular dependency between files. curated/ reads from sli/; sli/ reads from curated/. The rules load but the values lag by one tick. Symptom: the SLO panel reads values from the previous tick; the alert is late.
5. File renamed in git but old version still on disk. A PR renames api.recording.yml to legacy.api.recording.yml. The old file remains on disk; the rule_files glob picks up both; one shadows the other. Symptom: duplicate-name error; one team’s rules win.
6. Layered pipeline inverted. curated/ references a rule in sli/. The dependency direction is wrong; the rule reads the previous tick of the SLI, which itself depends on the curated rule. Symptom: the curated rule’s output is always one tick behind the SLI rule.

How to troubleshoot it

When a rule is missing or a reload fails, the diagnosis order matters.

1. Did the reload succeed? Check prometheus_config_last_reload_successful. A value of 0 means the latest SIGHUP failed; the rule is not loaded.
2. Which file is the rule in? Query /api/v1/rules; each group exposes its source file. The answer is the on-call engineer’s first step.
3. Is the file’s owner metadata present? Open the file; confirm the # Owner: and # On-call: headers are populated. If not, the file is in an ownership vacuum.
4. Does the file pass promtool? Run promtool check rules <file> locally. A failure points to a YAML or PromQL error.
5. Are dependencies in the right direction? Confirm the file is in the right layer of the pipeline. A curated file that reads an SLI rule is inverted.
6. Is the rule shadowed by a duplicate name? Grep all files for the rule name:

   grep -rh '^      - record: ' /etc/prometheus/rules/ \
     | sort | uniq -d

   Any duplicate is a shadow.

Security implications

Rule organisation does not introduce a new attack surface beyond what lesson 01 covers. Three considerations apply specifically to the directory and CI.

1. The rule_files glob is read by the Prometheus process. A path that points outside the intended directory (e.g. /etc/prometheus/rules/*.yml plus a symlink to /etc/shadow) reads files the operator did not intend. The glob should be constrained to the rule directory.
2. CI runs promtool on every PR. A PR that includes a malicious rule (e.g. a rule that pulls a sensitive label into a new metric) is caught by code review, not by promtool. Code review is the security boundary; CI is the hygiene check.
3. The file header leaks ownership metadata. A header that says # On-call: payments-oncall exposes the PagerDuty schedule. Treat the file headers as part of the team’s access model; do not put credentials or secrets in them.

Performance implications

Rule organisation is free at evaluation time; the cost is at review and reload time.

• Review cost: proportional to the number of files changed in a PR. A PR that touches one file is faster to review than a PR that touches ten. File-per-team keeps PRs scoped to one team.
• Reload cost: proportional to the number of files reloaded. A single SIGHUP reloads every file; the cost is the sum of the parse costs. With dozens of files, the reload is still fast (milliseconds), but the discipline scales.
• TSDB cost: not affected by file organisation.

Production guidance

• One file per team per layer. The API team owns curated/api.raw.recording.yml, sli/api.sli.recording.yml, slo/api.slo.recording.yml, alerts/api.alerts.yml. PRs are scoped to one team per layer.
• Document ownership in the file header. # Owner:, # On-call:, # Source:, # Consumers:, # Last review: are mandatory fields. CI rejects files without them.
• Use git as the source of truth. Every change is a PR; every PR runs promtool check rules, the naming linter, and the dependency-direction linter.
• Alert on failed reloads. A panel on prometheus_config_last_reload_successful == 0 pages the on-call engineer immediately.
• Review the layered pipeline quarterly. A new rule that breaks the layer convention (e.g. an sli/ rule that reads from slo/) is a code-review failure; a quarterly audit catches the ones that slipped through.
• Deprecate rules explicitly. A rule that is no longer needed should be removed in a PR that also updates the consumers. A graveyard of unused rules is a TSDB and review burden.

Verification

You should now be able to answer:

• What are the three conventions of rule organisation, and how do they appear in a directory layout?
• What is the layered pipeline (curated → SLI → SLO → alerts), and how does the directory structure document it?
• Which Prometheus metric tells you whether the last reload succeeded, and what does a value of 0 mean?
• What should the file header contain, and why is the ownership metadata mandatory?

Quiz