Rules and the Alerting Pipeline

The page arrives at 03:12: “disk will fill within four hours on db-02”. Between “a metric crossed a line” and “a phone rang” there is an entire pipeline: rule files, an evaluation loop, a small state machine, and a notification queue into Alertmanager. Each stage fails differently, and “no alerts” is ambiguous until you know which stage is silent. This lesson is the map.

How rules are loaded

prometheus.yml points at rule files with rule_files (globs are fine). Each file contains groups, and each group is evaluated on its own clock:

groups:
  - name: node-health
    interval: 30s            # default: global evaluation_interval (1m)
    rules:
      - record: job:node_cpu_utilisation:ratio_rate5m
        expr: 1 - avg by (job, instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]))

      - alert: NodeDown
        expr: up{job="node"} == 0
        for: 5m
        keep_firing_for: 10m
        labels:
          severity: critical
        annotations:
          summary: 'Node {{ $labels.instance }} unreachable for 5m'

Groups run concurrently; the rules inside a group run sequentially, in file order, all stamped with the same evaluation timestamp. That ordering is what lets a recording rule feed a later rule inside one group cleanly.

Recording rules vs alerting rules

A recording rule (record:) precomputes a PromQL expression and stores the result as a brand-new series, written at evaluation time. It exists for speed and sanity: the expensive fleet-wide join runs once per interval, not once per dashboard panel per viewer per refresh.

An alerting rule (alert:) evaluates an expression and feeds the result into the alert state machine. Labels become the alert’s identity (severity is the routing key by convention); annotations become the human-readable payload.

The alert state machine

            expression true              for satisfied
 inactive  ---------------->  pending  ---------------->  firing
    ^                             |                          |
    |      expression false       |    expression false      |  expression false
    +-----------------------------+                          |  (keep_firing_for
    |        (timer resets)       +--------------------------+   delays resolution)
    +--------------------------------------------------------+

• pending: the expression is true, but for has not yet been satisfied. Any evaluation where the expression is false resets the timer to zero. for: 0 fires on the first true evaluation.
• firing: the expression has held continuously for for. Only firing alerts are sent to Alertmanager.
• keep_firing_for (default 0): after the condition clears, the alert stays firing for this long before resolving. It exists to dampen flapping page-worthy alerts; use it deliberately.

Two synthetic series track live alerts: ALERTS\{alertname, alertstate="pending"|"firing"\} is 1 while an alert is active, and ALERTS_FOR_STATE carries the Unix timestamp at which the alert went active. The latter is how a restarted Prometheus restores in-progress for timers — as long as the series is still inside the five-minute lookback window.

The notify pipeline

Firing alerts are POSTed to every configured Alertmanager (/api/v2/alerts), continuously, for as long as they fire. Two config surfaces shape what leaves the building:

• alert_relabel_configs rewrites or drops alerts before sending — the standard way to keep severity: info off the pager entirely.
• external_labels are attached to every alert. They are also how Alertmanager deduplicates an HA pair: two identical Prometheus servers send identical alerts, and identical external labels make them collapse into one notification.

Prometheus is deliberately stupid about routing. Grouping, inhibition, silences, receiver selection and notification cadence (group_wait, group_interval, repeat_interval) all live in Alertmanager. If the wrong person got paged, the bug is almost never in the rule file.

Evaluation limits

Every rule evaluation is a PromQL query and obeys the server-wide limits: --query.timeout (default 2m) and --query.max-samples (default 50 million). A rule that hits them fails that evaluation: prometheus_rule_evaluation_failures_total increments and the alert state stays whatever it was.

The subtler failure: if a whole group cannot finish within its interval, iterations are skipped. prometheus_rule_group_iterations_missed_total climbs and prometheus_rule_group_last_duration_seconds exceeds the interval. Your one-minute alert quietly became a two-minute alert, and nothing tells you unless you look.

How to configure it

Wiring in prometheus.yml:

rule_files:
  - '/etc/prometheus/rules/*.yml'

alerting:
  alert_relabel_configs:
    # never send info-severity alerts to the pager stack at all
    - source_labels: [severity]
      regex: 'info'
      action: drop
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager-01:9093', 'alertmanager-02:9093']
      timeout: 10s               # give up on a slow Alertmanager

How to validate it

# Syntax and semantics of a rule file before it ever reaches prod
promtool check rules /etc/prometheus/rules/node.yml
# Checking /etc/prometheus/rules/node.yml
#   SUCCESS: 12 rules found

# Unit-test rules against synthetic series (CI-friendly)
promtool test rules /etc/prometheus/rules/tests/node.test.yml

# What is loaded and healthy, right now
curl -s localhost:9090/api/v1/rules \
  | jq '.data.groups[] | {name, interval,
        rules: [.rules[] | {name, health, lastError}]}'

# What is pending or firing, right now
curl -s localhost:9090/api/v1/alerts \
  | jq '.data.alerts[] | {alert: .labels.alertname, state: .state}'

And the PromQL view of pipeline health:

ALERTS{alertstate="firing"}                            # live alerts
prometheus_rule_evaluation_failures_total              # should be flat
prometheus_rule_group_last_duration_seconds            # vs the group interval
prometheus_rule_group_iterations_missed_total          # should be flat
prometheus_notifications_errors_total                  # Alertmanager sends failing
prometheus_notifications_dropped_total                 # queue full; alerts lost

How it can fail

1. A broken rule file at reload. The old rules keep running, prometheus_config_last_reload_successful flips to 0, and the new alert simply never appears in /api/v1/rules. Silent unless you alert on the reload metric.
2. An expression too expensive for its interval. Symptom: last_duration above the group interval, missed iterations climbing, alerts firing minutes late.
3. for misjudged in either direction. for: 0 on a noisy metric flaps and pages twice a night; for: 15m on a disk-filling-fast alert pages after the disk is full. for is the trade-off knob, not a formality.
4. alert_relabel_configs drops or rewrites severity. Symptom: every alert lands on the default route — or nowhere, if the default route is a blackhole.
5. Alertmanager unreachable and the queue fills. Symptom: alerts firing in /api/v1/alerts, nothing paging, prometheus_notifications_dropped_total growing.
6. HA pair with mismatched external_labels. Alertmanager cannot deduplicate what does not look identical; every firing alert pages twice.

How to troubleshoot it

1. Is the rule loaded and healthy? /api/v1/rules — check health and lastError before anything else.
2. Is it evaluating on time? Group last_duration against the interval, plus iterations_missed.
3. What state is the alert in? /api/v1/alerts and the ALERTS series. Pending forever usually means the condition is flapping across the threshold and resetting for.
4. Did the notification leave Prometheus? The notifications metrics above; then the Alertmanager side — its API, its logs, amtool config routes show for where the alert would land.
5. Only then start doubting the expression itself.

Security implications

• Write access to rule files is effectively production access: rules decide who gets woken up and when. Ownership and review should match that.
• Annotations template label values into notification text. If a secret ever leaks into a label (lesson 02), it lands in Slack or PagerDuty verbatim.
• Alertmanager targets take credentials in prometheus.yml; the Alertmanager API itself lets anyone who can reach it create silences. Restrict both.

Performance implications

Rule cost is rule count × query cost ÷ interval, paid on the same query engine as dashboards. Recording rules are the lever: they convert an expensive per-viewer query into a cheap stored series, at the price of ingestion and storage. Keep group intervals at or above the group’s slowest evaluation with headroom; a group that cannot keep up does not degrade gracefully — it skips.

Verification

You should now be able to answer:

• What does a recording rule produce that an alerting rule does not, and when would you choose each?
• What exactly does for measure, and what resets it?
• Where do grouping, inhibition and silences live — and what must Prometheus attach for an HA pair to deduplicate?
• Which three metrics tell you rule evaluation itself is unhealthy?
• Why does “no alerts in /api/v1/alerts” not prove the pipeline is working?

Quiz