Rule Failure Detection

A recording rule that has been quietly failing evaluation for six weeks. The recording rule file shipped with a typo three deployments ago. The error counter ticks up every evaluation interval. Nobody notices because no alert watches the rule failure counter itself. The dashboards that depend on the recording rule show stale data; the alerts that depend on the recording rule do not fire when they should. The platform “works”. The platform is wrong.

Rule failure detection is the discipline of watching the rule evaluator the same way the previous lessons watched the scraper and the process. Prometheus exposes its own evaluation metrics, and the right self-monitoring setup reads them.

What it is

Prometheus evaluates rules on a schedule. Each rule_files entry is parsed into rule groups; each rule group is evaluated on its interval (default evaluation_interval, which is 1 minute by default). The evaluation produces either a sample written to the TSDB (recording rule) or an alert sent to Alertmanager (alerting rule).

Prometheus exposes the following rule-evaluation metrics:

• prometheus_rule_evaluation_failures_total{rule_group} — counter of evaluation failures per rule group. Increments every time the group fails to evaluate.
• prometheus_rule_evaluation_duration_seconds{rule_group} — histogram of evaluation wall time per group.
• prometheus_rule_group_last_evaluation_timestamp_seconds \{rule_group, file\} — gauge. Wall-clock timestamp of the last successful evaluation.
• prometheus_rule_group_last_duration_seconds \{rule_group, file\} — gauge. Wall-clock duration of the last evaluation.
• prometheus_rule_group_rules{rule_group, file} — gauge. Number of rules in the group.
• prometheus_rule_group_iterations_total{rule_group} — counter of completed iterations.
• prometheus_rule_group_iterations_missed_total \{rule_group\} — counter of iterations skipped because the previous one had not finished.
• prometheus_notifications_dropped_total — counter of alerts dropped on the path to Alertmanager.
• prometheus_notifications_queue_length — gauge of pending notifications.

The right approach is to alert on three of these:

1. rate(prometheus_rule_evaluation_failures_total[5m]) > 0 — rule is failing.
2. prometheus_rule_group_iterations_missed_total increasing — group is too slow for its interval.
3. time() - prometheus_rule_group_last_evaluation_timestamp _seconds > interval * 2 — group has stopped evaluating.

Why a sysadmin cares

Rule evaluation failures are the second most expensive silent failure in a Prometheus stack, after scrape failures. A recording rule that has not evaluated for an hour means every downstream dashboard is querying raw data instead of the precomputed series, and every downstream alert has stopped firing. The on-call engineer finds out only when an investigation depends on the missing data.

A second reason: rule evaluation is the most common source of CPU pressure on a Prometheus host. A single recording rule that runs an expensive aggregation over a 30-day window can saturate the engine. The right self-monitoring catches the drift before the engine does.

How it works

evaluation_interval (default 1m)
          |
          v
+----------------------+
| rule group scheduler |
+----------------------+
   |       |       |
   v       v       v
+---+   +---+   +---+
| R |   | R |   | R |       R = rule
+---+   +---+   +---+
   |       |       |
   +-------+-------+
           |
           v
   +-----------------+
   | query engine    |
   +-----------------+
           |
           v
   +-----------------+
   | TSDB write      |  (recording rule)
   +-----------------+
           |
           v
   +-----------------+
   | notify queue    |  (alerting rule)
   +-----------------+
           |
           v
     Alertmanager

Each group is evaluated independently. The metrics above are emitted per group and per file. The most important diagnostic is prometheus_rule_group_last_evaluation_timestamp _seconds: if it stops advancing, the group has stopped running.

When a group’s evaluation exceeds the group interval, the next iteration is missed, not queued. The prometheus_rule_group_iterations_missed_total counter increments. The rule’s last value persists at the TSDB until the next successful evaluation. Alerting rules that depend on the rule continue to fire on stale data.

Under the hood

How to configure it

Rule-failure alerts

# /etc/observer/rules/rule-failures.yml
groups:
  - name: rule-evaluation-health
    rules:
      # 1. Any rule group is failing evaluation.
      #    Pages if a rule expression is broken or timing out.
      - alert: RuleEvaluationFailing
        expr: |
          rate(prometheus_rule_evaluation_failures_total[5m]) > 0
        for: 5m
        labels:
          severity: critical
          team: platform
        annotations:
          summary: |
            Rule group {{ $labels.rule_group }} is failing
            evaluation. Inspect Prometheus logs.

      # 2. Group has missed iterations.
      #    Catches a group that is too slow for its interval.
      - alert: RuleGroupMissingIterations
        expr: |
          increase(prometheus_rule_group_iterations_missed_total[1h]) > 0
        for: 5m
        labels:
          severity: warning
          team: platform

      # 3. Group has not evaluated within 2x its expected interval.
      #    Catches a stuck group that is not failing but has
      #    stopped running.
      - alert: RuleGroupStale
        expr: |
          (time() - prometheus_rule_group_last_evaluation_timestamp_seconds)
            > (2 * 60)
            # 2x evaluation_interval; tune to your config
        for: 5m
        labels:
          severity: warning
          team: platform

      # 4. Last evaluation duration is close to the interval.
      #    Catches a group that will miss iterations next time.
      - alert: RuleGroupAlmostMissingIteration
        expr: |
          prometheus_rule_group_last_duration_seconds
            > (0.8 * 60)
            # 0.8x evaluation_interval; tune to your config
        for: 5m
        labels:
          severity: warning
          team: platform

      # 5. Notification queue is growing.
      #    Catches the case where Alertmanager is unreachable.
      - alert: PrometheusNotificationQueueGrowing
        expr: |
          prometheus_notifications_queue_length > 1000
        for: 5m
        labels:
          severity: warning
          team: platform

      # 6. Alerts dropped on the path to Alertmanager.
      - alert: PrometheusNotificationsDropped
        expr: |
          rate(prometheus_notifications_dropped_total[5m]) > 0
        for: 5m
        labels:
          severity: critical
          team: platform

Make rule groups small

A group that contains one expensive rule blocks every other rule in the group on the same interval. Splitting rules into smaller groups is the cheapest performance fix:

# /etc/prometheus/rules/app.yml
groups:
  # Group A: cheap rules that must run every 30s.
  - name: app-availability
    interval: 30s
    rules:
      - record: app:http_error_ratio:5m
        expr: |
          sum by (service) (rate(http_requests_total{status=~"5.."}[5m]))
            / sum by (service) (rate(http_requests_total[5m]))

  # Group B: expensive aggregation that runs every 5 minutes.
  - name: app-aggregates
    interval: 5m
    rules:
      - record: app:requests_per_service:24h
        expr: |
          sum by (service) (increase(http_requests_total[24h]))

The two groups run on independent schedules. A slow group B does not delay group A.

How to validate it

Confirm rules are evaluating on schedule:

# READ-ONLY - last evaluation timestamp per group.
curl -s http://prom-primary.internal:9090/api/v1/rules \
  | jq '.data.groups[] | {name: .name, lastEval: .file, interval: .interval, lastEvalAgo: (now - .rules[0].lastEvaluation)}'

A healthy output shows lastEvalAgo less than twice the group’s interval for every group. A group with a stale lastEvalAgo has stopped evaluating.

Confirm rule evaluation failures are visible:

# READ-ONLY
curl -s 'http://prom-primary.internal:9090/api/v1/query?query=prometheus_rule_evaluation_failures_total' \
  | jq '.data.result[] | {group: .metric.rule_group, total: .value[1]}'

A healthy output shows the counter at the value it had at startup (zero for newly shipped groups, the historical value for groups with prior failures). A counter that is climbing in real time is the rule-failure signal.

Confirm the rule evaluator can keep up:

# READ-ONLY - iterations vs missed iterations.
curl -s 'http://prom-primary.internal:9090/api/v1/query?query=prometheus_rule_group_iterations_missed_total' \
  | jq '.data.result[] | {group: .metric.rule_group, missed: .value[1]}'

Any non-zero value means a group has missed an iteration. Sustained missed iterations mean the group is too slow for its interval; the rule on the next lesson covers the fix.

Inject a failure to confirm the alerts fire:

# CONFIGURATION - add a rule with a deliberate syntax error.
sudo tee /etc/prometheus/rules/canary-broken.yml >/dev/null <<'YAML'
groups:
  - name: canary-broken
    rules:
      - alert: CanaryBroken
        expr: this_is_not_valid_promql((
        for: 1m
YAML

sudo systemctl reload prometheus

# After 5 minutes, expect RuleEvaluationFailing to fire.
curl -s 'http://observer.internal:9090/api/v1/alerts' \
  | jq '.data.alerts[] | select(.labels.alertname=="RuleEvaluationFailing")'

# Undo.
sudo rm /etc/prometheus/rules/canary-broken.yml
sudo systemctl reload prometheus

A working setup shows a firing alert within five minutes. Restoring the config removes the alert within one interval.

How it can fail

1. Syntax error in the rule file

The rule file does not parse. Symptom: Prometheus logs an error at startup and on every reload; the rule does not load; no metric is emitted for that group. Action: promtool check rules /etc/prometheus/rules/...yml validates syntax before reload. Add this to the CI pipeline.

2. Query timeout in a recording rule

The recording rule’s expression is valid but takes longer than --query.timeout. Symptom: prometheus_rule_evaluation _failures_total increments for the group; the recording rule output is stale. Action: reduce the time window in the expression, pre-aggregate the source metric, or move the expensive aggregation to a longer interval group.

3. Cardinality blow-up in the rule output

The rule expression produces millions of series. Symptom: TSDB head series count jumps when the rule evaluates; OOM risk follows. Action: inspect the labels of the rule output via group() ... by (...) clauses; drop the offending high-cardinality label.

4. Missing source metric

The rule references a metric that the upstream exporter has removed or renamed. Symptom: rule evaluates successfully but produces empty output; downstream alerts never fire. Action: use absent(up{job="expected_source"}) to alert when the source itself disappears.

5. Group evaluation order dependency

A rule in group B depends on a recording rule in group A. Group A evaluates on its own interval; group B reads stale data on the first interval. Symptom: rule B’s output oscillates between correct and one-interval-behind. Action: move the dependency into the same group, or accept the lag and document it.

6. Notification queue full

The rule evaluator fires alerts faster than Alertmanager can absorb them. Symptom: prometheus_notifications_queue_length climbs; prometheus_notifications_dropped_total increments. Action: investigate Alertmanager health (next lesson); short-term, increase --alertmanager.notification-queue-capacity.

How to troubleshoot it

Security implications

Rule files are typically stored in a git repo or a config management system. A rule that has access to scrape credentials or that writes to an external URL is a small attack surface. The rule evaluator itself does not grant new permissions; the existing scrape credentials are reused.

The --query.timeout is a resource limit, not a security limit. A rule that exceeds the timeout can be used as a denial-of-service against the engine by an operator who controls the rule files. Treat rule-file write access as production write access.

Performance implications

Rule evaluation is the dominant CPU cost on a busy Prometheus host. The right sizing pattern is:

• Keep groups small. One expensive rule should not delay cheap rules.
• Match the group interval to the rule cost. A 24-hour aggregation runs every 5 minutes, not every 15 seconds.
• Pre-aggregate with recording rules so dashboards and alerting rules query precomputed series, not raw data.
• Alert on prometheus_rule_group_last_duration_seconds crossing 80 percent of the interval. That is the early warning before the group starts missing iterations.

Production guidance

• Run promtool check rules in CI on every change to a rule file. The 30 seconds it takes prevents hours of silent failures.
• Split rule files into groups by cost, not by topic. The cheap rules stay together; the expensive ones get their own group with their own interval.
• Alert on prometheus_rule_evaluation_failures_total and prometheus_rule_group_iterations_missed_total together. The first catches a broken rule; the second catches a slow rule.
• Use group_left() or group_right() in alert rules that depend on recording rules, to surface the dependency in the alert annotation.
• Re-test the alerts quarterly. Inject a syntax error in a canary rule file and confirm the alert fires.

Verification

You should now be able to answer:

• What does prometheus_rule_evaluation_failures_total measure, and what is its correct alerting threshold?
• What is the difference between last_evaluation_timestamp _seconds and last_duration_seconds?
• What does prometheus_rule_group_iterations_missed_total indicate, and what is the immediate operational consequence?
• Why is a slow rule group a worse failure shape than a failing one?
• How does promtool check rules fit into the CI pipeline?

Quiz