Rule Evaluation

At 14:02 an SLO burn-rate alert fires. The on-call engineer opens the SLI recording rule. The rule is named job:api:request_rate:rate5m. Its expression references job:http_requests_total:rate5m. The dashboard panel that uses the SLI shows a value from fourteen minutes ago. The SLO rule is correct; the SLI it depends on is stale.

The cause is in a different rule file. A new top-k rule was added yesterday; the rule takes ninety seconds to evaluate; the rule’s group interval is thirty seconds; the rule manager logs rule_group_iterations_missed_total climbing. The downstream SLI group waits for its tick; the tick fires before the upstream rule finishes; the SLI reads the previous tick’s value of the upstream metric. The lag compounds.

This is the operational shape of rule evaluation. Rules are not independent goroutines; they share a clock and a budget. A team that does not understand group ordering and interval discipline will add a rule that takes ninety seconds, miss the warning metrics, and chase a stale SLO for hours.

What it is

A rule group is a list of rules that share an evaluation interval and an evaluation goroutine. Each group is dispatched to its own goroutine by the rule manager (rules/manager.go in the Prometheus source). The goroutine ticks at the group’s interval, evaluates the group’s rules in declaration order, and writes the results to the local TSDB. The goroutine is independent of other goroutines; a slow group does not block a fast group, but a group whose evaluation runs past its next tick will skip that tick.

The default evaluation interval is the global evaluation_interval from prometheus.yml (one minute in stock Prometheus). A group can override the interval with interval: 30s (or any Go duration). The override is local to the group.

Three properties follow:

1. Within a group, rules evaluate sequentially. Rule N+1 reads the result of rule N from the current tick (if rule N has finished and written its output) or from the previous tick (if rule N has not finished yet).
2. Between groups, there is no guaranteed order. Groups tick at the same wall-clock instant, but their goroutines are independent. A rule in group A that references a rule in group B reads whatever the previous tick produced.
3. A slow group misses ticks. When a group’s evaluation takes longer than its interval, the rule manager skips the next tick. The metric prometheus_rule_group_iterations_missed_total increments. Downstream rules read the value from the last successful tick.

The evaluation interval is the contract between the rule and its consumers. Changing the interval without telling consumers is a silent behaviour change.

Why a sysadmin cares

Three operational consequences follow from the group model.

1. Order matters for SLI dependencies. A burn-rate SLO rule that reads job:api:errors:ratio_rate5m (a recorded metric) depends on the upstream rule having run. If the upstream rule is in a different group with a longer interval, the burn-rate rule reads the previous tick. The SLO is one interval stale; the alert may be late.
2. The cost of adding a new rule is paid per evaluation. Each rule adds CPU proportional to the number of series its expression produces. A group with twenty cheap rules and one expensive rule pays the expensive rule’s cost on every tick. A team that adds a top-k rule to a group of cheap rules pays the top-k cost forever.
3. Missed ticks are silent. A group whose evaluation overruns its interval does not raise an error. The rule manager logs the missed iteration and moves on. The downstream consumers read stale data; the dashboard reads stale data; the alert may be late.

A team that understands the group model splits rules by cost (cheap in one group, expensive in another), by interval (fast in one group, slow in another), and by dependency (rules that reference each other in the same group). The default — one group per file, all rules at the default interval — works for small deployments and falls apart at scale.

How it works

The rule manager dispatches one goroutine per group. The goroutine runs a ticker at the group’s interval. On each tick, the goroutine:

1. Snapshots the TSDB head block at the tick timestamp.
2. Evaluates each rule in declaration order against the snapshot.
3. Writes the result back to the head block through the same appender used by scrapes.
4. Records the evaluation time in prometheus_rule_evaluation_duration_seconds and the group’s last duration in prometheus_rule_group_last_duration_seconds.
5. Increments prometheus_rule_group_iterations_missed_total if the previous tick has not finished.

  wall clock: 00:00   00:30   01:00   01:30   02:00
              |       |       |       |       |
  group A     [--- tick A1 ---]
  (interval 60s)              [--- tick A2 ---]
                                          [--- tick A3 ---]
                                                  ...
  group B     [-- tick B1 --]
  (interval 30s)    [-- tick B2 --]
                       [-- tick B3 --]
                              [-- tick B4 --]
                                     [-- tick B5 --]
                                            ...

Two things to notice:

• Group A ticks at 00:00 and 01:00. Group B ticks twice between A’s ticks. A rule in group B that depends on a rule in group A reads the value from the previous A tick (if the current A tick has not finished) or from the current A tick (if it has, which is rare when A is slower than B).
• The tick is wall-clock-based, not aligned to scrape intervals. A rule that depends on a metric with a fifteen- second scrape interval reads twenty samples per evaluation if the rule’s interval is five minutes. The samples are the five-minute-window averages the rule needs.

Dependencies between rules

There are three shapes of rule dependency.

1. Same group, later rule. Rule N+1 references rule N. Rule N+1 reads the current tick’s value of N because N has been written to the head block before N+1 evaluates. The dependency is local and tight.
2. Different group, same interval. Rule X in group A references rule Y in group B. Both groups tick at the same instant. X reads Y’s current tick only if Y has finished and written its output before X evaluates. There is no guaranteed order between groups; in practice, the order is the order the goroutines were scheduled. X may read Y’s previous tick.
3. Different group, different interval. Rule X in group A (interval 1m) references rule Y in group B (interval 5m). X ticks five times for every one tick of Y. X reads Y’s previous tick four times out of five. The lag is silent unless you instrument for it.

The shape with the highest operational risk is the third. The team that builds an SLO rule in a fast group that depends on a SLI rule in a slow group will see the SLO reading lag the SLI. The alert may be late by minutes. The fix is to align the intervals or to place the dependency in the same group.

How to configure it

The configuration is the rule file. Three settings matter.

# /etc/prometheus/rules/api.recording.yml
groups:
  # Cheap rules at the default interval. Each rule is one
  # `sum by (job) (rate(...))` expression — a few milliseconds
  # per evaluation.
  - name: api-recording-cheap
    interval: 1m
    rules:
      - record: job:http_requests_total:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))

      - record: job:http_requests:errors:ratio_rate5m
        expr: |
          sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
          /
          sum by (job) (rate(http_requests_total[5m]))

  # Slow rules in a separate group with a longer interval.
  # Each rule is a `topk` over a wide input — tens of seconds
  # per evaluation. Sharing a group with the cheap rules would
  # make the cheap group miss ticks.
  - name: api-recording-slow
    interval: 5m
    rules:
      - record: instance:topk_by_job:http_requests_total:rate5m:top10
        expr: |
          topk(10,
            sum by (job, instance) (rate(http_requests_total[5m]))
          )

      - record: instance:topk_by_route:http_request_duration_seconds:p99:top10
        expr: |
          topk(10,
            histogram_quantile(
              0.99,
              sum by (route, le) (
                rate(http_request_duration_seconds_bucket[5m])
              )
            )
          )

Three things to notice:

• Two groups, two intervals. The cheap rules tick every minute. The slow rules tick every five minutes. The slow rules do not block the cheap rules.
• Group names carry the cost category. cheap and slow are not magic; they are conventions that tell reviewers what to expect. Lesson 05 covers the file-organisation discipline.
• Dependencies are documented in the file header. The slow group reads the cheap group’s output; a header comment makes the dependency explicit. Reviewers can catch a PR that adds a dependency in the wrong direction.

The prometheus.yml reference:

rule_files:
  - /etc/prometheus/rules/*.yml

global:
  evaluation_interval: 1m
  # External labels and other global settings...

How to validate it

Four checks.

1. promtool validates the file.

promtool check rules /etc/prometheus/rules/api.recording.yml

Expected output:

SUCCESS: rules are valid

2. Confirm the groups are loaded at the expected intervals.

curl -s http://localhost:9090/api/v1/rules \
  | jq '.data.groups[] | {name, interval, file}'

Expected output (illustrative):

{
  "name": "api-recording-cheap",
  "interval": 60,
  "file": "/etc/prometheus/rules/api.recording.yml"
}
{
  "name": "api-recording-slow",
  "interval": 300,
  "file": "/etc/prometheus/rules/api.recording.yml"
}

The interval field is the group’s interval in seconds. A value of 60 means one minute; a value of 300 means five minutes.

3. Confirm the groups are keeping up.

curl -s http://localhost:9090/api/v1/rules \
  | jq '.data.groups[] | {
      name,
      lastEvaluation: .lastEvaluation,
      lastDuration: .lastEvaluationTime,
      evaluationTime: .evaluationTime
    }'

lastEvaluation is the wall-clock timestamp of the most recent evaluation. The diff from “now” should be less than the group’s interval; a larger diff means the group is behind.

4. Confirm no group is missing iterations.

prometheus_rule_group_iterations_missed_total > 0

Expected: empty result. Any series returned means at least one group has missed an evaluation tick. The label set on the series identifies the group; the value is the count of missed ticks.

How it can fail

Six failure modes, in the order they appear in production.

1. Slow rule in a fast group. A top-k rule is added to a group with a thirty-second interval. The rule takes ninety seconds to evaluate. The group’s rule_group_iterations_missed_total increments on every tick. Downstream rules read stale data; the alert may be late by minutes.
2. Dependency between groups with different intervals. A burn-rate SLO rule in a one-minute group depends on an SLI rule in a five-minute group. The SLO rule ticks five times for every SLI tick; the SLO reads the SLI’s previous tick four times out of five. Symptom: the SLO panel updates in five-minute jumps; the alert fires late.
3. Recursive rule without backstop. Rule A depends on rule B; rule B depends on rule A. The two rules are in different groups. Each reads the previous tick of the other. The result lags by the larger interval. Symptom: the values are stable but consistently one tick behind reality.
4. Group interval shorter than the slowest scrape interval in the source metric. A rule references a metric scraped every two minutes; the rule’s group ticks every minute. Every other tick reads a stale sample. Symptom: the rule’s output is half-stale on average.
5. Group interval shorter than the rate window. A rule uses rate(metric[10m]); the group’s interval is one minute. The rate window is ten times the evaluation interval; the rule is sampling the rate window ten times faster than the rule ticks. Symptom: the rule output is fine, but the evaluation cost is higher than needed because the rule is doing work the consumer does not see.
6. Failure of one rule blocks the rest of the group. A rule with a syntax error or a missing source metric produces no output; the rule manager logs the failure but continues to the next rule. Symptom: a single broken rule is silent; the rest of the group evaluates normally.

How to troubleshoot it

When a rule’s output is stale or a group’s evaluation is overrunning its interval, the diagnosis order matters.

1. Is the group keeping up? Check prometheus_rule_group_iterations_missed_total. A non-zero value means the group missed at least one tick.
2. How long does each rule take? Check prometheus_rule_evaluation_duration_seconds. Sort by the _sum / _count ratio for a mean; the top decile is where the cost lives.
3. What is the group’s last duration? Check prometheus_rule_group_last_duration_seconds. A value approaching or exceeding the group’s interval is the warning sign.
4. Are dependencies aligned? For every rule that references another rule, confirm the intervals match. A rule in a one-minute group that reads a rule in a five-minute group reads stale data 80 percent of the time.
5. Is the source metric’s scrape interval aligned? A rule’s interval should be at least four times the source scrape interval. A one-minute rule against a one-minute scrape is too tight.
6. Are groups ordered correctly? The rules in a group are evaluated in declaration order. A rule that depends on another rule should be declared after it.

Security implications

Rule evaluation does not introduce a new attack surface beyond what lesson 01 covers. Three considerations apply specifically to group ordering.

1. Rule names leak through /api/v1/rules and include the group name and file path. A naming convention that includes internal codenames (team-payments-internal) leaks more than a convention that uses public names.
2. Rule output is queryable immediately after evaluation. A rule that references a sensitive metric inherits the metric’s labels. A rule named instance:customer_email:count exposes the metric regardless of which file the rule lives in.
3. Group intervals are visible in /api/v1/rules. An attacker with read access to the API can infer the team’s observability cadence. Treat the API as part of the access model.

Performance implications

A rule’s cost is paid per evaluation. The cost has three components.

• CPU: proportional to the number of series the expression produces. A rule that reads rate() over ten thousand series costs more per evaluation than one that reads a hundred.
• TSDB write: proportional to the number of output series. One sample per output series per evaluation.
• TSDB storage: proportional to output series times retention. See lesson 04 for the storage math.

The group’s cost is the sum of its rules’ costs, paid serially because the goroutine evaluates them in order. The goroutine cannot parallelise within a group; that is what the group boundary is for. A team that has a slow rule should put it in its own group with a longer interval; lesson 04 covers the cost breakdown in depth.

A useful rule of thumb: a rule that takes more than ten percent of its group’s interval to evaluate is a candidate for splitting into a separate group.

Production guidance

• Split rules by cost. Cheap rules in one group, expensive rules in another. The expensive group’s interval can be longer without affecting the cheap group’s freshness.
• Align dependencies. Rules that depend on each other should be in the same group, declared in dependency order. Rules that depend on rules in another group should match intervals; otherwise they read stale data.
• Use the default interval unless the rule has a reason not to. A rule that ticks every thirty seconds because the dashboard refreshes every thirty seconds is wasting CPU; a rule that ticks every five minutes because the SLO window is five minutes is correctly configured.
• Monitor missed iterations. A Grafana panel on prometheus_rule_group_iterations_missed_total is a one-line addition that catches the slow-group failure.
• Sort rules in declaration order. A rule that reads another rule’s output should be declared after it. The order is preserved across reloads.
• Document dependencies in the file header. A reviewer reading a PR can see the dependency chain and verify it.

Verification

You should now be able to answer:

• How are rules evaluated within a group, and what is the ordering guarantee between groups?
• Which metric tells you whether a group has missed an evaluation tick, and what is the threshold for action?
• How does a dependency between rules in groups with different intervals affect the freshness of the downstream rule?
• When should a rule be moved from one group to another?

Quiz