Rule Performance

At 11:47 Prometheus CPU climbs from 30 percent to 99 percent in ninety seconds. Alertmanager fires a PrometheusCPUTooHigh alert. The on-call engineer opens the rule evaluation dashboard. The top panel shows ten rules; nine of them have a mean evaluation duration under five milliseconds. The tenth has a mean of nine hundred milliseconds and a rule_group_iterations_missed_total counter climbing by four per minute.

The cause is a rule added last Tuesday. The rule computes the top-100 p99 latencies across every route, every instance, every cluster. The expression fans out to four hundred thousand buckets. The query engine evaluates the histogram_quantile over all of them on every tick. The rule ticks every thirty seconds. The cost is paid forever.

This is the operational shape of rule performance. A recording rule is a database query that runs on a schedule. The cost of the rule is paid every evaluation interval regardless of who reads the result. A team that does not measure rule cost discovers it under an incident. A team that does measure it finds the slow rule before it pages anyone.

What it is

Rule performance is the discipline of measuring the CPU, memory, TSDB write, and storage cost of every rule in the system, and of keeping the cost within a budget that does not crowd out other Prometheus workloads.

The metrics that measure rule performance are:

• prometheus_rule_evaluation_duration_seconds — a histogram of per-rule evaluation durations, labelled by rule name and rule group. The _sum / _count ratio is the mean per-rule duration.
• prometheus_rule_group_last_duration_seconds — a gauge of the most recent group evaluation duration. A value approaching the group’s interval is a warning sign.
• prometheus_rule_group_iterations_missed_total — a counter of evaluation ticks the group has missed because the previous evaluation overran. Non-zero is a warning sign.
• prometheus_rule_evaluation_failures_total — a counter of evaluations that produced an error (syntax error, missing metric, OOM during evaluation). Non-zero is a failure.
• prometheus_engine_query_duration_seconds — a histogram of all query engine evaluations, including ad-hoc queries from Grafana and Alertmanager. Useful for attributing total Prometheus CPU to the query path.
• prometheus_engine_query_samples_total — a counter of the total number of samples loaded by queries, labelled by the query’s slice (inner_eval, query, rule). Sample count is the dominant cost driver for aggregations; a query that loads ten million samples is expensive regardless of how cheap the aggregation is.

The “1, 5, 9” pattern is a heuristic for sorting rules by cost: the rule at the top of the cost ranking (the “1” of the top ten percent) is where most of the CPU lives; the next four are where the secondary cost lives; the remaining rules (“5” and beyond) are inexpensive. The team that audits the top decile finds the rule that dominates the budget.

The prometheus_engine_query_samples_total metric is the most useful single number for attributing cost. A query that loads ten million samples in a one-second evaluation is doing more work than a query that loads ten thousand samples in ten milliseconds, even though the duration is the same. Sample count is the leading indicator; duration is the lagging indicator.

Why a sysadmin cares

A recording rule is paid for whether anyone reads its output. A rule that takes one second to evaluate and ticks every minute burns 1.7 percent of a CPU core continuously. A rule that takes nine hundred milliseconds and ticks every thirty seconds burns 3 percent of a CPU core continuously. Twenty such rules in the same Prometheus instance consume sixty percent of the CPU. The remaining rules, the ad-hoc queries from Grafana, and the scrapes compete for the remaining forty percent.

Three operational consequences follow:

1. A single rule can dominate the budget. The top decile of rules consumes the majority of rule-evaluation CPU. The other ninety percent are rounding errors. Finding the slow rule is a one-line query; fixing it is a re-aggregation or a move to a slower group.
2. Rule cost grows with cardinality. A rule that reads rate(http_requests_total[5m]) is cheap when the metric has ten series; it is expensive when the metric has ten thousand series. The label dimensions passed through a rule multiply cost. A rule that retains a high-cardinality label is a budget bomb.
3. Rule cost grows with rate-window length. A rule that uses rate(metric[5m]) loads five minutes of samples; a rule that uses rate(metric[1h]) loads sixty minutes of samples. The longer the window, the more samples per evaluation.

A team that measures rule performance quarterly finds the slow rule before it pages anyone. A team that does not measures it under an incident, when the metric is already climbing.

How it works

A rule evaluation has three cost phases.

1. Snapshot. The query engine reads the relevant series from the TSDB head block at the tick timestamp. The number of samples read is proportional to the number of series in the input range multiplied by the number of samples per series (which depends on the rate window and the scrape interval).
2. Evaluate. The query engine applies the expression’s operators. Aggregations (sum, avg, topk, histogram_ quantile) have cost proportional to the number of input series. Selectors (rate, irate, increase) have cost proportional to the number of samples read. Functions (histogram_quantile, predict_linear) have cost proportional to the bucket count per series.
3. Write. The output is appended to the TSDB head block. The cost is proportional to the number of output series.

  rule: job:http_requests_total:rate5m
  expr: sum by (job) (rate(http_requests_total[5m]))

  snapshot:  1000 series * 20 samples/series = 20,000 samples
  evaluate: rate() = O(samples); sum by (job) = O(series)
  write:     ~10 output series (one per job)

  cost:      small; output is much smaller than input

A “bottleneck” rule looks like:

  rule: instance:topk_by_route:http_request_duration_seconds:p99:top100
  expr: |
    topk(100,
      histogram_quantile(
        0.99,
        sum by (route, le) (
          rate(http_request_duration_seconds_bucket[5m])
        )
      )
    )

  snapshot:  5000 series * 12 buckets * 20 samples = 1.2M samples
  evaluate: sum by (route, le) = O(series); histogram_quantile = O(buckets);
            topk = O(output * log(input))
  write:     ~100 output series

  cost:      large; histogram_quantile over wide input is expensive

The two rules look superficially similar (both read http_request* and compute a quantile). The first is cheap; the second is expensive. The difference is the cardinality of the input and the number of operators in the expression.

Five expensive aggregation patterns

Five patterns appear repeatedly in the slow-rule dashboard.

1. Wide histogram_quantile. A histogram_quantile() over sum by (...) where ... retains a high-cardinality label (route, tenant, user_id). The function walks every bucket for every series.
2. Top-k over a wide input. A topk(N, sum by (...) (...)) where the input has hundreds of thousands of series. The topk operator is O(output * log(input)); the cost of the underlying sum is the dominant factor.
3. Large join. A binary operator (+, -, *, /, and, or, unless) over two wide vectors. The join matches label sets; the cost is the cross-product of the two input cardinalities.
4. Long rate window over a high-cardinality metric. A rate(metric[1h]) over a metric with many series. The engine reads the full hour of samples for every series; the cost is series * samples-per-hour.
5. Regex matcher on a label. A metric{label=~"pattern"} where the pattern is a complex regex. The matcher evaluates the regex against every series in the metric family; a complex regex is O(pattern) per series.

The rule-iteration cycle

The rule manager ticks the goroutine at the group’s interval. On each tick, the goroutine evaluates the rules in declaration order. If the evaluation takes longer than the interval, the goroutine misses the next tick. The missed-iteration counter increments; the group continues on the following tick.

  interval: 30s
  duration: 90s

  tick 00:00  [start evaluation]
  tick 00:30  skipped (previous still running)
  tick 01:00  skipped
  tick 01:30  [previous evaluation finishes here]
              [start next evaluation]
  tick 02:00  skipped
  tick 02:30  [previous evaluation finishes here]
              [start next evaluation]
  ...

The result is that the group’s output is produced at roughly ninety-second intervals even though the group’s interval is thirty seconds. Downstream rules and dashboards read stale data.

How to configure it

There is no Prometheus flag to make a rule faster. The configuration is the rule itself.

groups:
  - name: api-recording-cheap
    interval: 1m
    rules:
      # Cheap: input is small, output is small.
      - record: job:http_requests_total:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))

      # Cheap: histogram_quantile over a coarse grouping.
      - record: job:http_request_duration_seconds:p99
        expr: |
          histogram_quantile(
            0.99,
            sum by (job, le) (
              rate(http_request_duration_seconds_bucket[5m])
            )
          )

  - name: api-recording-slow
    interval: 5m
    rules:
      # Expensive: top-k over a wide input.
      # Moved to its own group with a longer interval so the
      # cheap group does not miss ticks.
      - record: instance:topk_by_route:http_request_duration_seconds:p99:top10
        expr: |
          topk(10,
            histogram_quantile(
              0.99,
              sum by (route, instance, le) (
                rate(http_request_duration_seconds_bucket[5m])
              )
            )
          )

Three patterns to apply:

• Drop labels with sum by (...). The clause should keep only the labels the consumer needs. sum by (job, le) (...) drops instance and route; the cost is the cardinality of (job, le), which is small.
• Bound the rate window. rate(metric[5m]) loads five minutes of samples; rate(metric[1h]) loads sixty. Use the shortest window that produces the value the consumer needs.
• Move expensive rules to slow groups. A rule that takes ninety seconds belongs in a group with a five-minute interval, not in a group with a thirty-second interval.

How to validate it

Four checks.

1. Identify the slow rules.

topk(10,
  sum by (rule_group, rule) (rate(prometheus_rule_evaluation_duration_seconds_sum[5m]))
  /
  sum by (rule_group, rule) (rate(prometheus_rule_evaluation_duration_seconds_count[5m]))
)

This query returns the ten rules with the highest mean evaluation duration over the last five minutes. A rule with a mean over fifty milliseconds is a candidate for optimisation.

2. Identify the sample-heavy rules.

topk(10,
  sum by (rule_group, rule) (rate(prometheus_rule_evaluation_samples_total[5m]))
)

The prometheus_rule_evaluation_samples_total counter records the number of samples each rule reads per evaluation. A rule that reads millions of samples per evaluation is a candidate for optimisation.

3. Confirm no group is missing iterations.

sum by (rule_group) (rate(prometheus_rule_group_iterations_missed_total[5m])) > 0

Expected: empty result. Any series returned means a group has missed iterations.

4. Confirm the slowest rule fits in its group’s interval.

prometheus_rule_group_last_duration_seconds
  / on (rule_group) group_left
  group(prometheus_rule_evaluation_duration_seconds_count) > 0.1

This query returns groups whose last duration is more than ten percent of their interval. The threshold is a heuristic; adjust to match the team’s tolerance.

How it can fail

Six failure modes.

1. High-cardinality label passed through. A rule retains a high-cardinality label (user_id, request_id, email). The output series count explodes; the TSDB absorbs the series; OOM in days. Symptom: the rule’s evaluation duration climbs as cardinality grows; the TSDB head series count rises.
2. Top-k over a wide input. A top-k rule reads the full cardinality of the source metric. The top-k operator is cheap; the underlying sum is expensive. Symptom: the rule’s evaluation duration is hundreds of milliseconds; the rule’s group misses iterations.
3. Long rate window over a high-cardinality metric. A rule uses rate(metric[1h]) over a metric with many series. The engine reads the full hour of samples; the cost is series * samples-per-hour. Symptom: the rule’s sample count is millions per evaluation.
4. Regex matcher on a label. A rule uses metric{label=~"complex.*pattern.*"}. The regex is evaluated against every series. Symptom: the rule’s evaluation duration climbs with the source metric’s cardinality.
5. histogram_quantile over many buckets. A rule computes a quantile over a histogram with fifty buckets per series. The function walks every bucket. Symptom: the rule’s evaluation duration is tens of milliseconds per series.
6. Rule added without cost review. A team adds a rule to a group of cheap rules without measuring its cost. The group’s total evaluation duration rises; the group’s interval is no longer enough. Symptom: the group’s iterations-missed counter starts climbing.

How to troubleshoot it

When the rule evaluation dashboard shows a slow rule, the diagnosis order matters.

1. Identify the slow rule. Use the topk(10, ...) query above. The top result is the rule that dominates the cost.
2. Identify the cost driver. Read the rule’s expression. Look for high-cardinality sum by (...) clauses, long rate windows, regex matchers, and wide histogram_quantile inputs.
3. Confirm the rule’s group is missing iterations. Check prometheus_rule_group_iterations_missed_total for the rule’s group. A non-zero value confirms the rule is causing the group to overrun.
4. Move the rule to a slower group. Update the rule file to put the slow rule in a group with a longer interval. Validate with promtool check rules. Reload.
5. Drop labels with sum by (...). If the rule retains labels the consumer does not need, drop them. A rule that reads sum by (job, le) (...) is cheaper than one that reads sum by (job, instance, route, le) (...).
6. Bound the rate window. If the rule uses a rate window longer than the consumer needs, shorten it. A rule that uses [5m] instead of [1h] is twelve times cheaper in sample count.
7. Recompute the rule on a different cadence. If the rule feeds a dashboard that refreshes every five minutes, set the rule’s group interval to five minutes.

Security implications

Rule performance does not introduce a new attack surface beyond what lesson 01 covers. Two considerations apply.

1. Rule evaluation is observable in /api/v1/rules. The API returns each rule’s evaluationTime (the duration of the most recent evaluation). An attacker with read access can infer the team’s observability cadence and identify expensive rules.
2. A misconfigured rule can be a denial-of-service vector. A rule with rate(metric[1h]) over a high-cardinality metric is a CPU consumer; if the cardinality grows because of an attacker who controls a label (e.g. email in a public-facing form), the rule becomes a CPU bomb. Treat cardinality as a security boundary.

Performance implications

The performance implications of rule performance are recursive: the metrics that measure rule cost are themselves produced by rules and scrapes. A team that over-instruments Prometheus adds cost to the thing it is trying to measure. The recommendation is to keep the instrumentation dashboard to ten panels and one recording rule.

A useful rule of thumb:

• Mean evaluation under ten milliseconds per rule. A rule that takes longer is a candidate for optimisation.
• Sample count under one million per rule per evaluation. A rule that reads more is reading too much.
• Group duration under ten percent of the group interval. A group that overruns ten percent of its interval is at risk of missing ticks.

A team that measures rule cost quarterly and applies these heuristics keeps Prometheus within budget.

Production guidance

• Measure rule cost with prometheus_rule_evaluation_duration_seconds and prometheus_engine_query_samples_total. The first is the lagging indicator (duration); the second is the leading indicator (sample count).
• Audit the top decile of rules. The slowest ten percent of rules consume the majority of rule-evaluation CPU.
• Move expensive rules to slow groups. A rule that takes ninety seconds belongs in a five-minute group, not a thirty-second group.
• Drop labels with sum by (...). The clause should keep only the labels the consumer needs.
• Bound the rate window. Use the shortest window that produces the value the consumer needs.
• Validate with promtool test rules. A unit-test fixture catches an expensive expression before it ships.
• Watch the missed-iterations counter. A Grafana panel on prometheus_rule_group_iterations_missed_total catches the slow-group failure at the moment it starts.

Verification

You should now be able to answer:

• Which two metrics give you the leading and lagging indicators of rule cost?
• What are the five expensive aggregation patterns, and what is the cheapest fix for each?
• How does the rule-iteration cycle work, and what does prometheus_rule_group_iterations_missed_total > 0 tell you?
• When should an expensive rule be moved to a different group?

Quiz