PromQL Performance Tips

A dashboard panel renders histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))) across six services and forty instances. The dashboard loads in 6 seconds. Every refresh re-evaluates the same histogram_quantile against the same time window. The cost is paid every page-load. The cost is paid every refresh. The cost is paid every dashboard load.

The fix is a recording rule. One histogram:http_request_duration_seconds:p95 per service per interval. The dashboard panel reads a single sample per service per time step; the cost drops by orders of magnitude. The rule itself is a constant cost that the engine pays once per evaluation interval and that Alertmanager, the Grafana datasource, and the federation consumer all share.

This lesson is the production tactics for keeping PromQL fast. Six tips cover 80% of the performance wins.

What it is

“PromQL performance” is the operational discipline of making the engine’s per-evaluation cost predictable and bounded. The engine is single-threaded per query by default; the question is not “how do I make my queries fast?” — it is “how do I pay the cost once and amortise it across every consumer?”

The six tips:

1. Split heavy queries into recording rules.
2. Align lookback windows to scrape intervals.
3. Bound subqueries with :step.
4. Reuse recording rules across alerts, dashboards, and remote consumers.
5. Cap the per-query cost with engine flags.
6. Apply server-side metric name matching for federation and remote-read.

The lesson covers each in production depth.

Why a sysadmin cares

Performance here means three operational things:

• Latency: the time the operator waits for a panel to render. Production-grade dashboards load in under 1s.
• Throughput: the number of distinct queries the engine can evaluate per minute. Default capacity is ~20 parallel queries; a single expensive query consumes the budget for the others.
• Stability: the bound on engine memory and CPU. A query path that exceeds the cap returns errors instead of results; the operators see panel errors rather than blocked panels.

All three are visible to operators during incidents. The relative cost of “PromQL is slow” during an incident is measured in time-to-detection.

How it works: the six tips

Tip 1: split heavy queries into recording rules

The single biggest performance win. A recording rule pre-aggregates a heavy expression once per evaluation interval; every downstream consumer reads the result directly instead of re-evaluating.

# /etc/prometheus/rules/latency.yml
groups:
  - name: latency.p95
    interval: 30s
    rules:
      - record: histogram:http_request_duration_seconds:p95
        expr: |
          histogram_quantile(0.95,
            sum by (job, le) (
              rate(http_request_duration_seconds_bucket[5m])
            )
          )
      - record: histogram:http_request_duration_seconds:rate5m
        expr: |
          sum by (job) (
            rate(http_request_duration_seconds_count[5m])
          )

The dashboard panel that consumes the rule:

histogram:http_request_duration_seconds:p95

Single-sample-per-service per step. Cost is bounded; re-renders are cheap.

The convention is to namespace the recording-rule output with the source metric name and the aggregation: histogram:, fleet:, sli:. The convention makes the rule’s purpose readable in dashboard panels.

Tip 2: align lookback windows to scrape intervals

rate(x[5m]) over a 15-second scrape interval averages over 20 samples. That is the production minimum for stable rate values. A 30-second window would average over only 2 samples; the result is jagged. A 10-minute window averages over 40 samples; the cost is roughly 2x and the smoothness is slightly better.

scrape_interval    | recommended minimum | typical production
-------------------+---------------------+-----------------------
5s                 | 30s                 | 5m
15s                | 1m                  | 5m
30s                | 2m                  | 5m
1m                 | 5m                  | 5m-15m

The discipline is:

• rate window ≥ 4x scrape interval.
• Window ≥ 2x the typical scrape latency to absorb transient pauses.
• Window small enough that the rate reflects recent conditions.

Tip 3: bound subqueries with :step

A subquery rate(x[1h])[10m:30s] evaluates the inner expression every 30 seconds over the last 10 minutes. The :step parameter is the per-step evaluation interval; the length (the bracketed value) is the subquery window.

# Wrong: subquery with no :step — defaults to the engine's
# evaluation interval (1s for range queries); 600 samples
# per series
max_over_time(rate(http_requests_total[5m])[10m:])

# Correct: bound the step
max_over_time(rate(http_requests_total[5m])[10m:30s])

The cost formula is series * ceil(window / step) * inner expression cost. A 10-minute subquery at 30s on 1000 series is 20000 evaluations of the inner expression per query. The :step parameter is the lever; the production default is 30s for panel queries that look at “recent maximum” or “average over recent windows.”

Tip 4: reuse recording rules across consumers

Every consumer should not re-evaluate the same expression. The federation endpoint, the remote-write consumer, the Alertmanager evaluator, the Grafana datasource, and the ad-hoc curl from an operator are all consumers. A single recording rule serves them all.

producer         Prometheus
                  recording-rule evaluation
                    |
                    v
                 fleet:http_requests:rate5m
                    |
            +-------+-------+---------+
            |               |         |
        Alertmanager    Grafana    Remote-read consumer

The cost of the rule is paid once. The cost of every consumer reading the result is O(1) per series per evaluation interval.

Tip 5: cap the per-query cost with engine flags

Three flags control the engine budget:

--query.max-samples=N
   Maximum number of samples that one query can load
   across the head block. The production default is
   50,000,000 (50M).

--query.timeout=DURATION
   Maximum wall-clock duration for an evaluation. The
   default is 2m. Promtool rejects queries that would
   exceed this in alert rule evaluation.

--query.max-concurrency=N
   Number of parallel query evaluations. The default is 20.
   Each goroutine consumes memory; the rule of thumb is to
   leave at least 50% of the host's CPU free for the
   scrape pool.

The flags are at the head of the engine’s enforcement boundary. A panel that exceeds the limit returns an error. A recording rule that exceeds the limit is dropped from the next evaluation (the rule evaluator logs the error and continues).

Tip 6: server-side metric name matching for federation and remote-read

The match[] parameter on the federation endpoint and the remote-read endpoint filters the series the server reads before shipping them across the wire. A federated Prometheus that exports every series at every federation interval consumes bandwidth; a match[]-filtered federation ships only the series the consumer asked for.

# Consumer-side filter on the federation endpoint
GET /federate?match[]={job="node"}&match[]={__name__=~"node_cpu.*"}

The server-side metric name matcher runs before the series are read into the consumer’s evaluation context. The cost saving is at the wire boundary, not at the query boundary. On a 100M-series remote read, the savings are the difference between a 500ms query and a 30-second query.

How to configure it

The recording rules file is the primary configuration surface. The engine flags are command-line options on the Prometheus process.

# /etc/prometheus/rules/optimised.yml
groups:
  # High-cost recording rules. Each rule's interval is
  # matched to the consumer's expected cadence. A rule
  # evaluated at 30s consumed by an alert with for: 5m is
  # wasted work; align the interval to the consumer's
  # threshold.
  - name: latency.aggregates
    interval: 30s
    rules:
      - record: latency:http_request_duration_seconds:p50
        expr: |
          histogram_quantile(0.50,
            sum by (job, le) (
              rate(http_request_duration_seconds_bucket[5m])
            )
          )
      - record: latency:http_request_duration_seconds:p95
        expr: |
          histogram_quantile(0.95,
            sum by (job, le) (
              rate(http_request_duration_seconds_bucket[5m])
            )
          )
      - record: latency:http_request_duration_seconds:p99
        expr: |
          histogram_quantile(0.99,
            sum by (job, le) (
              rate(http_request_duration_seconds_bucket[5m])
            )
          )

  # Lower-cost recording rules aligned to the dashboard
  # refresh cadence. Two-minute interval is acceptable for
  # the operations overview.
  - name: operations.overview
    interval: 1m
    rules:
      - record: ops:http_requests:rate5m
        expr: |
          sum by (job, code) (
            rate(http_requests_total[5m])
          )
      - record: ops:http_requests:error_ratio:5m
        expr: |
          sum by (job) (rate(http_requests_total{code=~"5.."}[5m]))
          / sum by (job) (rate(http_requests_total[5m]))

The Prometheus command-line flags:

prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus/metrics2 \
  --web.console.libraries=/usr/share/prometheus/console_libraries \
  --web.enable-lifecycle \
  --query.max-samples=50000000 \
  --query.timeout=2m \
  --query.max-concurrency=20 \
  --scrape.staleness=5m

The federation match[] syntax:

# Federate from a remote Prometheus, filter to two metrics
curl -sf 'http://prometheus-remote:9090/federate' \
  --data-urlencode 'match[]={job="node"}' \
  --data-urlencode 'match[]={__name__="up"}'

How to validate it

# 1. Inspect rule evaluation latency. A rule group whose
#    last_evaluation_time_seconds approaches its interval
#    is the early warning.
curl -sf 'http://prometheus:9090/api/v1/query' \
  --data-urlencode 'query=prometheus_rule_group_last_evaluation_time_seconds'
# Compare against the configured interval; anything more
# than half the interval is a smell.

# 2. Inspect engine query duration. The 99th percentile
#    is the dashboard panel that loads slowly.
curl -sf 'http://prometheus:9090/api/v1/query' \
  --data-urlencode 'query=histogram_quantile(0.99, sum by (le) (rate(prometheus_engine_query_duration_seconds_bucket[slice="all"}[5m])))'
# Above 5s is "the panel will feel slow."

# 3. Validate recording-rule output cardinality. Every
#    recording rule emits a bounded number of series; the
#    count is the contract.
curl -sf 'http://prometheus:9090/api/v1/query' \
  --data-urlencode 'query=count(latency:http_request_duration_seconds:p95)'
# A count that grows across scrapes means the rule's labels
# are wrong; the rule has the four pattern-2 problems from
# the previous lesson.

# 4. Run the rules through promtool to catch syntax and
#    unit-test against fixtures.
promtool check rules /etc/prometheus/rules/optimised.yml
promtool test rules /etc/prometheus/tests/optimised_test.yml

# 5. Confirm the engine flags are in effect.
curl -sf 'http://prometheus:9090/api/v1/status/runtimeinfo' \
  | jq '.status.GoroutineCount'
# A goroutine count exceeding
#   --query.max-concurrency + scrape pool goroutines
# is the metric that engine flags have not been applied.

A fixture for a subquery with explicit :step:

# /etc/prometheus/tests/optimised_test.yml
rule_files:
  - /etc/prometheus/rules/optimised.yml
evaluation_interval: 1m
tests:
  - interval: 1m
    input:
      - series: 'http_requests_total{job="x",code="200"}'
        values: '0+10x60'
    promql_expr_test:
      - expr: |
          max_over_time(
            rate(http_requests_total[5m])[10m:30s]
          )
        # The fixture asserts that the subquery evaluates
          # 20 inner expressions (600s window / 30s step).
        # If the step were omitted, the engine would use the
          # 1s default and the expression would be 20x more
          # expensive.
        exp_samples:
          - labels: '{job="x"}'
            value: 0.166

How it can fail

1. Recording rule with no labels constrained. A rule like record: foo defined as rate(http_requests_total[5m]) produces one series per existing label combination. The recorder emits unbounded cardinality and the engine chokes on its own output. Detected by count(foo) > expected_count.
2. Look-back window smaller than 4x scrape interval. A rate(x[30s]) against a 15s scrape interval produces jagged values. Alerts misfire. p50 and p95 quantiles shift on every refresh. Detected by inspecting the rate over the same window but with an alternative offset.
3. Subquery with default :step and a long window. max_over_time(rate(x[5m])[1h:]) evaluates the inner expression 3600 times. The query path saturates the engine pool. Detected by the prometheus_engine_query_duration_seconds histogram shifting right.
4. Multiple recording rules with overlapping output. Two rules emit latency:p95 because someone refactored the rule file but kept the old name. The recording-rule conflict is logged at load time. Detected by inspecting the rule-loader log line and by searching for duplicate metric names.
5. Federation match[] over a high-cardinality label. A federated Prometheus with match[]={__name__=~".+"} and no constraint. The federation endpoint emits every series; the receiving side re-stores them. Detected by the wire cost of the federation interval.
6. @ modifier used carelessly with subqueries. rate(x[5m]) @ end() evaluates as if at the end of the range. Combined with subqueries the resulting time series is a constant series with one sample at the modifier’s instant; the panel looks like a horizontal line. Detected by inspecting the panel and the legend’s last-value calc.

How to troubleshoot it

Diagnostic order:

1. Inspect the rule-evaluation latency. A rule group whose last_evaluation_time exceeds its interval is the first symptom.
2. Inspect the query-duration histogram. The 99th percentile bucket is the dashboard panel that loads slowly.
3. Inspect the rule output cardinality. A recording rule whose output count is unbounded or rising is the cardinal-1 problem from the previous lesson.
4. Inspect the active queries. Prometheus admin API exposes /-/queries for live queries. The list includes the consumer, the expression, and the duration.
5. Inspect the rule file for overlapping output names. promtool check rules reports duplicate recording-rule output names at load time.
6. Inspect the federation match[] list. A federation consumer whose match[] list is unrestricted is the wire-cost the operator has to pay.

Security implications

The --query.max-samples flag is a denial-of-service boundary. The federation endpoint is also an attack-surface: an unauthenticated federation consumer can ask for match[]={__name__=~".+"} and exhaust the producer’s bandwidth. Both are addressed in the platform security part of the course; the shorter version is to bind the production Prometheus’s API to a private network and to limit federation to known consumers with explicit match[] lists.

Performance implications

The performance gains from the six tips:

• Recording rule: 10x-100x reduction in dashboard panel cost; the rule itself adds a fixed cost.
• Look-back alignment: 5x reduction in rate noise; modest reduction in samples per query.
• Subquery bounds: 10x reduction in query path memory cost; bounded execution time.
• Rule reuse: linear reduction in total engine cost with consumer count.
• Engine flags: hard ceiling on per-query cost.
• Server-side match[]: linear reduction in wire cost with cardinality.

The full set, applied together, takes a Prometheus installation from “panel loads in 6 seconds” to “panel loads in 200ms” with the same underlying metrics.

Production guidance

• Recording rule for every dashboard panel that consumes a histogram_quantile, every alert that consumes a rate, every panel that consumes an aggregate that touches more than ~10k series in the input.
• Look-back windows ≥ 4x scrape interval; align to the consumer’s refresh cadence.
• Subqueries always explicit :step; default step is 30s for human-facing panels, 10s for short-window alerts.
• One rule per output metric name; the name is the contract.
• Federation match[] is restrictive; never wildcard.
• Engine flags have production defaults; deviations need an annotation in the runbook.

Verification

You should now be able to answer:

• Why is the recording-rule layer the single biggest performance win in production PromQL?
• What is the production minimum for rate(x[N]) against a scrape interval of 15s?
• When should a subquery have an explicit :step, and what is the default if no step is given?
• What is the production default of --query.max-samples, and what failure shape is the cap designed to bound?
• Why is the federation match[] parameter always present in production federation consumers?

Quiz