Cardinality Explosion in Queries

A Grafana dashboard opens. Six panels load. One of them spins a 30-second query, the panel shows “Loading…” for 30 seconds, then renders 14,000 series with a legend that is 600 pixels tall and unusable. Prometheus meanwhile records prometheus_engine_query_duration_seconds{slice="median"} climbing steadily while the panel is open. The on-call investigation a week later finds the same query path with the same median latency; the dashboard has become a permanent load on the platform.

The query was simple. It was wrong.

group by (request_id) (rate(http_requests_total[5m]))

A counter labelled with request_id produces a series for every single request. The rate returns the rate per request, the group by (request_id) collapses every series into a single-sample-per-request output, and the result is thousands of series with the only label being request_id. The engine processes them in one query. The result is useless and the cost is paid on every page-load.

This lesson is the four patterns that produce that shape, the metrics that make the cost visible, and the bounds that prevent the platform from being held hostage by a single dashboard.

What it is

A cardinality explosion in a query is the outcome of a selector, aggregation, or join that produces an output vector whose series count is dramatically larger than what the operator expected — typically by one or more orders of magnitude. The explosion is at query time, not at scrape time; the underlying TSDB has the same shape before and after the bad query runs, but the output of the query itself is unbounded in size.

Three failure shapes look identical in dashboards:

• An output vector with thousands of series, each carrying one sample per evaluation step. Most line panels render the legend once and the panel becomes unusable above ~50 series.
• An output vector that returns fewer series than the operator expected. Vector matching has silently dropped the rest.
• An output vector that returns the right number of series but with one of them carrying the full aggregated value because the wrong label was used in group by. The surviving series is the orphan; the rest were collapsed into it.

The lesson focuses on the first shape; the second and third are caught by lessons 01 and 05 respectively.

Why a sysadmin cares

The query path is shared. Every dashboard panel, every alert evaluation, every recording rule runs against the same query engine. A single expensive query can:

• Drive prometheus_engine_query_duration_seconds to seconds-and-up. The query scheduler queues other queries behind it. Other panels delay.
• Eat the memory budget. Series-level caching in the engine holds the result of every distinct query in the cache until evicted; a 14,000-series result occupies the cache slot for minutes.
• Page on its own backpressure. The --query.max-samples cap rejects a query that would load too many samples in one go; the response is a rejection message that the alert cannot interpret.
• Trip the query timeout. --query.timeout defaults to 2m for evaluations; the dashboard panel that crosses the threshold returns no data and Grafana renders an empty panel.

The cost is paid by every other consumer of the engine. A single change to a dashboard can degrade the entire Grafana view.

How it works: the four patterns

Pattern 1: regex over a high-cardinality label

# Wrong: regex on a user-id label
{__name__=~"http_requests.*", user_id=~".+"}

# Correct: regex on a low-cardinality label, never on a
# high-cardinality label
{__name__=~"http_requests.*", http_status=~"5.."}

PromQL’s =~ is a full regex match over the label values. A label with user_id (unbounded) versus http_status (bounded to ~50) produces radically different result sizes. The pitfall is that a regex looks cheap syntactically but its cost scales with the number of values in the head that match the prefix.

Pattern 2: group by a high-cardinality label

# Wrong: collapses to one series per request_id; 14000
# series on a busy service
group by (request_id) (rate(http_requests_total[5m]))

# Correct: never group by a label the operator does not
# need; the rate itself is already per-series
sum by (job, code) (rate(http_requests_total[5m]))

The by clause determines the output dimensions. A label that is bounded in the TSDB (e.g. job, instance, code, method) is fine to group by. A label that is unbounded (e.g. user_id, trace_id, request_id, pod_name for short-lived containers) produces an output vector that explodes.

Pattern 3: count( ... ) for “how many” without bounding the input

# Wrong: count of distinct values of a high-cardinality label
count(count by (user_id) (rate(http_requests_total[5m])))

# Correct: count with a selector that constrains the input
count(rate(http_requests_total{job="checkout",code=~"5.."}[5m]))

The first expression computes a per-user_id rate and then counts the resulting series. The count is the answer to “how many users had a 5xx?” which is rarely the question. The question is usually “how many requests” or “how many services” — both of which are answered by aggregations on low-cardinality labels.

Pattern 4: topk/bottomk/quantile over the full head

# Wrong: top-100 hottest series across the entire head; on a
# TSDB with 10M series this is 10M series read into a vector
topk(100, rate(http_requests_total[5m]))

# Correct: bound the selector first
topk(10, sum by (job) (rate(http_requests_total[5m])))

topk and bottomk over the whole head process every series in the input. quantile operates on a vector, which when not constrained first reads the entire input. The defensive use of these operators is to bound the input explicitly before passing it in.

How to configure it

The configuration surface is three flags and one rule.

Bounded query path

# Default engine limits; the production defaults are
# adequate for most platforms.
--query.max-samples=50000000          # 50M samples per query
--query.timeout=2m                    # 2m per evaluation
--query.max-concurrency=20            # parallel query
                                      # evaluations

The --query.max-samples flag is the load-bearing one. A query that exceeds it returns an error message rather than attempting to evaluate. The cost is in the operator experience: a panel that returns “query was rejected” is harder to triage than one that hangs for 2m before timing out. The default value is a trade-off between “we can evaluate the dashboard” and “the engine will not eat the host.”

Rule file with bounded output

# /etc/prometheus/rules/capacity.yml
groups:
  - name: capacity.control
    interval: 30s
    rules:
      # Recording rule that pre-aggregates the fleet total
      # once per scrape interval. Every downstream panel and
      # alert consumes this single sample per interval
      # instead of recomputing the aggregate at query time.
      - record: fleet:http_requests:rate5m
        expr: |
          sum by (job, code) (
            rate(http_requests_total[5m])
          )

      # Alert that depends on the recording rule. The
      # recording rule bounds the alert's per-evaluation
      # cost at one sample per (job, code) pair; the
      # alert itself is one comparison.
      - alert: FleetErrorRateHigh
        expr: |
          (
            sum by (job) (fleet:http_requests:rate5m{code=~"5.."})
            / sum by (job) (fleet:http_requests:rate5m)
          ) > 0.05
        for: 5m
        labels:
          severity: page
          category: errors
        annotations:
          summary: 'fleet error rate for {{ $labels.job }} above 5%'

      # Meta-alert for query-path load. Catches dashboards
      # or alerts that cross the 50M-sample cap.
      - alert: PrometheusQueryRejectionsHigh
        expr: |
          (
            rate(prometheus_engine_query_samples_total{
              type="evaluated"
            }[5m])
            - on() rate(prometheus_engine_query_samples_total{
              type="executed"
            }[5m])
          ) > 10000000
        for: 5m
        labels:
          severity: warning
          category: capacity
        annotations:
          summary: 'query samples dropped by matching/aggregation'

Server-side metric name matching

# Use the URL parameter "match[]" or the POST body field
# "match[]" to filter the series the engine loads into the
# candidate set. This is much cheaper than the selector
# inside the expression because the matcher runs before the
# head block is read.

The --query.lookback-delta is 5m by default. The engine looks back at most that long to find samples for rate() and other range functions. A panel that uses rate(x[1h]) will look back 1h; the engine honours the bigger of the two. The cap matters for memory; a multi-hour dashboard panel does not pull a year of data.

How to validate it

Five commands confirm the cardinality pipeline is healthy.

# 1. Top-10 expensive queries by total samples. The
#    prometheus_http_requests_total counter carries
#    codegen=prometheus/api/v1/query_range samples
#    measurements under prometheus_tsdb_compactions.
curl -sf 'http://prometheus:9090/api/v1/status/runtimeinfo' \
  | jq '.status.GoroutineCount, .status.GOMAXPROCS'
# High goroutine count with high query latency is the
# signature of query path saturation.

# 2. Engine sample-flow counters. The gap between
#    "evaluated" and "executed" measures how many series
#    were dropped by matching/aggregation.
curl -sf 'http://prometheus:9090/api/v1/query' \
  --data-urlencode 'query=prometheus_engine_query_samples_total'
# {"status":"success","data":{"resultType":"vector","result":[
#   {"metric":{"type":"evaluated"},"value":[1724000000,"2.5e10"]},
#   {"metric":{"type":"executed"},"value":[1724000000,"1.4e10"]},
#   {"metric":{"type":"post-filtering"},"value":[1724000000,"1.0e10"]}
# ]}}
# The 1.1e10 gap is the cost of the four patterns above.
# The 4e9 gap on the post-filter side is the cost of
# vector matching (lesson 05).

# 3. Engine timing histogram. The 99th percentile shows
#    the time spent evaluating queries across all
#    consumers.
curl -sf 'http://prometheus:9090/api/v1/query' \
  --data-urlencode 'query=histogram_quantile(0.99, sum by (le) (rate(prometheus_engine_query_duration_seconds_bucket{slice="all"}[5m])))'
# A 99th percentile above 5s means a non-trivial fraction
# of queries is taking seconds. That is the dashboard panel
# the operator sees as "Loading..." for more than 5s.

# 4. Recording-rule evaluation order. The recording rule
#    fleet:http_requests:rate5m should have a scrape_rate
#    that matches its evaluation interval; a mismatch
#    means the rule is being re-evaluated at a cadence
#    that does not match the dashboard panel's rate.
curl -sf 'http://prometheus:9090/api/v1/query' \
  --data-urlencode 'query=count(fleet:http_requests:rate5m{code=~"5.."})'

# 5. The cap has been hit recently. The
#    http_requests_rejected_total counter on
#    /federate is the high-cardinality tail signal.
curl -sf 'http://prometheus:9090/api/v1/query' \
  --data-urlencode 'query=increase(prometheus_http_requests_total{handler="/api/v1/query",code="422"}[1h])'
# A non-zero value means the cap has been hit. Investigate
# which consumer — dashboard panel, alert, or recording
# rule — is producing the rejection.

A coding-level fixture for the four patterns:

# /etc/prometheus/tests/cardinality_test.yml
rule_files:
  - /etc/prometheus/rules/capacity.yml
evaluation_interval: 1m
tests:
  - interval: 1m
    input:
      # A series with a request_id label and a high rate
      # simulates the pattern-2 explosion.
      - series: 'http_requests_total{job="x",code="200",request_id="r1"}'
        values: '0+1x60'
      - series: 'http_requests_total{job="x",code="200",request_id="r2"}'
        values: '0+2x60'
    # Recording rule produces one series per (job, code),
    # not one per request_id. The fixture asserts the
    # output series count.
    promql_expr_test:
      - expr: 'sum by (job, code) (rate(http_requests_total[5m]))'
        exp_samples:
          - labels: 'http_requests_total_total{job="x",code="200"}'
            value: 0.05

How it can fail

1. A new high-cardinality label is added to a popular counter. A developer introduces a customer_id label on http_requests_total. The next day the dashboard panel for sum by (job) (rate(http_requests_total[5m])) is unchanged; the panel for histogram_quantile(...) becomes 10x more expensive. The aggregate is fine; the per-series panel explodes. Detected by prometheus_tsdb_head_series rising without a scrape change.
2. Regex over a label with bounded-but-large cardinality. A regex like customer_id=~"c_.*" on a service with 50M customers per day still touches every customer. The cost is in the regex evaluation, not the output cardinality. The result is the same shape as the user_id case but the explanation is different.
3. topk(100, ...) over the entire head. Without an explicit selector first the function processes every series. The cap is reached before the result is computed. Symptom: dashboard returns “exceeded maximum samples resolution” with the panel still loading.
4. quantile over a high-cardinality time-range vector. quantile(0.95, rate(x[1h])) over a 1h range at 15s scrape interval produces a 240-sample-per-series vector per series; the engine then sorts and selects quantile. On a 100k-series input this is 24M samples in the computation.
5. Recording-rule output label set differs from the alert selector. A recording rule emits fleet:http_requests:rate5m{job="x",code="200"} and the alert reads {job="x",code=~"5.."}. The recording rule fires the alert; the alert is more selective than the recording rule; the alert’s per-evaluation cost is the difference between selectors. The alert is correct; the rule is wrong if it should emit only the consumed labels.
6. Engine memory exhausted. prometheus_engine_query_pool _series_in_use saturates the --query.max-concurrency * result vector cache size. The query path returns errors instead of results. Symptom: dashboards turn to “Panel failed to load” rather than “Loading…”.

How to troubleshoot it

Ordered diagnostics:

1. Inspect the panel’s JSON. Identify the selector and the aggregation. The four patterns above account for most cases.
2. Run the expression with a longer rate window. If the duration drops, the problem is in the range-vector evaluation; if it stays the same, the problem is in the selector.
3. Inspect the sample-flow counters. The evaluated-to-executed gap is the cardinality cost; evaluated-to-post-filtering is the matching cost.
4. Inspect the timing histogram. The 99th percentile buckets give a quick read of the worst queries in the last 5m.
5. Replay against a fixture. The recording-rule fixture verifies the rule’s output labels and bounds the rule’s per-evaluation cost.
6. Bound the query path. Add --query.max-samples=<flag>,<limit> to the dashboard panel via the URL parameter max_samples to fail fast.

Security implications

Cardinality in queries is a denial-of-service surface. An attacker who can submit a query — through the Grafana datasource, through a federation endpoint, through a remote query API — can use any of the four patterns to exhaust engine resources. Two mitigations:

• The Prometheus HTTP API is unauthenticated by default. Bind to a private network or place behind a reverse proxy with authentication.
• The --query.max-samples flag bounds the per-query memory cost. The default is 50M; reduce for environments with untrusted consumers.
• The query user on the API has no label-value restrictions in default Prometheus; the platform security part of this course returns to this surface under the “credential exposure in series labels” section.

Performance implications

The performance costs of the four patterns:

• Pattern 1: full head scan for the regex. On a 10M-series TSDB the cost is in the hundreds of millions of samples per query evaluation.
• Pattern 2: N-series output, where N is the cardinality of the high-cardinality label.
• Pattern 3: a count of a count; on a 100k input the intermediate count is 100k series read into a vector of N values.
• Pattern 4: topk/bottomk/quantile over a high-cardinality input produces a high-cardinality intermediate that sorts the full input vector.

Across patterns, the engine budget is consumed in the selector/matching step. The post-filter step is bounded by the engine’s post-filtering cap.

Production guidance

• Use sum by (job, code) (rate(...)) and similar bounded aggregations as the recording-rule outputs. The fleet: namespace prefix is the convention.
• Alert on --query.max-samples rejections; track the consumer causing the rejection.
• Audit all dashboard panels for the four patterns. The audit is a CI pass that runs the dashboard JSON against a list of banned selectors.
• Apply the --query.lookback-delta only when the panel explicitly uses a longer range than the default; the default is 5m and is correct for most panels.
• Use Grafana’s max data points setting on each panel to bound the per-panel time-series cost.

Verification

You should now be able to answer:

• Which of the four patterns is most commonly the cause of an incident page that says “Prometheus is slow”?
• Why is regex over a high-cardinality label expensive even when the output cardinality is small?
• What is the production default for --query.max-samples, and what is the trade-off the default encodes?
• What does the gap between prometheus_engine_query_ samples_total\{type="evaluated"\} and type="executed"\} measure?
• When should a recording rule be added in front of an alert?

Quiz