High-Cardinality Queries

An exporter starts emitting a new label. The label carries a session ID. The exporter was previously emitting forty thousand series; overnight it is emitting four million. Three queries in production touch this metric. Each query now has to walk an index of four million series for every step. Two panels time out. The third panel serves a partial result and visually masks the regression.

The label was added to fix a debugging question. It did fix that question. It also paid a tax on every query that touches the metric, including queries that have never heard of the new label. The label was the right answer for the debugging question and the wrong answer for the platform.

This lesson is about the slow-query shape that lives in the index, not the scan. The fix is rarely a query rewrite; the fix is to drop the label at relabel-config, or to aggregate upstream of the query.

What a high-cardinality query is

A high-cardinality query is one whose evaluation cost is dominated by the index lookup phase. The index phase resolves label matchers to a set of series IDs. Its cost is proportional to the number of series the metric carries and to the selectivity of the matchers.

  cost  =  total_series  x  matcher_selectivity  x  pipeline_ops

       =  cardinality   x  fractional match      x  pipeline_ops

The number of distinct label-value combinations for a single metric is its cardinality. A metric with 3 labels, each with 10 values, has a cardinality of 1 000. A metric with five labels, two of which carry millions of values, has cardinality in the tens of millions.

A label whose value is “always unique” is the high-cardinality shape: request_id, user_id, trace_id, build_id. The cardinality of the metric equals the cardinality of the label. The metric is unusable for any aggregation that has to walk the full set.

Why a sysadmin cares

A high-cardinality label costs every query that touches the metric, not just the query the label was added for. A label added to fix a debugging question becomes a platform tax the moment the metric is queried for any other purpose.

The cost is paid three ways:

• CPU during the index lookup, proportional to the cardinality of the metric and to the number of matchers in the query.
• Memory to hold the index for the head block. Each new series costs roughly three kilobytes of head-block index memory.
• Disk to store the series. Each series costs roughly two bytes per sample in the compressed block, plus the index overhead.

A single label of cardinality ten million added to a metric with cardinality ten thousand takes the metric from ten thousand series to ten billion. The platform cannot store that. The slow query is the symptom; the head-block out-of-memory is the failure.

How to detect a cardinality explosion

Two metrics, read together, give a complete picture.

# READ-ONLY. Total series in the head block, scraped.
curl -s http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=prometheus_tsdb_head_series' \
  | jq '.data.result[0].value[1]'

"10428812"

Ten million active series. The deployment was sized for twenty. The platform is past the ceiling.

# READ-ONLY. Top ten metrics by series count. The metric
# carrying the most series is the offender.
promql='topk(10, count by (__name) ({__name__!=""}))'
curl -s --data-urlencode "query=${promql}" http://prometheus:9090/api/v1/query

{
  "data": {
    "result": [
      { "metric": { "__name__": "http_requests_total" },
        "value": [1735000000.000, "8421000"] },
      { "metric": { "__name__": "node_cpu_seconds_total" },
        "value": [1735000000.000, "512000"] }
    ]
  }
}

Eight million series in http_requests_total. The metric itself is the offender. The next step is to identify which label is exploding.

# READ-ONLY. Cardinality of http_requests_total by label.
promql='topk(20, count by (__name__) ({__name__="http_requests_total"}))'
# Same shape for label breakdown.
promql='sum by (label) (count by (__name__, le) ({__name__="http_requests_total"}))'
# Note: Prometheus does not expose label-cardinality directly,
# but the series count of a metric is the cardinality.

In practice the next step is to inspect the exporter’s configuration or its relabel pipeline to identify the new label.

How to fix it

The fix is to drop the high-cardinality label at the boundary between the exporter and the TSDB. The drop happens in metric_relabel_configs of the scrape job. The label never reaches the platform; the platform never pays the tax.

# /etc/prometheus/prometheus.yml -- relevant fragment.
scrape_configs:
  - job_name: 'api'
    static_configs:
      - targets: ['api:8080']
    metric_relabel_configs:
      # Drop the high-cardinality label before it lands in
      # the TSDB. The label is preserved at the exporter for
      # debugging via the exporter's own log; it does not
      # survive into Prometheus.
      - source_labels: [request_id]
        regex: '.+'
        action: drop

Two other relabel actions are useful.

# /etc/prometheus/prometheus.yml -- relevant fragment.
metric_relabel_configs:
  # Hash a high-cardinality label to a bounded set of buckets
  # when the label is needed for correlation but its raw
  # value is not. Replaces cardinality of 10 million with
  # cardinality of 100.
  - source_labels: [request_id]
    regex: '(.{4}).*'
    action: replace
    target_label: request_id_bucket

  # Aggregate upstream of the query: drop the labels and let
  # the sum land in the TSDB. Useful when the high-cardinality
  # labels have already been aggregated at the exporter and
  # are only ever summed, never grouped by.
  - source_labels: [request_id, user_id, build_id]
    regex: '.+'
    action: labeldrop

labeldrop removes the labels from every series that matches; the sum survives only if the rule is applied to the aggregated result, not the raw series. labelmap is for renaming; labelkeep is the inverse of labeldrop. Read the relabel-config reference before using any of them.

For Loki, the equivalent is to drop labels at the Alloy or Promtail pipeline. Loki’s index is held in BoltDB; high label cardinality has the same cost shape.

// /etc/alloy/config.alloy -- relevant fragment.
loki.source.file "all" {
  targets = discover_files("/var/log/containers/*.log")
  forward_to = [loki.process.us.drop]
}

loki.process "us" {
  stage.drop {
    source     = "request_id"
    drop_counter_reason = "high_cardinality_request_id"
  }
  forward_to = [loki.write.default.receiver]
}

How to validate it

Validation is a two-step process: confirm the label is dropped at the boundary, and confirm the cardinality of the metric is back under the ceiling.

# READ-ONLY. Inspect a single series of the metric to confirm
# the label is gone.
curl -s http://prometheus:9090/api/v1/query \
  --data-urlencode 'query=http_requests_total{job="api"}' \
  | jq '.data.result[0].metric'

{
  "__name__": "http_requests_total",
  "job": "api",
  "service": "checkout",
  "status": "200"
}

request_id is absent. The relabel has landed.

# READ-ONLY. Confirm the metric's cardinality is back under
# the ceiling.
promql='count(http_requests_total)'
curl -s --data-urlencode "query=${promql}" http://prometheus:9090/api/v1/query

{ "data": { "result": [{ "metric": {}, "value": [1735000000.000, "412000" }] } }

Four hundred and twelve thousand series, down from eight million. The head block memory and the index lookup cost are both back under the ceiling.

Reload Prometheus to apply the relabel-config:

# CONFIGURATION + SERVICE-IMPACT (reload only, no restart).
curl -X POST http://prometheus:9090/-/reload

A /-/reload triggers a configuration reload. The TSDB itself does not restart; existing series persist; new series respect the new relabel. The cardinality of the metric will drop over the next compaction cycle as the older series age out of the head block.

How it can fail

Five failure shapes that account for most high-cardinality incidents in production.

1. A new label was added by an exporter upgrade. The exporter started emitting trace_id in the new release. The platform team did not catch the change in the upgrade notes. The cardinality exploded overnight.
2. A label was renamed and the cardinality doubled. A service changed version from 1.2.3 to a per-commit hash. The number of distinct values went from ten to one million.
3. Cardinality ceiling was hit silently. The deployment was sized for ten million series. The platform reached ten million three years in. The next scrape pushed it to ten million and one. Every query that touches the newest series started failing.
4. A label that should have been a metric was added. The exporter started emitting http_response_time_seconds with bucket as a label. The cardinality of the metric is the cardinality of the buckets times the cardinality of every other label.
5. Relabel was applied to the wrong job. The metric carried the right name but the wrong job label. The metric_relabel_configs in the new job did not match. The label survived.

How to troubleshoot it

1. Find the metric. topk(10, count by (__name__) ({__name__!=""})). Note the metric with the highest cardinality.
2. Find the labels. Inspect the metric’s series:

   curl -s --data-urlencode 'query=http_requests_total' \
     http://prometheus:9090/api/v1/query \
     | jq '.data.result[].metric | keys'

   Compare the keys to the expected keys for the metric.
3. Find the source. Identify the exporter that emits the metric. Read its configuration or its relabel pipeline.
4. Drop or aggregate. Apply the relabel-config fix above.
5. Reload and validate. /-/reload, then re-read the cardinality.
6. Audit other metrics. The exporter emits more than one metric. Apply the same audit to the family of metrics, not just the offender.

Security implications

High-cardinality labels are also a privacy concern. A label that carries a user ID, an IP address, or a request body is PII. Once it is in the TSDB, the TSDB has a copy. The relabel-config is the place to drop the label before it lands. Drop at the boundary, not at query time.

A user with the ability to define a relabel-config can also write one that explodes the cardinality. Restrict write access to the prometheus configuration and the alloy configuration files with file permissions. Audit changes with a code-review gate on the configuration repository.

Performance implications

• CPU. Index lookup is proportional to series count. A metric with one million series and a path=~"/api/.*" matcher walks the full index of one million for every step. At fifteen-second scrape interval, the engine walks the index roughly once every fifteen seconds for this query.
• Memory. Three kilobytes per active series in the head block. Ten million series is thirty gigabytes.
• Disk. Two bytes per sample in the compressed block, plus index overhead. A one-million-series metric at one sample per second over thirty days is roughly five terabytes.
• Network. A scrape that returns eight million series per target is multi-megabyte. The scrape budget is per job, not per target.

Production guidance

• Set a ceiling. Decide the maximum active series count the platform will serve and alert when it is approaching.
• Audit every exporter upgrade for label additions. The exporter’s release notes are the right place; if the release notes do not mention labels, assume they have been added.
• Apply metric_relabel_configs at the job level. Drop the labels that the platform does not need.
• Ship the dropped values to logs, not to metrics, when the debugging value is real. A log line per request is cheap; a series per request is not.
• Monitor prometheus_tsdb_head_series against the ceiling. Alert at 80 % of the ceiling, not at the ceiling.

Verification

You should now be able to answer:

• What three costs does a high-cardinality label impose on every query that touches the metric?
• Why is the right place to drop a label at metric_relabel_configs, not in the query?
• How do you distinguish a label that should have been a metric from a label that has a stable set of values?
• What is the cardinality ceiling of your deployment?

Quiz