Cardinality Bottlenecks

Prometheus OOMs every four days. The head block has 14 million active series. Memory grew from 6 GB to 22 GB in the same four days. The platform team restarts Prometheus and watches the number climb again. The cause is not the scrape pipeline; the cause is the cardinality of one particular metric, which a single application team began emitting with a request_id label last Tuesday.

This lesson is about cardinality bottlenecks: how one unbounded label can collapse an otherwise healthy Prometheus, and what the operator can do to bound the damage.

What it is

A cardinality bottleneck is the condition where the number of unique label combinations exhausts Prometheus memory and CPU. Each series — a unique combination of metric name plus labels — costs roughly 3 KB of memory in the head block. Each scrape appends one sample per series. A metric that adds a new series every second grows memory linearly and without bound.

The bottleneck is silent until it is not. Memory grows for hours or days. Scrape duration creeps up. Queries slow. Eventually Prometheus hits its memory limit and the OOM killer fires.

Why a sysadmin cares

Cardinality is the single largest variable in the Prometheus cost equation. Three production pains concentrate in cardinality bottlenecks:

1. Memory growth without an obvious cause. A new deployment that emits a UUID label looks identical to the previous deployment to a deploy script. Memory grows from the next scrape onwards. The operator notices the symptom four days later.
2. Query latency drift. A query that walked 100 000 series yesterday walks 14 million today. The query engine is bounded by CPU, and the CPU is bounded by the series count.
3. Compactor falls behind. The compactor merges blocks at a cost proportional to the series count. A ten-fold cardinality explosion triples the compactor CPU and lengthens every merge cycle.

The lesson is that cardinality is a budget. The platform has a fixed memory ceiling; the budget for that ceiling is the per-team allowance of unique series.

How it works

A series in Prometheus is the unique combination of metric name and label set. The fingerprint is a hash of the labels; the index maps fingerprints to series IDs; the series ID maps to chunk files on disk.

   metric_name + labels
        |
        v
   +-----------------+
   |  fingerprint    |  =  hash(metric_name, sorted labels)
   +-----------------+
        |
        v
   +-----------------+
   |  series ID      |  =  integer assigned on first scrape
   +-----------------+
        |
        v
   +-----------------+
   |  head block     |  =  in-memory posting list + chunk
   +-----------------+

The memory cost per series is dominated by the posting list and the chunk headers. The official documentation cites roughly 3 KB per active series, but the actual cost depends on the number of labels per series. A metric with 10 labels costs more than a metric with 2.

The scrape rate compounds the cost: a 10-second scrape interval means 6 samples per minute per series; a 30-second interval means 2 samples per minute. Cardinality multiplied by scrape rate is the platform’s true load.

How to configure it

Cardinality is bounded at three layers: the scrape target, the relabel pipeline, and the platform-wide budget. All three are necessary in production.

# Severity: CONFIGURATION
# /etc/prometheus/prometheus.yml
scrape_configs:
  - job_name: 'application'
    sample_limit: 5000            # cap on samples per scrape
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: 'debug_.*'
        action: drop
      - source_labels: [request_id]   # an unbounded label
        regex: '.+'
        action: labeldrop
      - source_labels: [user_id]      # a PII label
        regex: '.+'
        action: drop

Three rules govern these knobs:

1. metric_relabel_configs runs after parsing. Drop unwanted metrics or labels here, before they hit the head block.
2. sample_limit caps the number of samples per scrape. A scrape that exceeds the limit drops the surplus.
3. relabel_configs runs before parsing, on target labels. Use it to drop entire targets that exceed a cardinality budget.

A platform-wide budget is best expressed as an alert on prometheus_tsdb_head_series:

# Severity: READ-ONLY
# Alert when active series exceed 5 million
prometheus_tsdb_head_series > 5e6

How to validate it

Validation is read-only. Confirm the head block size, the top metrics by series count, and the label cardinality of the worst offender:

# Severity: READ-ONLY
curl -s http://prometheus:9090/api/v1/status/runtimeinfo \
  | jq '.data'

The expected result exposes the active series count and the head block statistics.

The top metrics by series count:

# Severity: READ-ONLY
topk(10, count by (__name__) ({__name__=~".+"}))

A healthy platform shows tens to hundreds of series per metric. A platform with a cardinality problem shows one metric with millions.

The label cardinality of the worst metric:

# Severity: READ-ONLY
topk(10, count by (__name__) ({__name__="http_requests_total"}))
count by (request_id) ({__name__="http_requests_total"})

The first query returns the series count. The second returns the unique values of request_id — typically a UUID with cardinality approaching the request rate. If that cardinality exceeds a few thousand, the metric is the bottleneck.

How it can fail

Six failure shapes account for nearly every cardinality incident:

1. A UUID label such as request_id on every request. The metric gains a new series per request. Symptom: prometheus_tsdb_head_series grows at the request rate.
2. A timestamp label. A timestamp label such as event_time produces a new series per second per source. Symptom: the head block grows at one series per second per source indefinitely.
3. A URL label with the query string. A label such as url="https://api.example.com/users?id=42" produces a new series per unique URL. Symptom: cardinality grows linearly with API surface area.
4. A K8s pod name being unbounded. A label that contains the pod name is bounded by the number of pods — until a deployment loop creates thousands of pods in minutes. Symptom: a sudden 1000x jump in head series.
5. Multi-tenant labels without per-team governance. A label that includes a team ID is bounded by the number of teams — until a new team emits a metric that fans out across all the existing teams’ series. Symptom: the head series for the shared metric multiplies by the number of teams.
6. A debug metric turned on in production. A metric that emits per-request debug data turns a 1k-series metric into a 1M-series metric. Symptom: the head series count doubles within a single scrape interval.

How to troubleshoot it

The diagnostic order is consistent across all six failure shapes:

1. Confirm the symptom is cardinality growth. Inspect prometheus_tsdb_head_series. A flat line for days means stability. A rising line means growth.
2. Identify the worst metric. Run the top-k query above. Identify the metric with the largest series count.
3. Inspect its label cardinality. Run the per-label count query. Identify the label with the largest distinct count.
4. Identify the source. Grep the source code or the exporter output for the offending label. Most cardinality bugs are introduced in a single PR.
5. Add a relabel rule. Drop the offending label or metric via metric_relabel_configs. Reload Prometheus.
6. Wait for the next scrape. The head block retains the offending series for up to 2h after the source stops emitting them. Recovery is not instantaneous.

Security implications

Cardinality bottlenecks frequently intersect with data protection. Three disciplines matter in production:

1. PII must not appear in label values. A label of email or phone or ip is a privacy violation independent of its cardinality. Drop these labels at the relabel stage, before they reach the head block.
2. A label that contains a secret is a secret disclosure. A token label or api_key label leaks the credential to every consumer of the metric. Treat label values with the same care as log lines.
3. Cardinality exposure is a surface. A label that contains an internal hostname or instance ID discloses the platform’s topology. Restrict the label set to operational dimensions.

Performance implications

Cardinality is the dominant variable in the Prometheus cost equation. The arithmetic is:

active_memory
    = active_series * per_series_cost
ingestion_cpu
    = active_series * scrape_rate * per_sample_cost
query_cpu
    = series_in_expression * step_count * per_step_cost

Halving the active series count halves the memory, roughly halves the ingestion CPU, and halves the compaction CPU. It also halves every query that walks the affected metric. There is no other single knob in Prometheus with a comparable leverage.

Verification

You should now be able to answer:

• How many bytes of memory does each active series typically cost in the Prometheus TSDB head?
• Which two relabel rules are most useful for bounding cardinality, and where do they run in the pipeline?
• Which PromQL function ranks metrics by series count, and which query inspects a single label’s cardinality?
• What is the first metric to alert on for cardinality growth, and what threshold makes sense for a small platform?
• How long does it take for the head block to forget a series that is no longer being scraped?

Quiz