Cardinality Incident Anatomy

At 02:13 the on-call rotation gets paged. Prometheus RSS is climbing past 11 GiB on a host that was sitting at 5 GiB yesterday. The next alert, ten minutes later, fires on OOM. The restart cycle begins: Prometheus comes back, memory is sane for four minutes, then climbs the same curve. The scrape targets have not changed. The configuration has not changed. Something has arrived in the metric stream and is multiplying faster than the storage can flush.

This is a cardinality incident. A cardinality incident is a production event in which Prometheus is asked to retain more unique time series than its head block, working set, and WAL can hold. The proximate symptom is memory; the proximate cause is almost always a label that grew a dimension it did not have before.

What a cardinality incident is

A cardinality incident is not the same as a noisy metric. A noisy metric is one whose values bounce. A cardinality incident is one whose label set has multiplied: a metric that previously had thousands of unique series now has millions, or tens of millions. Prometheus 2.55.x keeps each series in memory until it is flushed to a persisted block, and the working set is bounded by host RAM, not by metric volume. When the series count climbs faster than the head block can flush, the head grows and the process is OOM-killed.

The shape is recognisable: a step change in prometheus_tsdb_head_series, a climb in process RSS that mirrors the series count, and a restart loop if the source keeps pushing.

Why a sysadmin cares

A cardinality incident takes Prometheus down. When Prometheus is down, alerts stop evaluating, recording rules stop firing, and Grafana panels that depend on the affected series turn blank. This is more painful than a single-service outage: the observability platform itself is the casualty. Recovery is gated on finding the offending label, which requires inspecting millions of series under memory pressure. The cost of the incident is paid in the next two hours of the on-call engineer’s night, and the cost of the next incident is paid by everyone who keeps adding metrics without a cardinality budget.

How it works

Cardinality in Prometheus is the count of unique time series currently held in storage. A time series is uniquely identified by the tuple of metric name plus all labels. Two samples are the same series only if every label matches.

Metric: http_requests_total
    |
    +-- method=GET status=200 path=/api/v1/users     => 1 series
    +-- method=GET status=200 path=/api/v1/users/a1b2 => 1 series
    +-- method=GET status=200 path=/api/v1/users/c3d4 => 1 series
    ...
    +-- method=GET status=200 path=/api/v1/users/z9y8 => Nth series

Each label that is unbounded multiplies the series count. The classic offenders are user identifiers, email addresses, raw URL paths containing IDs, session tokens, request UUIDs, query strings, build IDs that include commit metadata, and Kubernetes pod IPs. A single metric with one unbounded label can grow from tens of thousands of series to tens of millions in a day.

How to configure it

Cardinality is not configured; it is earned. The configuration that matters is the one that prevents a cardinality incident from becoming an outage. Two settings carry most of the weight: series-level limits in scrape configs, and metric_relabel_configs that drop the offending label or series before ingestion.

# /etc/prometheus/prometheus.yml
global:
  scrape_interval: 15s
  evaluation_interval: 15s
  external_labels:
    cluster: prod-eu-west-1

scrape_configs:
  - job_name: app
    metrics_path: /metrics
    sample_limit: 10000   # refuse a scrape that returns more than this
    scrape_interval: 30s
    static_configs:
      - targets: ['app-1:9100', 'app-2:9100']
    metric_relabel_configs:
      # Drop series whose label cardinality has exploded
      - source_labels: [__name__, path]
        regex: 'http_requests_total;.*/[a-f0-9]{8,}.*'
        action: drop
      # Strip an unbounded label and keep the rest of the series
      - regex: 'request_id'
        action: labeldrop
      # Cap a numeric label that should be a small set of buckets
      - source_labels: [status]
        regex: '.*'
        target_label: status_class
        replacement: '${1}'
        action: replace

The sample_limit rejects scrapes that exceed a per-scrape threshold; it is the cheap safety net. The metric_relabel_configs block applies after scrape, before ingestion, so the dropped series never enter the head block. The order in the file is significant: each rule is applied to the output of the previous one.

How to validate it

When memory is climbing, the first read is the head block series count and the per-target samples-added counter. Both come from the Prometheus /metrics endpoint.

# READ-ONLY. Current active series in the head block.
curl -s http://prometheus:9090/metrics \
  | grep '^prometheus_tsdb_head_series '

# READ-ONLY. Series created per scrape pool.
curl -s http://prometheus:9090/metrics \
  | grep '^prometheus_target_scrape_pool_samples_added_total'

# READ-ONLY. Top-N series by metric family.
promtool tsdb analyze /var/lib/prometheus/data \
  --human-readable

Illustrative output during an incident:

# HELP prometheus_tsdb_head_series Total number of series in the head block.
# TYPE prometheus_tsdb_head_series gauge
prometheus_tsdb_head_series 1.84e+07

Eighteen million active series. The host has 16 GiB of RAM. The arithmetic is obvious.

How it can fail

The six failure shapes that account for the great majority of cardinality incidents:

1. New unbounded label from an upstream change. A service starts emitting http_requests_total{user_id="..."} where user_id is a per-account UUID. Series count climbs in step with the deployment.
2. URL path label with embedded IDs. A reverse proxy is scraped and the path label carries the request path, including a session token in the URL. Each session is a new series.
3. Auto-discovered label from a scrape target. A new scrape target emits a metric labelled with its container ID or pod IP. The label is unique per process.
4. Replay amplification. Prometheus restarts, replays the WAL, and the head grows during replay before the first block can flush. Memory pressure spikes before the new scrape cycle even begins.
5. Federation or remote_write feeding high-cardinality series. A second Prometheus forwards a metric family that was acceptable in its origin but multiplies the receiver’s series count.
6. Cardinality drift. No single change is large. A hundred small additions each add a few thousand series. The total drifts upward over months and crosses the head-block budget during a quiet weekend when nobody is watching.

How to troubleshoot it

The diagnostic order is fixed: confirm the symptom, locate the metric, locate the label, then act.

1. Confirm the symptom. Read prometheus_tsdb_head_series and process RSS. Both must be climbing together. If only one is climbing, the problem is not cardinality.
2. Locate the metric. Query count by (__name__) ({__name__=~".+"}) in PromQL, or run promtool tsdb analyze to list the top-N series by metric family.
3. Locate the label. For the suspect metric, run count by (label_name) (metric_name) and look for a label whose cardinality is on the same order of magnitude as the series count.
4. Form a hypothesis. Identify which scrape target emits the metric and what changed about its label set recently: deployment, config push, new telemetry line.
5. Find evidence. Cross-reference with the change log. The culprit is almost always correlated with a deploy window.
6. Act. Drop the label or drop the series via metric_relabel_configs. Reload Prometheus. Watch the head series count fall.

Security implications

A cardinality incident can be triggered from outside the trust boundary if Prometheus accepts metrics from an untrusted source: a /metrics endpoint exposed without authentication, a Pushgateway reachable from the internet, or a remote_write target whose URL is reachable by a third party. An attacker who can push arbitrary label values into your Prometheus can deny the observability platform to the rest of the team. Mitigation is network policy and basic auth on the scrape target; the relabel rules apply after scrape, so they do not replace upstream auth.

The label values themselves can carry sensitive data. A label containing a user email, an API key, or a session token is stored in every block file for the retention period and is indexed for every query. The fix is to drop the label at scrape time, not to redact later.

Performance implications

Cardinality is the dominant performance variable for Prometheus. The cost shows up in four places: head-block memory, WAL append throughput, query latency for high-cardinality aggregations, and on-disk index size. A ten-million-series Prometheus writes more bytes per scrape, replays a larger WAL on restart, and returns slower aggregation results than a one-million-series Prometheus doing the same work.

A useful working ceiling for a single Prometheus 2.55.x instance on commodity hardware is roughly five million active series. Past that, sharding via Thanos, Mimir, or VictoriaMetrics pays for itself.

Production guidance

• Treat every label as cardinality debt. The label that does not multiply today may multiply tomorrow.
• Set a sample_limit on every scrape job. 10000 samples per scrape is a generous ceiling for most services; lower it for noisy exporters.
• Track prometheus_tsdb_head_series per instance and alert before the head reaches 80 percent of host memory.
• Run promtool tsdb analyze quarterly against a copy of the data directory; diff against last quarter.
• Keep a change log that ties deploys to metric changes. The fastest path from “Prometheus is OOM” to “label X is the cause” is the diff between the current behaviour and the pre-deploy behaviour.

Verification

• In Prometheus 2.55.x terms, what is a series, and what is its approximate resident memory cost?
• What is the diagnostic order when prometheus_tsdb_head_series climbs in step with process RSS?
• What is the most common cause of a cardinality incident?
• Where in prometheus.yml is the offending label or series dropped before it enters the head block?

Quiz