The Cardinality Budget

At 14:12 on a Tuesday, the monitoring Prometheus for the payments cluster is OOM-killed for the third time in an hour. Nothing is wrong with payments. What is wrong is that a service deployed at 13:40 now emits a metric labelled by customer ID, and the head block grew from 3 million to 11 million active series before anyone noticed. The host has 32 GiB of RAM. The maths lost.

Every Prometheus capacity conversation eventually becomes a conversation about cardinality. This lesson makes the maths explicit, and turns it into a budget you can defend.

What it is

Cardinality is the number of unique time series a system holds. In Prometheus terms, one active series is one unique combination of metric name plus label name/value pairs:

http_requests_total{job="api", instance="10.0.1.4:8080",
                    method="GET", status="200"}

is one series. Change any single label value — status="500", instance="10.0.1.5:8080" — and you have another. The series count of a metric is the product of the sizes of its label domains.

A cardinality budget is an explicit, written cap on how many active series a team, service, or scrape job may contribute, paired with a way to measure consumption and an alert before the cap is reached. It converts an invisible shared resource (the Prometheus head block) into an allocated one.

Why a sysadmin cares

Prometheus scales along one axis that hurts before all others: active series. CPU is usually fine. Disk throughput is usually fine. Memory is not, because every active series costs resident memory in the head block for as long as it is active.

The operational consequences:

• OOM kills. The kernel OOM killer picks the largest process. On a monitoring host, that is Prometheus. Each kill costs you a WAL replay and a gap in rule evaluation.
• Slow everything. Query time, rule evaluation, head GC, and compaction all scale with series count. A Prometheus that was snappy at 1 million series is visibly ill at 10 million.
• Slow restarts. Startup replays the WAL. More series and more samples in the WAL means minutes of downtime per restart.
• Bill shock. If you remote-write to a metered backend, series count is literally the invoice line.

Without a budget, the platform’s limit is discovered by the worst possible probe: production traffic interacting with the newest deploy.

How it works

The mental model is a single reservoir with many taps:

 job: node_exporter      job: api-gw        job: checkout
   ~5k series/host        ~40k series          ~80k series
         \                    |                   /
          \                    |                  /
           v                   v                 v
        +---------------------------------------------+
        |          Prometheus head block              |
        |   active series held in RAM, appended via   |
        |   WAL, compacted to 2h blocks on disk       |
        +---------------------------------------------+
                           |
              budget alert fires at 80% of capacity

Every scrape job pours series in. The reservoir has a practical size set by host memory. The budget is the agreement about how much of the reservoir each tap is allowed to use.

Two numbers describe the inflow:

1. Level — how many active series exist right now (prometheus_tsdb_head_series).
2. Churn — how fast series are created and removed (rate(prometheus_tsdb_head_series_created_total[1h]) and rate(prometheus_tsdb_head_series_removed_total[1h])).

A platform can survive a high level with low churn, and die from moderate level with extreme churn, because churn drives garbage collection, index rebuilds, and WAL size. A healthy platform watches both.

How to configure it

A budget has three parts: a number, a measurement, and a tripwire.

1. Write the number down. Keep it in version control next to prometheus.yml:

# cardinality-budgets.yaml — owned by the observability team
# Unit: active series contributed to the shared Prometheus head.
defaults:
  max_active_series_per_job: 50000   # per scrape job, per Prometheus
  warn_at_fraction: 0.8              # alert at 80% of any cap
platform:
  head_series_soft_cap: 8000000      # sized from host RAM, see below
teams:
  - team: checkout
    jobs: [checkout-api, checkout-worker]
    max_active_series: 250000
  - team: platform
    jobs: [node_exporter, kube-state-metrics]
    max_active_series: 900000

Size the platform cap from memory, not from hope: with a planning figure of 4-8 KiB per series and a 32 GiB host reserved to 20 GiB for Prometheus, the defensible cap is roughly 2.5-5 million series. Measure the real per-series cost on your data (see validation below) and revise.

2. Add hard per-scrape limits so one bad job cannot drain the reservoir before the alert fires:

# prometheus.yml
scrape_configs:
  - job_name: checkout-api
    scrape_interval: 15s
    static_configs:
      - targets: ['10.0.1.4:8080', '10.0.1.5:8080']
    # Fail the scrape, loudly, instead of accepting a flood.
    sample_limit: 20000        # max samples accepted per scrape
    label_limit: 40            # max labels per target after discovery
    label_name_length_limit: 120
    label_value_length_limit: 512

A breached sample_limit fails the scrape and increments prometheus_target_scrapes_exceeded_sample_limit_total, which is exactly the behaviour you want: noisy at the source, silent at the reservoir.

3. Alert on the budget, not just on memory:

groups:
  - name: cardinality-budget
    rules:
      - record: job:active_series:count
        expr: count by (job) ({__name__=~".+"})

      - alert: JobCardinalityOverBudget
        expr: job:active_series:count > 50000
        for: 30m
        labels: {severity: warning, team: observability}
        annotations:
          summary: 'Job {{ $labels.job }} exceeds its series budget'

      - alert: HeadSeriesApproachingCap
        expr: prometheus_tsdb_head_series > 6400000   # 80% of 8M
        for: 15m
        labels: {severity: critical, team: observability}

      - alert: SeriesChurnHigh
        expr: rate(prometheus_tsdb_head_series_created_total[1h]) > 20000
        for: 30m
        labels: {severity: warning, team: observability}

How to validate it

Confirm the level, the largest contributors, and the churn. All commands are READ-ONLY.

# 1. Current active series in the head (also in Grafana)
curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=prometheus_tsdb_head_series' | jq .

# 2. Series per metric name, worst first
topk(15, count by (__name__) ({__name__=~".+"}))

# 3. Series per scrape job (compare against the budget file)
count by (job) ({__name__=~".+"})

# 4. Churn: creations and removals per second
rate(prometheus_tsdb_head_series_created_total[1h])
rate(prometheus_tsdb_head_series_removed_total[1h])

# 5. The TSDB status API: per-metric and per-label breakdowns
curl -s http://localhost:9090/api/v1/status/tsdb | jq '
  .data.seriesCountByMetricName[:10]'

Illustrative output:

[
  { "name": "http_requests_total",        "value": "412088" },
  { "name": "container_memory_working_set_bytes", "value": "96514" },
  { "name": "node_cpu_seconds_total",     "value": "88201" }
]

# 6. Offline: per-block cardinality and churn analysis
promtool tsdb list /var/lib/prometheus/data
promtool tsdb analyze /var/lib/prometheus/data 01J8K3XW7M0R2Y3T4V5B6N8Q9

promtool tsdb analyze reports the block’s series count, churn, and the highest-cardinality label pairs — the same answers the HTTP API gives for the head, but for an on-disk block, so it works on a snapshot copied off the host.

How it can fail

1. The slow leak. Series count grows 5% a week for months. No alert fires because nobody set one; the first symptom is a page for OOM at an unrelated traffic peak. Symptom: prometheus_tsdb_head_series on a 90-day graph is a staircase.
2. The instant explosion. A deploy introduces an unbounded label. Head series grow by millions per hour; OOM within the retention of the on-call coffee. Symptom: near-vertical series graph starting at a deploy marker.
3. Churn without growth. Kubernetes pods reschedule all day; pod-level series are created and removed constantly. Level looks fine, but head GC, compaction and WAL size climb. Symptom: prometheus_tsdb_head_series_created_total rate is a large fraction of the total series count.
4. Compaction storm. After an explosion, the oversized head must compact. Compaction competes with ingestion for CPU and I/O; scrapes start timing out. Symptom: prometheus_tsdb_compactions_total spiking together with scrape_duration_seconds exceeding scrape_interval.
5. Restart purgatory. After the OOM, WAL replay of a huge head takes tens of minutes; the host pages again or the boot is killed and retried. Symptom: Prometheus logs replay WAL progress lines for a very long time.
6. Remote-write backlog. If you remote-write, a series surge can outrun the receiver’s per-tenant limits; samples are retried then dropped. Symptom: rising prometheus_remote_storage_samples_failed_total and prometheus_remote_storage_enqueue_retries_total.

How to troubleshoot it

Ordered, cheapest first:

1. Confirm it is cardinality. Graph prometheus_tsdb_head_series and host RSS together. If they climb together, it is cardinality; if RSS climbs alone, suspect a query or recording-rule cost instead.
2. Find the contributor. Run the topk query from the validation section, then drill into the worst metric with count by (job, instance) (worst_metric_name) and count by (suspect_label) (worst_metric_name).
3. Correlate with change. Overlay deploy annotations in Grafana; check process_start_time_seconds for recently restarted targets; ask what shipped.
4. Check churn, not just level. A flat level with high created rate points at ephemeral labels (pod names, IDs), not at a new metric.
5. Decide: contain or evict. If the platform is stable, fix forward with relabel rules. If the host is thrashing, mitigate first (lesson 05 in this module is the runbook).

Security implications

Cardinality is an availability boundary, and availability is a security property. Any endpoint that can influence label values — a user-controlled URL path echoed into a label, an unauthenticated /metrics that accepts query parameters — is a denial-of-wallet or denial-of-service vector against the monitoring platform. The per-scrape limits above are therefore also security controls.

Label values are also data. If a dangerous label carries user identifiers (the next lesson), those identifiers are replicated into the TSDB, remote-write receivers, backups and snapshots. Treat the metrics pipeline as a data store in its own right when classifying what may flow through it; the platform security part of the course covers authentication and transport for each endpoint.

Performance implications

• Memory: the planning figure is 3-8 KiB resident per active series; measure yours with process_resident_memory_bytes divided by prometheus_tsdb_head_series during steady state.
• CPU: ingestion cost per sample is modest (prometheus_tsdb_head_samples_appended_total tells you the rate); the cardinality-driven CPU costs are head GC, compaction, and rule evaluation over many series.
• Disk: WAL and blocks grow with churn; du the data directory weekly and graph it.
• Query cost: queries that touch many series (sum by () over millions of inputs) get slower roughly linearly; budget alerts indirectly protect dashboard latency.

The trade-off of a tight budget: teams must sometimes aggregate away a dimension they wanted (per-endpoint latency becomes per-route-group latency). That is a real observability loss. The budget forces it to be a conscious loss, priced in series, instead of an unconscious one priced in RAM.

Production guidance

• Put the platform cap, per-team numbers and the measurement queries in one reviewed file (cardinality-budgets.yaml). A budget that is not written down is a rumour.
• Alert at 80% of budget per job and at 80% of the platform cap. Page on the platform cap; ticket on per-job overage.
• Set sample_limit on every job. Accept that a misbehaving job failing its scrape is better than a misbehaving job taking the platform with it.
• Review the topk report weekly in the observability team’s own dashboard. Budgets decay without attention.
• Re-measure the per-series memory cost after every major Prometheus upgrade; the figure moves between releases.

Verification

You should now be able to answer:

• What is one active series, and what does one cost in RAM?
• Which three numbers (level, churn, ingestion rate) describe cardinality pressure, and which metric exposes each?
• How do you find the job and metric most responsible for current head size, live and from an on-disk block?
• What does sample_limit do when breached, and why is that the failure shape you want?
• How do you size a platform series cap from host memory?

Quiz