Runbook: Investigate High Cardinality

A cardinality incident is a meta-incident: the thing that breaks is the thing you use to see everything else break. While Prometheus is OOM-looping, alert rules are not evaluating, recording rules are not producing, and every dashboard that depends on the affected series is blank. The business is flying without instruments, and it is usually flying without instruments during a deploy window, because the deploy window is when the bad label arrived.

That framing sets the order of this runbook. The expensive mistake is not a wrong regex; it is doing things in the wrong sequence. There are two sequencing traps, and between them they account for most of the hours lost to this incident class.

Trap one: restarting before stopping the inflow

A restart frees the head block. It does nothing about what is arriving. If the offending label is still being scraped, the head refills on the same curve and you have bought exactly one WAL replay of time - while making the next hour harder, because replay itself consumes memory and delays the first scrape cycle.

Stop the inflow first. Free the memory second. That order holds even when the host is visibly close to the OOM line, because a restart that happens before the relabel rule exists has to happen again afterwards.

Trap two: writing the rule before measuring the blast radius

A labeldrop collapses N series into one. Anything downstream that grouped by (that_label) now returns a single row where it returned a breakdown, and it does so silently: no error, just a different answer. An action: drop on a whole metric family blanks every panel and every recording rule that reads it.

The check is thirty seconds - grep the rule files and the provisioned dashboards for the metric name and the label name - and it is the difference between one incident and two.

When this runbook applies, and when it does not

It applies when prometheus_tsdb_head_series and process_resident_memory_bytes are climbing together. That pairing is the definition of the symptom.

It does not apply when:

• RSS is high and head series is flat. The memory is going somewhere else - concurrent queries, exemplar storage, a large range selector running repeatedly. Re-diagnose before touching a relabel rule.
• One scrape job is failing its sample_limit while the head is stable. That is the limit working as designed: loud at the source, quiet at the reservoir. It is a ticket for the job owner, not a page.
• Prometheus is down for an unrelated reason. A process that will not start is a different runbook; this one assumes a running Prometheus you can still query.
• The growth is a staircase across months with no step. That is drift, and drift is a budget review, not an incident. It is the shape the step detector is blind to and the quarterly review exists to catch.

Blast radius

Steps 1 to 6 are read-only. From step 7 the runbook changes what Prometheus stores, and the changes are not symmetrical:

Read that table in order. Every step above the line is preceded by the thing that makes it reversible - a promtool check config, a repository commit, a measured blast radius. Everything below the line needs a second person.

Step 1 - Prove it is cardinality

curl -s http://localhost:9090/metrics \
| grep -E '^(prometheus_tsdb_head_series|process_resident_memory_bytes) '

prometheus_tsdb_head_series 1.84e+07
process_resident_memory_bytes 1.1817e+10

Eighteen million active series on a host with 16 GiB of RAM. With a planning figure of roughly 3 to 8 KiB of resident memory per active series, the arithmetic does not need a graph.

The cheap negative result matters as much as the positive one. If head series is flat, close this runbook. Every subsequent step in it is designed to reduce series, and reducing series does nothing for a process whose memory is going into query execution.

Step 2 - Separate the level from the churn

Two different failures wear the same badge on a memory graph.

PROM=http://localhost:9090

curl -s "$PROM/api/v1/query" --data-urlencode \
'query=prometheus_tsdb_head_series' | jq -r '.data.result[].value[1]'

curl -s "$PROM/api/v1/query" --data-urlencode \
'query=rate(prometheus_tsdb_head_series_created_total[1h])' | jq -r '.data.result[].value[1]'

curl -s "$PROM/api/v1/query" --data-urlencode \
'query=rate(prometheus_tsdb_head_series_removed_total[1h])' | jq -r '.data.result[].value[1]'

A high level with creations and removals roughly balanced is accumulation: series arrived and are staying. A high creation rate with a matching removal rate is churn: the same conceptual series keeps being recreated under a new identity, which is what a per-pod or per-restart label value does.

The distinction changes what recovery looks like. Accumulation drains when the label stops arriving. Churn keeps paying WAL and index cost even after the level stabilises, and it is the shape that makes restarts expensive, because the WAL is large.

Step 3 - Name the metric family

The TSDB status endpoint answers this without running a query over the whole index.

curl -s http://localhost:9090/api/v1/status/tsdb | jq '{
series: .data.seriesCountByMetricName[:10],
labels: .data.labelValueCountByLabelName[:10]
}'

{
"series": [
  { "name": "http_requests_total", "value": "8421301" },
  { "name": "http_request_duration_seconds", "value": "1204421" },
  { "name": "node_cpu_seconds_total", "value": "88201" }
],
"labels": [
  { "name": "session_id", "value": "8390114" },
  { "name": "instance", "value": "412" },
  { "name": "job", "value": "37" }
]
}

Read both halves. The first names the family that grew; the second names the label whose value space is on the same order of magnitude. When those two numbers are close, you have your answer in one call.

The equivalent PromQL is topk(10, count by (__name__) ({__name__=~".+"})). Use count by, never sum by: count walks the inverted index, sum reads sample values. On a Prometheus that is already under memory pressure that difference is the difference between an answer and a timeout.

If the live instance is too sick to answer, the offline path is promtool tsdb analyze against a copy of the data directory. Copy it - the analyzer reads the WAL, and running it against /var/lib/prometheus/data on the live host adds work to a process that has none to spare.

Step 4 - Name the label, and confirm it

The status endpoint gives a strong candidate. Confirm it against the suspect family before writing anything, because a label that is large platform-wide is not necessarily the label that is large on this family.

Run the two queries below against the suspect family. Substitute the family name and the candidate label name from step 3 before running them.

# How many series does the suspect family have in total?
count(http_requests_total)

# How many distinct values does the candidate label take on that family?
count(count by (session_id) (http_requests_total))

PROM=http://localhost:9090

curl -s "$PROM/api/v1/query" --data-urlencode \
'query=count(http_requests_total)' | jq -r '.data.result[].value[1]'

curl -s "$PROM/api/v1/query" --data-urlencode \
'query=count(count by (session_id) (http_requests_total))' | jq -r '.data.result[].value[1]'

Two numbers within a factor of two of each other means the label is carrying the explosion: nearly every series is unique because of it. A label with a few dozen values on a family with millions of series is innocent, however alarming its name looks.

Step 5 - Name the target, then anchor to a change

curl -s http://localhost:9090/api/v1/query --data-urlencode \
'query=topk(5, sum by (job, instance) (rate(prometheus_target_scrape_pool_samples_added_total[5m])))' \
| jq -r '.data.result[] | "\(.metric.job)\t\(.metric.instance)\t\(.value[1])"'

Then anchor. process_start_time_seconds on the emitting targets, the deploy markers on the dashboard, and the change log together turn a suspect into a cause.

The output of a finished investigation is one sentence with four filled clauses: label X on metric Y from job Z appeared at HH:MM, with change N. If the last clause is blank, you have a correlation and not a cause, and you are about to write a relabel rule you cannot justify in the review. The rule may still be correct - but you will not know, and if the head count does not fall you will not know whether the regex was wrong or the label was.

Step 6 - Choose the mitigation by blast radius

Three actions are available in metric_relabel_configs, and the choice is a question about downstream consumers, not about elegance.

# /etc/prometheus/prometheus.yml
scrape_configs:
  - job_name: app
    metrics_path: /metrics
    sample_limit: 10000
    static_configs:
      - targets: ['192.0.2.11:9100', '192.0.2.12:9100']
    metric_relabel_configs:
      # Preferred: keep the metric, remove the unbounded label.
      # labeldrop matches LABEL NAMES.
      - regex: 'session_id'
        action: labeldrop

      # Heavier: discard the whole family. drop matches label VALUES
      # via source_labels, which is why __name__ appears here.
      # - source_labels: [__name__]
      #   regex: 'http_requests_total'
      #   action: drop

labeldrop matches label names; drop matches label values selected by source_labels. Confusing the two is the commonest reason a syntactically valid rule changes nothing, and the confusion survives several rewrites because the reload keeps succeeding.

Before you commit either one:

grep -rn 'http_requests_total' /etc/prometheus/rules/
grep -rn 'session_id' /etc/prometheus/rules/
grep -rln 'http_requests_total' /var/lib/grafana/dashboards/

A hit in a rule file that groups by (session_id) means labeldrop will collapse that rule to one row. That is not a reason to skip the mitigation; it is a reason to tell the rule owner before they discover it from a flat graph.

Step 7 - Apply, and confirm the reload actually happened

promtool check config /etc/prometheus/prometheus.yml

kill -HUP "$(pidof prometheus)"
# or, where Prometheus runs with --web.enable-lifecycle:
# curl -s -X POST http://localhost:9090/-/reload

sleep 30
curl -s http://localhost:9090/metrics \
| grep '^prometheus_config_last_reload_successful '

prometheus_config_last_reload_successful 1

The exit code of kill is not evidence. Container runtimes and some process managers accept SIGHUP without forwarding it, and the failure mode is an operator rewriting a correct regex three times while the configuration on disk was never read. prometheus_config_last_reload_successful is the only statement about what the running process is using; the effective configuration is also readable at /api/v1/status/config, and on an HA pair it is worth reading on both replicas, because patching one and reloading the other is a real and frequent mistake.

Step 8 - Confirm the inflow stopped, before touching memory

Re-run the step 4 query. The count of series carrying the dropped label must flatline within one scrape interval.

It flattens; it does not fall. This is the part that makes operators think the rule failed. A relabel rule changes what is appended, not what is stored: series already in the head keep their memory until they stop receiving samples, go stale, and are dropped at the next head truncation. The graph goes flat immediately and RSS descends later, sometimes an hour or two later.

If the count is still climbing after two scrape intervals, the rule did not match. Fix the rule. Do not restart - a restart at this point destroys the evidence and buys nothing.

Step 9 - Decide about the head, deliberately

Now the question is only: can this host survive until the head drains itself?

If it can, hold. Holding is a first-class outcome here, not a failure to act. It has an owner (the operator on the incident), a checkpoint (head series and RSS re-read every ten minutes), and an exit condition (RSS below the pre-incident band, or crossing an agreed ceiling that triggers the restart). Write those three things into the incident channel so the next person does not restart it on your behalf.

If it cannot:

systemctl restart prometheus

journalctl -u prometheus | grep -i 'wal' | tail -5
curl -s http://localhost:9090/-/ready

The restart is safe now in a way it was not at step 1, because the relabel rule is live and the head will not refill on the same curve. The cost is a local data gap for the length of the WAL replay - minutes to tens of minutes, proportional to WAL size. On an HA pair, restart one replica at a time and let the surviving replica cover the window.

What “recovered” means

Not “the alert stopped firing”. The alert can stop firing because the scrape config now produces fewer samples, which is a statement about ingestion and not about health.

Recovered is: reload confirmed, offending series flatlined, head series inside the documented budget, RSS descended and settled, churn back at baseline, rule groups evaluating on schedule, and the gap in the graphs written down with a start and an end time. Closing before RSS settles hands the next shift a Prometheus at eighty percent memory that the next ordinary deploy will push over the line.

Common patterns

Prevention, in one paragraph

The relabel rule is the brake. The fix is at the source, and it needs a named owner and a ticket before this incident is closed. The controls that stop the next one are unglamorous and cheap: a sample_limit on every scrape job so a misbehaving target fails loudly instead of quietly draining the reservoir; an alert on both an absolute head-series ceiling and a step change against a recent baseline, because each catches what the other misses; a per-metric ceiling with an owner, reviewed on a fixed cadence; and a drop-rules snippet already in the repository so that mitigation under fire is a merge rather than an act of authorship.