Cardinality Incident Response

16:04 — page: PrometheusDown. The host’s monitoring Prometheus has been OOM-killed twice in twenty minutes. 16:07 — head series were 2.8 M at 15:30; the last scrape before death showed 7.1 M and climbing. 16:09 — topk names the family: http_requests_total from the storefront job, which deployed at 15:28 with a new session_id label. 16:12 — a labeldrop rule is merged, Prometheus reloaded. Ingestion of new session_id series stops. 16:15 — the host is still dying, because the 4 M poisoned series already in the head do not leave until they age out. Prometheus is restarted to free the head; WAL replay takes nine minutes. 16:26 — series stable at 2.9 M. Dashboards back. Incident review scheduled.

This lesson is that runbook, written down before you need it.

What it is

A cardinality incident is any event where series growth or churn threatens the availability of the telemetry platform itself. The response has four phases, and the order is the discipline:

RECOGNISE        IDENTIFY          MITIGATE          RECOVER + PREVENT
is this          whose series      stop the inflow,  verify head drains,
cardinality?     are these?        then free RAM     fix at source, guard

The temptation to resist is jumping to restart. A restart frees memory but does nothing about inflow; if the bad label is still arriving, the head refills and you have bought one WAL replay of time. Mitigate inflow first, then decide whether the head needs a restart.

Why a sysadmin cares

A cardinality incident is a meta-incident: it takes away the platform you use to see every other incident. While Prometheus is OOM-looping, alerts are not evaluating (prometheus_rule_group_iterations_missed_total climbs), dashboards are stale, and the business is flying without instruments — often during a deploy window, which is when the bad label arrived.

The second-order risk is the response itself. Hasty relabel rules drop labels someone dashboards on; hasty restarts during compaction can corrupt nothing but cost twenty minutes; hasty delete_series calls are irreversible. The runbook exists so that the stressed version of you does the boring correct thing.

How it works

Phase 1 — Recognise. Cardinality incidents announce themselves through a small set of signals:

• prometheus_tsdb_head_series climbing faster than deploy cadence explains, or vertical.
• Host RSS tracking head series; then journalctl -k shows Out of memory: Killed process ... (prometheus).
• Compaction pressure: prometheus_tsdb_compactions_total and head truncation activity spiking; scrape durations exceeding the interval; prometheus_target_scrape_pool_exceeded_... family moving.
• Remote-write backpressure if you ship samples: prometheus_remote_storage_samples_failed_total and enqueue_retries_total rising as the receiver rejects or the queue saturates.
• On Loki: loki_discarded_samples_total gaining stream_limit / per_stream_rate_limit reasons and ingester restarts (lesson 04’s runbook applies from there).

Phase 2 — Identify. Two queries and one API call:

# Which metric families grew?
topk(10, count by (__name__) ({__name__=~".+"}))

# Who emits them, and which label carries the explosion?
count by (job) ({__name__=~"http_requests_total"})
topk(10, count by (session_id) ({__name__=~"http_requests_total"}))

# Live per-label breakdown straight from the head (READ-ONLY)
curl -s http://localhost:9090/api/v1/status/tsdb | jq '
  {series: .data.seriesCountByMetricName[:5],
   labels: .data.labelValueCountByLabelName[:10]}'

Then anchor to time: deploy markers in Grafana, process_start_time_seconds of the emitting targets, the change log. The answer is almost always “a deploy at HH:MM added a label” or “a rescheduling storm raised churn”.

Phase 3 — Mitigate, in order of increasing blast radius:

# 3a. Stop the inflow: drop the offending label at ingestion.
#     (CONFIGURATION — requires reload; takes effect on next scrape)
scrape_configs:
  - job_name: storefront
    metric_relabel_configs:
      - action: labeldrop
        regex: 'session_id'
      # If the whole family is poison, drop the metric instead:
      # - source_labels: [__name__]
      #   regex: 'http_requests_total'
      #   action: drop
    # Belt and braces while the incident runs:
    sample_limit: 10000

# Apply without a restart (CONFIGURATION)
promtool check config /etc/prometheus/prometheus.yml && \
  kill -HUP "$(pidof prometheus)"
# or, with --web.enable-lifecycle:
curl -s -X POST http://localhost:9090/-/reload

3b. If the host is thrashing now, restart to free the head (SERVICE-IMPACT): systemctl restart prometheus. The WAL replays; rule evaluation and dashboards gap for the replay duration. The poisoned series already ingested are not deleted by this — they simply stop being held live once the relabel rule excludes them, and head memory after restart reflects only what the WAL still references.

3c. If specific series must be actively deleted (rare; DATA-LOSS-RISK), the admin API must have been started with --web.enable-admin-api; then POST /api/v1/admin/tsdb/delete_ series with a selector, followed by clean_tombstones after the next compaction. Deletion marks tombstones; space returns on compaction, not on the call.

Phase 4 — Recover and prevent (covered below).

How to configure it

Pre-stage the incident before it happens:

# Alerting: catch it in minutes, not at the OOM. (CONFIGURATION)
groups:
  - name: cardinality-incident
    rules:
      - alert: HeadSeriesGrowthAnomaly
        expr: |
          (prometheus_tsdb_head_series
            - prometheus_tsdb_head_series offset 1h)
            / prometheus_tsdb_head_series offset 1h > 0.25
        for: 10m
        labels: {severity: critical, team: observability}
        annotations:
          summary: 'Head series grew >25% in 1h'

      - alert: ScrapesHittingSampleLimit
        expr: increase(prometheus_target_scrapes_exceeded_sample_limit_total[10m]) > 0
        for: 5m
        labels: {severity: warning, team: observability}

• Keep a drop-rules snippet file in the same repo as prometheus.yml, with the exact metric_relabel_configs blocks for the known-bad labels, so mitigation is a merge, not authorship under fire.
• Run Prometheus with --web.enable-lifecycle so reloads do not need a process signal from a stressed operator; decide deliberately on --web.enable-admin-api (see security).
• Budget per job from lesson 01 with sample_limit already set: the incident where the limit fires first is the incident you handle as a ticket.

How to validate it

Post-mitigation verification, in order (all READ-ONLY):

# 1. Inflow stopped: new series for the family flatline
curl -s 'http://localhost:9090/api/v1/query' --data-urlencode \
  'query=count({__name__="http_requests_total",session_id!=""})' | jq .

# 2. Churn normalising
curl -s 'http://localhost:9090/api/v1/query' --data-urlencode \
  'query=rate(prometheus_tsdb_head_series_created_total[15m])' | jq .

# 3. If restarted: replay completed and readiness true
journalctl -u prometheus | grep -i 'wal' | tail -5
curl -s http://localhost:9090/-/ready

# 4. Alerts evaluating again
curl -s 'http://localhost:9090/api/v1/query' --data-urlencode \
  'query=prometheus_rule_group_iterations_missed_total' | jq .

# 5. Evidence for the review: block-level attribution
promtool tsdb analyze /var/lib/prometheus/data

Declare recovery when: head series back inside budget, churn at baseline, rule groups evaluating on schedule, and the gap in the graphs is documented with start and end times.

How it can fail

1. Restart-first. The restart frees RAM while the bad label is still being scraped; the head refills. Symptom: second OOM, one WAL replay later.
2. The wrong label dropped. labeldrop on instance instead of session_id. Symptom: dashboards collapse to one line; alerts lose their per-instance grouping; the cardinality is unchanged.
3. Reload never happened. The config was edited on one of two HA replicas, or the HUP went to the wrong pid. Symptom: /-/config shows the old config; growth continues on the unpatched replica.
4. Compaction storm during recovery. The oversized head compacts while ingestion resumes; CPU and I/O contention cause scrape timeouts. Symptom: scrape_duration_seconds over interval, gaps in up, high iowait on the host.
5. Remote-write hole. Samples failed during the incident are gone from the long-term store even after local recovery. Symptom: permanent gap in the remote backend longer than the local outage; prometheus_remote_storage_samples_failed_ total has a step.
6. Loki twin incident. The same deploy also labelled logs; the team fixes Prometheus and misses the ingester restarts. Symptom: loki_discarded_samples_total climbing in the background of the metrics post-mortem.

How to troubleshoot it

When the response itself is not working:

1. Is the rule live? curl -s localhost:9090/api/v1/status/ config (or /-/config) and read the effective metric_relabel_configs. Never trust the file on disk alone.
2. Is the source still emitting? curl the target’s /metrics and grep the label. If yes and the rule is live, your regex or action is wrong — labeldrop matches names, drop matches values.
3. Is memory actually cardinality? RSS high with head series flat points at queries or exemplars, not series growth; re-diagnose before restarting.
4. Is the WAL the boot blocker? If replay is pathologically long, the emergency lever is moving wal/ aside — accept the data loss explicitly (DATA-LOSS-RISK), record the window, and prefer one HA replica surviving while you do it.
5. Still growing after all that? There is a second contributor. Return to the topk query; explosions sometimes arrive in pairs.

Security implications

The incident response surface is an attack surface: /-/reload, /api/v1/admin/tsdb/* and the lifecycle endpoints must not be reachable from the network generally, or “incident response” becomes “anyone can flush your TSDB”. The admin API is disabled by default for exactly this reason; if you enable it, front it with authentication and network policy, and log every call.

The other security angle is why the label exploded. If the values came from request input (paths, headers), the incident may be an attack, not an accident: keep the promtool tsdb analyze output and the emitting access logs for whoever investigates. And if the label values contained personal data or credentials, the incident has a data-protection dimension: deletion from the TSDB, remote backends and backups becomes a requirement, not an option.

Performance implications

• WAL replay is the recovery-time term: minutes to tens of minutes, proportional to WAL size. It is the strongest argument for keeping churn low in steady state.
• Compaction storms after an explosion can cost more availability than the explosion itself; consider briefly raising scrape intervals on the noisiest jobs during recovery rather than letting scrapes time out.
• Sample limits shift cost from “platform absorbs anything” to “offending job fails visibly”. That is the correct allocation during an incident.
• Rule evaluation degrades when head series are high; if alerts matter more than dashboards during the incident, temporarily disable heavyweight dashboard refreshes at the Grafana org level rather than touching alert rules.

Production guidance

• Write the runbook (this lesson, adapted to your hostnames) and link it from the HeadSeriesGrowthAnomaly alert annotation.
• Rehearse: pick a staging Prometheus, add a bad label, run the four phases, time them. The first live run should not be the first run.
• Communicate impact precisely: “metrics gap 16:04-16:26 for jobs X, Y; alerts delayed up to 9 minutes; logs unaffected.” Vague status updates erode trust in the platform you just saved.
• Post-incident, fix at the source (instrumentation or exporter flags), add or tighten sample_limit, extend the drop-rules snippet, and feed the new label name into lesson 06’s governance list.
• If remote write lost data, record the gap in the review; do not quietly let the long-term store disagree with local.

Verification

You should now be able to answer:

• Which four signals distinguish a cardinality incident from a generic “Prometheus is slow”?
• Why must inflow mitigation precede a restart, and what does the restart actually buy?
• What does labeldrop change immediately, and what changes only after head truncation?
• Which endpoints must be protected for this runbook to be safe to enable, and why is the admin API off by default?
• What belongs in the impact communication and the post-incident prevention list?

Quiz