Detection

At 04:18 a teammate pings the channel because their dashboard is slow. The CPU panels on the database fleet are green, but Prometheus memory has crossed 80 percent in the last ninety minutes. The page never fires: nobody wired an alert on prometheus_tsdb_head_series. The OOM alert does fire, fifteen minutes later, by which point the restart loop has already started. The signal that would have caught this existed; it was just unwatched.

A cardinality incident is detected, not guessed. The detection metric is prometheus_tsdb_head_series. The detection shape is a step change, not a slope. The detection alert must trip before the OOM, with a margin wide enough that the on-call engineer can act.

What detection means here

Detection in a cardinality incident means early signal that the series count has crossed a budget. It is not the OOM alert. It is not the dashboard refresh. It is an active monitoring condition that fires while there is still time to triage.

Two metrics carry the signal in Prometheus 2.55.x:

• prometheus_tsdb_head_series — current active series in the in-memory head block. The primary cardinality gauge.
• prometheus_target_scrape_pool_samples_added_total — samples per scrape target. The metric that reveals which target is responsible for the climb.

Neither metric alone tells the whole story. The head series gauge says “we are above budget”. The per-target samples-added counter says “this target is the source”. The detection alert watches the gauge; the investigation uses the counter.

Why a sysadmin cares

Detection is the difference between a planned mitigation and a restart loop. A detected cardinality incident is a one-hour investigation followed by a relabel change. An undetected cardinality incident is an OOM, a restart, and the same investigation under heavier pressure, with the WAL replay compounding the load. The same on-call engineer handles both; the cost difference is real.

How it works

The detection chain has three layers: the host metric, the Prometheus self-metric, and the alert that bridges them.

Scrape target emits a new high-cardinality label
    |
    v
prometheus_target_scrape_pool_samples_added_total
    climbs for the affected scrape pool
    |
    v
prometheus_tsdb_head_series climbs
    |
    v
process_resident_memory_bytes climbs
    |
    v
Alert fires (head series above budget OR RSS above budget)
    |
    v
On-call receives a page with a metric, not a symptom

The shape that matters is the step change. A healthy Prometheus grows its series count slowly, in line with new deployments and new scrape targets. A cardinality incident produces a step that is several times the previous baseline in a single scrape interval.

How to configure it

The detection surface in Prometheus 2.55.x is the alerting rules file. The cardinality detection belongs alongside the platform’s own health rules.

# /etc/prometheus/rules/cardinality.yml
groups:
  - name: cardinality
    interval: 30s
    rules:
      - alert: PrometheusHighCardinality
        expr: |
          prometheus_tsdb_head_series > 5e6
        for: 5m
        labels:
          severity: warning
          team: observability
        annotations:
          summary: 'Prometheus {{ $labels.instance }} above 5M active series'
          description: 'Active series at {{ $value | humanize }} of budget.'

      - alert: PrometheusCardinalityStepChange
        expr: |
          prometheus_tsdb_head_series
            >
          3 * avg_over_time(prometheus_tsdb_head_series[6h] offset 1h)
        for: 10m
        labels:
          severity: critical
          team: observability
        annotations:
          summary: 'Head series 3x the previous-hour mean on {{ $labels.instance }}'
          description: 'Step change in active series. Investigate scrape targets.'

The first rule fires when the head block approaches its configured budget. The second rule fires when the head series count is several times the recent baseline, regardless of the absolute budget. Both are necessary: the first catches a slow drift; the second catches a sudden deploy-time jump.

How to validate it

Validation here means proving the alert would fire given the shape of an incident. Two paths:

# READ-ONLY. Inspect current head series count.
curl -s http://prometheus:9090/api/v1/query?query=prometheus_tsdb_head_series \
  | jq '.data.result[0].value[1]'

# READ-ONLY. Per-scrape-pool sample rate.
curl -s 'http://prometheus:9090/api/v1/query?query=\
topk%20by%20(instance)%20(5,%20rate(prometheus_target_scrape_pool_samples_added_total[5m]))' \
  | jq '.data.result'

# CONFIGURATION. Validate the rule file syntax.
promtool check rules /etc/prometheus/rules/cardinality.yml

Illustrative output during detection:

$ curl -s http://prometheus:9090/api/v1/query?query=prometheus_tsdb_head_series \
    | jq '.data.result[0].value[1]'
"18402103"

Eighteen million active series. The alert fires; the investigation starts while the head block still has headroom.

How it can fail

Six failure shapes for cardinality detection:

1. No alert configured on prometheus_tsdb_head_series. The metric is exported but unwatched. The first signal of the incident is the OOM.
2. Threshold set above the host memory ceiling. The alert fires only after the head block is already overflowing.
3. Long for clause that swallows a step change. A for: 30m window misses a deploy-time jump that resolves or crashes within the window.
4. rate() used on a gauge. rate(prometheus_tsdb_head_series) returns no rows because the counter-resetting behaviour of rate on a gauge is undefined. The alert never fires.
5. Alert silences that outlive the incident. A pre-existing silence from a previous on-call hides the new firing.
6. Cardinality drift, not step. No single change is large enough to trip a step detector. The total crosses the budget over weeks.

How to troubleshoot it

The diagnostic order when the alert has fired, or when suspecting one should have:

1. Confirm the gauge is climbing. Read prometheus_tsdb_head_series and the per-instance RSS.
2. Confirm the rule is loaded. Check prometheus_rules_evaluation_failures_total is zero and that the rule appears in the rule files list at /api/v1/rules.
3. Replay the alert logic. Run the alert’s expr directly against the Prometheus API and confirm the value is past threshold.
4. Inspect the alertmanager side. Check alertmanager_alerts and the route tree. A misrouted alert does not page anyone.
5. Confirm the notification path. The PagerDuty, Slack, or OpsGenie integration has its own failure mode. Page the alertmanager test action if unsure.

Security implications

The detection metric itself is benign, but the alert routing that depends on it is not. Cardinality alerts route to PagerDuty, Slack, or another notification channel. A misconfigured alertmanager routing tree can leak infrastructure details (instance IDs, label values) to an unintended recipient. Treat the alertmanager route configuration with the same discipline as a public-facing webhook.

The scrape targets that feed Prometheus are the upstream attack surface for cardinality incidents. An attacker who can push arbitrary labels into a scrape target can deny the observability platform to the rest of the team. The detection metric does not prevent this; it only shortens the time-to-detection.

Performance implications

The detection metric is a gauge sampled once per scrape. Its storage cost is negligible. The detection alert evaluation is also cheap: two queries per rule per evaluation interval. The performance cost of missing a cardinality incident, measured in OOM-killed processes and lost recording rules, is asymmetric.

Production guidance

• Configure the cardinality detection alert on every Prometheus instance; do not rely on the central platform to alert on its own cardinality.
• Use both an absolute ceiling and a step detector. The absolute ceiling catches drift; the step detector catches deploys.
• Review the threshold quarterly. A budget set when the platform held 500000 series becomes irrelevant when the platform holds 5000000.
• Keep for: short on the step detector. A cardinality step change resolves or crashes in minutes, not in thirty-minute windows.

Verification

• Which Prometheus 2.55.x metric is the primary cardinality gauge?
• Why does a step detector catch incidents that an absolute ceiling misses?
• What is the most common detection shape in a cardinality incident?
• Which alert rule expression catches a sudden jump in head series count?

Quiz