Cardinality Budget Recurring Review

A quarter after the cardinality incident, the same team runs a review. They take a copy of the data directory, run promtool tsdb analyze, and compare the output against the quarter before. Two metrics have grown by 800000 series each. Five new metric families have appeared. One scrape job has a sample budget that has drifted past its ceiling. None of this would have been visible without the review.

A cardinality budget is a per-metric ceiling on the number of unique series the platform will accept, with a named owner who is accountable for keeping the metric within that ceiling. The recurring review is the discipline that keeps the budget current; without it, the budget is a document, not a control.

What a cardinality budget recurring review is

A cardinality budget is not a Prometheus configuration value; it is an external artefact: a table in a version-controlled document, owned by a person, reviewed on a fixed cadence. The artefact has three columns per metric: the metric name, the ceiling on its series count, and the owner.

The recurring review is the scheduled walk through that artefact. The review produces three outputs: a refreshed ceiling for every metric that has grown, a list of new metrics that need ceilings assigned, and a list of metrics that have fallen below their ceiling and may need to be retired.

Why a sysadmin cares

Cardinality budgets exist to prevent the next cardinality incident. The recurring review is what makes the budgets current; without it, the budgets drift, the ceilings become unrealistic, and the next incident arrives without warning.

The review also produces the data for capacity planning. A metric whose series count is climbing at 10000 per quarter is on a trajectory that matters long before it crosses the absolute ceiling.

How it works

The review is a shell pipeline plus a documentation update. The pipeline produces a snapshot; the diff against last quarter produces the action list.

Quarter N review
    |
    v
Snapshot: copy data directory to a non-production host
    |
    v
Snapshot: promtool tsdb analyze on the copy
    |
    v
Snapshot: top-N metric families by series count
    |
    v
Snapshot: top-N labels for each metric family above its ceiling
    |
    v
Diff: compare against the previous quarter snapshot
    |
    v
Output: action list (new ceilings, new owners, retirements)
    |
    v
Document: refresh the cardinality budget artefact
    |
    v
Follow-up: tickets for any metric that breached its ceiling

Each step is small. The discipline is doing it on a fixed cadence; the cadence is what converts a one-off review into a recurring control.

How to configure it

The recurring review is a cron entry plus a documentation artefact. The cron entry produces the snapshot; the artefact holds the budget; the diff is a script.

# /etc/cron.d/prometheus-cardinality-review
# Quarterly snapshot of the TSDB for cardinality review.
0 3 1 */3 * operator /usr/local/bin/cardinality-snapshot.sh

#!/usr/bin/env bash
# /usr/local/bin/cardinality-snapshot.sh
# DATA-TRANSFER. Copies the Prometheus data directory to the
# review host and runs promtool tsdb analyze against the copy.

set -euo pipefail

PROM_DATA="/var/lib/prometheus/data"
REVIEW_HOST="review.internal"
SNAP_DIR="/srv/cardinality/$(date -u +%Y-Q%q)"
SNAP_FILE="${SNAP_DIR}/analyze.txt"

mkdir -p "$SNAP_DIR"
rsync -a --delete "${PROM_DATA}/" "${REVIEW_HOST}:${SNAP_DIR}/data/"
ssh "${REVIEW_HOST}" \
  "promtool tsdb analyze ${SNAP_DIR}/data --human-readable --limit=100" \
  > "${SNAP_FILE}"

# Optional: diff against the previous quarter.
PREV_DIR="/srv/cardinality/$(date -u -d '3 months ago' +%Y-Q%q)"
if [ -f "${PREV_DIR}/analyze.txt" ]; then
  diff -u "${PREV_DIR}/analyze.txt" "${SNAP_FILE}" \
    | mail -s "Cardinality review $(date -u +%Y-Q%q)" team@observability
fi

# /srv/cardinality/budget.yaml
# Per-metric cardinality budget. Review quarterly.
metrics:
  - name: app_http_requests_total
    ceiling: 1000000
    owner: payments-team
    reviewed: 2026-Q3
  - name: app_request_duration_seconds
    ceiling: 500000
    owner: payments-team
    reviewed: 2026-Q3
  - name: kube_pod_info
    ceiling: 50000
    owner: platform-team
    reviewed: 2026-Q3

The cron entry runs the snapshot on the first day of every quarter. The script copies the data directory, runs the analyzer, and emails the diff against the previous quarter to the observability team. The budget artefact is the source of truth for ceilings and owners.

How to validate it

Validation here means confirming the snapshot, the diff, and the budget are coherent.

# READ-ONLY. Confirm the latest snapshot exists.
ls -la /srv/cardinality/2026-Q3/analyze.txt

# READ-ONLY. Confirm the diff against the previous quarter.
diff -u /srv/cardinality/2026-Q2/analyze.txt \
        /srv/cardinality/2026-Q3/analyze.txt

# READ-ONLY. Confirm a metric is within its ceiling.
grep '^app_http_requests_total' /srv/cardinality/2026-Q3/analyze.txt
# Cross-reference against budget.yaml.

# CONFIGURATION. Validate the budget file syntax.
yq eval '.metrics[].name' /srv/cardinality/budget.yaml | sort -u \
  > /srv/cardinality/budget-metrics.txt
grep -oP '^[a-zA-Z_][a-zA-Z0-9_]*' /srv/cardinality/2026-Q3/analyze.txt \
  | sort -u > /srv/cardinality/observed-metrics.txt
comm -23 /srv/cardinality/observed-metrics.txt /srv/cardinality/budget-metrics.txt

Illustrative output during a review:

# /srv/cardinality/2026-Q3/analyze.txt
app_http_requests_total         812300 series   7 labels
app_request_duration_seconds    421900 series   6 labels
kube_pod_info                    12450 series   9 labels
go_goroutines                       1 series   0 labels
process_cpu_seconds_total           12 series   1 labels

app_http_requests_total is at 812300 of its 1 million ceiling; under budget. kube_pod_info is at 12450 of its 50000 ceiling; under budget. No new metrics outside the budget file. The review passes.

How it can fail

Six failure shapes for the recurring review:

1. Review skipped. The cron runs, but the output is not read. The platform grows past its ceilings between reviews and the next incident arrives without warning.
2. Budget file not version controlled. A budget update lives in a wiki page that drifts out of sync with the actual platform state. The next review compares the wrong ceilings.
3. Owners not assigned. A metric has a ceiling but no owner. When the ceiling is breached, nobody is accountable and the breach persists for another quarter.
4. Snapshot taken from the live data directory. The analyzer reads the active WAL and interferes with the running Prometheus. The review itself becomes an incident.
5. Diff produced but tickets not opened. The diff shows three metrics above their ceilings. The tickets are not opened. The next review shows the same three metrics above their ceilings.
6. Budget ceilings set per scrape job, not per metric. Two scrape jobs emit the same metric. The job-level budget under-counts the metric’s true series count. The metric breaches its effective ceiling while staying under each job’s ceiling.

How to troubleshoot it

The diagnostic order when a review is overdue or producing unexpected results:

1. Confirm the snapshot ran. Check the cron log and the output directory. The latest snapshot’s modification time must be on or near the review cadence.
2. Confirm the data copy completed. rsync exit codes in the cron log must be 0. A partial copy produces a truncated snapshot.
3. Confirm the analyzer ran against the copy, not the live directory. Check the analyzer’s command line in the snapshot script.
4. Confirm the diff is meaningful. The diff is between two snapshots that used the same --limit and the same sort order. Re-run the diff with consistent flags if it is not.
5. Confirm the budget file matches the platform. The set of metric names in the budget file must be a superset of the metric names in the snapshot. Anything outside the budget file is undocumented cardinality.

Security implications

The snapshot of the data directory contains the same label values the production Prometheus stores. The review host is sensitive by the same measure: label values that include user identifiers, session tokens, or API keys are present in the analyzer output. Treat the snapshot and the analyzer output as production data; encrypt the review host, restrict access to the snapshot directory, and apply the same retention policy as the production TSDB.

The diff against the previous quarter may include new label values that did not exist before. Treat this as a signal that a new label has been added and that the source-side change needs review.

Performance implications

The snapshot is a full copy of the TSDB. A 200 GiB data directory takes a non-trivial time to rsync and a non-trivial amount of disk on the review host. Budget for this; do not let the snapshot fill the review host’s disk and trigger its own incident.

The promtool tsdb analyze against a copy is bounded by the size of the data directory and the number of distinct metric families. On a healthy platform with 5 million series, the analyzer finishes in minutes. On a platform that has had a cardinality incident, the analyzer may take longer because the metric families are larger; this is itself a signal.

Production guidance

• Run the snapshot on a fixed cadence; do not let it slip. The discipline is the cadence; the cadence is what catches drift.
• Store the budget file in version control, alongside the Prometheus configuration. Changes to ceilings and owners are reviewed like any other change.
• Diff against the previous quarter, not against a snapshot from a year ago. Quarterly drift is what the review is designed to catch.
• Open a ticket for every metric that breaches its ceiling, even if the breach is small. A small breach is the early signal of a larger one.
• Treat the review host as production data. Encrypt the disk, restrict access, apply the same retention policy.

Verification

• What three columns does a cardinality budget entry contain?
• What is the canonical command for producing a cardinality snapshot from a data directory?
• What is the most common recurring cardinality shape?
• Why must the snapshot be taken from a copy of the data directory, not the live directory?

Quiz