high riskcluster affecting~30 min

Runbook: Investigate High Cardinality

1 · Prerequisites

Confirm every item is in place before any state change.

Prometheus server
access to /etc/prometheus directory

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Identify the metric with tsdb analysis
· Identify the high-cardinality label

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Use tsdb analyze or sum by to find series count per label set
2Inspect the metric source code or exporter configuration
3Determine the offending label
4Modify exporter configuration or relabel to drop the label
5Reload Prometheus
6Verify memory returns to baseline

4 · Verification

Confirm the procedure actually fixed the problem.

✓Active series count returns to expected
✓Memory returns to baseline
✓New metrics from the source continue to be ingested at expected cardinality

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶Revert the relabel change

6 · Escalation

When the runbook isn't enough, contact:

· Coordinate with the service team owning the metric
· Implement cardinality budget enforcement

Purpose

Investigate High Cardinality

When to use this runbook

Use this runbook when the operator needs a guided procedure to handle the situation described above.

Pre-checks

Before starting the procedure, confirm the prerequisites and pre-checks are met. The structured lists are rendered from the frontmatter by the page layout.

Procedure

Follow the steps from the frontmatter procedure steps. The page layout renders the steps as a checklist with copy-to-clipboard affordances.

Verification

After the procedure, the structured verification items from the frontmatter are rendered as a checklist.

Rollback

If the procedure fails or makes things worse, follow the structured rollback steps from the frontmatter.

Escalation

The structured escalation path is rendered from the frontmatter. Use it if the operator cannot complete the procedure safely.