← All runbooks in Observability
Runbook: Investigate Prometheus High Memory
1 · Prerequisites
Confirm every item is in place before any state change.
- Prometheus server
- Access to the host
2 · Pre-checks
Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.
- · Confirm the spike
- · Identify the timeframe
- · Check recent changes
3 · Procedure
Execute each step in order. Verify the expected output of a step before moving to the next.
- 1Check prometheus_tsdb_head_series (active series)
- 2If series count is high: identify the new high-cardinality metric via tsdb or sum by
- 3Confirm cardinality by inspecting the suspected metric
- 4Apply relabel to drop the offending label or series
- 5Reload Prometheus with the new relabel
- 6If cardinality is intentional: increase memory; plan a long-term scale solution
4 · Verification
Confirm the procedure actually fixed the problem.
- ✓Memory returns to the baseline
- ✓Active series returns to expected
- ✓Prometheus scrape and rule evaluation work normally
5 · Rollback
If verification fails, undo the procedure in reverse order.
- ↶Drop the new relabel
- ↶Revert the cardinality source
6 · Escalation
When the runbook isn't enough, contact:
- · Escalate to the platform team
- · Treat as incident if more than 1 hour to recover
Purpose
Investigate Prometheus High Memory
When to use this runbook
Use this runbook when the operator needs a guided procedure to handle the situation described above.
Pre-checks
Before starting the procedure, confirm the prerequisites and pre-checks are met. The structured lists are rendered from the frontmatter by the page layout.
Procedure
Follow the steps from the frontmatter procedure steps. The page layout renders the steps as a checklist with copy-to-clipboard affordances.
Verification
After the procedure, the structured verification items from the frontmatter are rendered as a checklist.
Rollback
If the procedure fails or makes things worse, follow the structured rollback steps from the frontmatter.
Escalation
The structured escalation path is rendered from the frontmatter. Use it if the operator cannot complete the procedure safely.