Skip to main content
RunBook Academy

← All break/fix scenarios in Observability

advancedprometheus-tsdb~30 min

Break/Fix: Prometheus OOM

Reported symptoms

  • ●Prometheus is OOM-killed
  • ●Memory has spiked

Evidence

  • · Prometheus process dies
  • · Active series are high
Diagnosis and resolutionclick to reveal

Root cause

Memory is exhausted; process dies.

Remediation

1. Verify OOM. 2. Identify high-cardinality source. 3. Drop via relabel or scale.

Verification

Memory returns to baseline; Prometheus stable.

Prevention

Scale memory over time, not in a panic.

Scenario

You are operating a production observability platform. The following symptoms appear:

  • Prometheus is OOM-killed
  • Memory has spiked

Available evidence:

  • Prometheus process dies
  • Active series are high

Your task

Determine the cause, recover, document, and validate.

Investigation

The investigation follows the discipline taught in Part XCVIII:

  1. Form hypothesis, find evidence, test, validate.
  2. Use the available evidence above to bound the search.
  3. Reach one of the likely root causes.

Recovery procedure

(Do not reveal until you have reasoned through the problem.)

  1. Identify the failing component.
  2. Apply the remediation pathway.
  3. Validate with the verification step.
  4. Document the incident.

Remediation

  1. Verify OOM. 2. Identify high-cardinality source. 3. Drop via relabel or scale.

Verification

Memory returns to baseline; Prometheus stable.

Rollback

Revert relabel change

Prevention

Scale memory over time, not in a panic.