Break/Fix: Prometheus Disk Full

Scenario

You are operating a production observability platform. The following symptoms appear:

• Prometheus TSDB storage full

Available evidence:

• —storage.tsdb.path full
• Prometheus stops ingesting

Your task

Determine the cause, recover, document, and validate.

Investigation

The investigation follows the discipline taught in Part XCVIII:

1. Form hypothesis, find evidence, test, validate.
2. Use the available evidence above to bound the search.
3. Reach one of the likely root causes.

Recovery procedure

(Do not reveal until you have reasoned through the problem.)

1. Identify the failing component.
2. Apply the remediation pathway.
3. Validate with the verification step.
4. Document the incident.

Remediation

1. Verify the disk is full. 2. Increase storage or reduce retention. 3. Restart if needed.

Verification

Disk has free space; Prometheus resumes ingestion.

Rollback

Revert retention change

Prevention

Capacity dashboard; alert at 50% / 75% / 90%.