critical riskcluster affecting~30 min

Runbook: Recover a Broken Prometheus

1 · Prerequisites

Confirm every item is in place before any state change.

Recent backup of TSDB
Configuration in version control

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Determine whether to restore from backup or rebuild from configuration
· Check the cause: configuration error, disk failure, OOM, etc.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Restore TSDB from backup
2Restore configuration from version control
3Verify /etc/prometheus is correct
4Start Prometheus
5Verify /-/ready and /-/healthy
6Verify /api/v1/targets is correct
7Verify alert rules are loaded

4 · Verification

Confirm the procedure actually fixed the problem.

✓Prometheus is back
✓All scrape targets are up
✓Alert rules fire correctly
✓No alerts about the platform itself

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶Failure to start: check journalctl -u prometheus
↶Configuration invalid: promtool check config
↶TSDB corruption: restore from backup

6 · Escalation

When the runbook isn't enough, contact:

· Escalate after 1 hour of effort

Purpose

Recover a Broken Prometheus

When to use this runbook

Use this runbook when the operator needs a guided procedure to handle the situation described above.

Pre-checks

Before starting the procedure, confirm the prerequisites and pre-checks are met. The structured lists are rendered from the frontmatter by the page layout.

Procedure

Follow the steps from the frontmatter procedure steps. The page layout renders the steps as a checklist with copy-to-clipboard affordances.

Verification

After the procedure, the structured verification items from the frontmatter are rendered as a checklist.

Rollback

If the procedure fails or makes things worse, follow the structured rollback steps from the frontmatter.

Escalation

The structured escalation path is rendered from the frontmatter. Use it if the operator cannot complete the procedure safely.