critical riskcluster affecting~90 min

Runbook: Recover a Broken Prometheus

1 · Prerequisites

Confirm every item is in place before any state change.

Prometheus Loss
RPO and RTO
Prometheus Backup
Filesystem Layout
systemd Integration
Permissions and Service User
Root on the affected host, and a way in that does not depend on this Prometheus being healthy
The location of the most recent TSDB snapshot, and the Prometheus version it was taken on
The configuration repository, and the commit that describes what this server was running
A recorded RPO and RTO for this server, so the choice between "restore history" and "restore alerting first" is a decision that was already made rather than one improvised at 03:00

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Establish which of four shapes this is before anything is stopped, moved or deleted. They look alike from a dashboard and have completely different recoveries: (1) the process is down, (2) the process is up but not ready, (3) the process is healthy but the data is gone or wrong, (4) the process and data are fine and the configuration is not.
· Read the journal first: journalctl -u prometheus -n 200 --no-pager. It is the cheapest check in the runbook and it usually names the fix outright - opening storage failed, read-only file system, no space left on device, unknown long flag and lock DB directory each point somewhere different.
· Check unit state and whether this is a crash loop: systemctl status prometheus --no-pager. activating (auto-restart) means Restart=on-failure is cycling it, and the real error is in the log excerpt directly underneath rather than in the status line.
· Check whether the process was killed rather than crashed: journalctl -k | grep -i "killed process" or systemctl show prometheus -p Result. An OOM kill is a sizing problem wearing a corruption costume, and restoring a snapshot will not fix it.
· Check the disk before blaming the database: df -h /var/lib/prometheus and findmnt /var/lib/prometheus. A full filesystem and a volume that failed to mount both produce "Prometheus is broken" and neither is repaired by touching the data.
· Confirm nothing else holds the data directory: pgrep -a prometheus and docker ps. Two processes on one data directory is a self-inflicted failure with a specific log line, and the second process is often a debug container somebody started an hour ago.
· If the process is up, ask it what it thinks: curl -sf http://localhost:9090/-/healthy and /-/ready answer different questions. Alive but never ready means the TSDB is still replaying, or failed to open, or the configuration is invalid.
· Locate the snapshot to restore, confirm it is readable, and confirm the Prometheus version it was taken on. A snapshot taken on a newer Prometheus is not guaranteed to be readable by an older one, and discovering that after the data directory has been moved aside is the worst possible order.
· Confirm there is free space to hold the restored data alongside the damaged directory. The damaged directory is not deleted by this procedure; if the volume cannot hold both, that is a decision to make now, deliberately, and not a surprise halfway through a copy.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Declare the shape and write it down. Everything below branches on it. If the evidence does not clearly support one shape, keep gathering rather than guessing - the destructive steps are all further down and none of them get easier by being reached sooner.
2Configuration-only failure: fix and reload, do not restart. If the data is intact and the configuration is bad, promtool check config /etc/prometheus/prometheus.yml names the file and line. Restore the file from the repository and systemctl reload prometheus. A reload re-reads config and rules with no WAL replay and no scrape gap; a restart costs both for nothing.
3Disk-full failure: make space, then start. Do not delete blocks by hand to buy room. Remove any leftover snapshots/ directory first - a forgotten snapshot pins old blocks against retention deletion and is the commonest cause of a volume filling weeks after a change. If that is not enough, grow the volume, or lower --storage.tsdb.retention.time and restart so retention can act.
4Stop the service before touching the data directory: sudo systemctl stop prometheus, then confirm with systemctl is-active prometheus and pgrep -x prometheus. Prometheus holds a lock file and keeps the head in memory; writing into the directory while the process is alive corrupts whatever is left of the head.
5Move the damaged data directory aside. Never delete it. sudo mv /var/lib/prometheus/data /var/lib/prometheus/data.broken.$(date +%Y%m%dT%H%M). It is simultaneously the only copy of everything the snapshot does not contain, the evidence for the post-mortem, and your way back if the restore turns out to be worse.
6Decide explicitly: alerting first, or history first. Starting empty restores scraping, rule evaluation and paging within a minute. Restoring the snapshot restores history but takes as long as the copy takes, during which nothing is being scraped and nothing pages. When in doubt, restore the alert source first and the history second - a platform with no history still protects production, and a platform with no alerting does not.
7Restore the snapshot into a fresh data directory. Create it, unpack the snapshot into it, and check the block inventory with promtool tsdb list before starting anything. A restore that fails is much cheaper to discover from a read-only tool than from a service that will not come up.
8Restore configuration and rules from the repository commit that matches the snapshot, then validate with promtool check config and promtool check rules. A snapshot restored under different rules evaluates a different set of alerts, and the difference is invisible until an alert that used to fire does not.
9Fix ownership and mode before starting: sudo chown -R prometheus:prometheus /var/lib/prometheus and confirm 0750. A restore performed as root leaves root-owned files inside a directory the service user must write to, and the failure surfaces as a permission error mid-startup rather than at the point of the mistake.
10Start the service and follow the journal through WAL replay. systemctl start prometheus, then journalctl -u prometheus -f. On a large restored head this is minutes, not seconds, and /-/ready staying silent during it is expected rather than a second failure.
11Verify against reality, not against the UI: readiness, targets, rules, and a query that returns data from before the incident. Then confirm the alerting chain end to end - /api/v1/alertmanagers lists the Alertmanagers, and only an alert that actually arrives somewhere proves delivery.
12Record the gap and keep the evidence. Write the start and end of the window when this server was not scraping into the incident record. Keep data.broken.* until the post-mortem has finished with it, then remove it deliberately and note that you did.

4 · Verification

Confirm the procedure actually fixed the problem.

✓systemctl is-active prometheus prints active and curl -sf http://localhost:9090/-/ready returns 200. /-/healthy alone is not enough: it says the process exists, not that the TSDB opened.
✓The journal for this start shows the storage path opened, WAL replay completed, and the configuration loaded, with no opening storage failed and no permission error. Read the whole start sequence, not just the last line.
✓curl -s http://localhost:9090/api/v1/targets lists every job from the configuration and every target reports up, or each exception is recorded with a reason. A job missing entirely means the configuration that was restored is not the one this server used to run.
✓curl -s http://localhost:9090/api/v1/rules lists every rule group with health: ok and a recent lastEvaluation. Rules that never load produce no error on any dashboard - the series simply stop existing.
✓A range query that spans the incident returns data from before it: history was actually restored, not merely a directory created. If the answer is empty, the server is running on an empty database and everyone downstream needs to know that.
✓curl -s http://localhost:9090/api/v1/alertmanagers lists every Alertmanager replica, and a test alert has arrived at a real receiver since recovery. "Alertmanager is reachable" and "a page would arrive" are different claims and only the second one matters tonight.
✓ls -ld /var/lib/prometheus and the contents of the data directory are owned by prometheus:prometheus with mode 0750, and df -h shows headroom rather than a volume about to repeat the incident.
✓The data.broken.* directory is still on disk, and its removal is scheduled rather than done. Deleting the evidence before the post-mortem is the second failure of the night, and it is entirely avoidable.

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶Stop the service before any rollback step: sudo systemctl stop prometheus, confirmed with pgrep -x prometheus. Every recovery mistake in this runbook gets worse if the process is running while you correct it.
↶The restore made things worse: the preserved data.broken.* directory is intact. Move the restored directory aside under its own name, move the original back, start, and escalate. This is exactly why step 5 moves rather than deletes.
↶The service will not start after the restore: read the journal before changing anything else. opening storage failed names the block it could not read; permission denied is the ownership step; read-only file system is ProtectSystem=strict without ReadWritePaths.
↶The restored snapshot is itself damaged: promtool tsdb list /var/lib/prometheus/data reports the blocks it can read. Remove the unreadable block directory into a quarantine path and start again - the rest of the history is still usable, and a partial restore beats none.
↶The snapshot is from a newer Prometheus than the binary on this host: install the matching version rather than forcing the data. Downgrading the data is not an option that exists; matching the binary is.
↶Configuration was restored from the wrong commit: revert the repository to the correct one, promtool check config, then systemctl reload prometheus. This one is genuinely reversible and does not need another restart.
↶If recovery has to be abandoned, leave the host stopped with the damaged directory intact, say so explicitly in the incident channel, and hand over. A half-restored Prometheus that is running is worse than a stopped one, because it looks like a working alert source and is not.

6 · Escalation

When the runbook isn't enough, contact:

· Escalate immediately - before diagnosis is finished - if this is the only Prometheus covering targets that page. The estate is currently unmonitored, and that fact needs an owner and a stated interim arrangement rather than waiting for the recovery to succeed.
· Escalate to a second operator before restoring a data directory or moving one aside. Both commands take a path as an argument, both are effectively one-way, and both have been run against the wrong path by a tired person at 03:00.
· Escalate to the storage owner if the volume is full, unmounted, or backed by a network filesystem. A TSDB on NFS or SMB will corrupt again; that is an architecture finding, not tonight incident.
· Escalate to the platform owner if the most recent readable snapshot is older than the recorded RPO. The gap in history is now a business fact that other people need, and it does not become smaller by being discovered late.
· Escalate to the capacity owner if the cause was an OOM kill. Restoring the data changes nothing about the sizing, and the next scrape cycle will reproduce the failure - usually within hours.
· Escalate after one hour without a working alert source, whatever the diagnosis is. At that point the correct move is usually to start an empty Prometheus to restore paging and recover the history as a separate, unhurried piece of work.

The page that brings you here is rarely “Prometheus is down”. It is a Grafana panel that says No data, or an alert that should have fired an hour ago and did not, or a colleague asking why the dashboard stops at 04:11. Four quite different failures produce those same three symptoms, and the recovery for each is different enough that guessing wrong costs you history you did not have to lose.

So the first half of this runbook is diagnosis and the second half is repair, and the boundary between them matters more than anything in either. Every destructive step is on the far side of that line. Nothing above it can make the situation worse.

The four shapes

Shape	What you see	Where the fix lives
Process down	`systemctl status` shows `failed` or `activating (auto-restart)`	The journal, in the first error line of the newest start attempt
Up, never ready	`/-/healthy` answers, `/-/ready` does not	WAL replay in progress, storage failed to open, or invalid config
Healthy, data wrong	Both endpoints answer, queries return nothing	The data directory: deleted, empty, or on the wrong mount
Healthy, config wrong	Both answer, targets or rules are missing	The configuration tree, and the repository it should have come from

They are told apart by two commands and a look at the disk, and the whole cost of skipping that is doing an unnecessary restore. A restore throws away everything since the snapshot; a reload throws away nothing. That is the size of the difference.

Triage, cheapest first

Read-only / Safethe first ninety seconds

# 1. What did it say? This alone resolves most cases.
journalctl -u prometheus -n 200 --no-pager

# 2. Is it running, crash-looping, or stopped?
systemctl status prometheus --no-pager

# 3. Was it killed rather than crashed?
journalctl -k | grep -i 'killed process'
systemctl show prometheus -p Result -p NRestarts

# 4. Is this a disk problem wearing a database costume?
df -h /var/lib/prometheus
findmnt /var/lib/prometheus

# 5. Is something else holding the data directory?
pgrep -a prometheus
docker ps --filter ancestor=prom/prometheus

Read the journal before forming an opinion. Prometheus is unusually good at saying what went wrong in one line, and the five lines it uses are each a different runbook branch:

opening storage failed: lock DB directory - another process holds the data directory. Check 5 finds it. Nothing is corrupted; you have two Prometheus.
read-only file system naming the data path - ProtectSystem=strict without a matching ReadWritePaths. The data is fine; the unit is wrong.
no space left on device - the disk, not the database. Check 4.
unknown long flag - the binary does not know a flag in ExecStart. Usually a half-applied upgrade or a downgrade nobody recorded.
A parse error naming a config or rule file with a line number - configuration only, and the cheapest shape of all to fix.

Check 3 deserves its own mention because it is the one that gets misdiagnosed most. An OOM kill leaves a data directory that needs WAL replay and looks exactly like corruption. Restoring a snapshot “fixes” it for as long as it takes the head to grow back, which on a busy server is a couple of hours. The actual fix is memory or cardinality, and that is a different runbook.

The configuration-only branch: reload, do not restart

Configuration changefix configuration without a restart

promtool check config /etc/prometheus/prometheus.yml
promtool check rules /etc/prometheus/rules/*.yml

# Restore from the repository rather than editing in place.
sudo install -o root -g prometheus -m 0640 \
/srv/monitoring-config/prometheus.yml /etc/prometheus/prometheus.yml

sudo systemctl reload prometheus
journalctl -u prometheus -n 20 --no-pager

If the process is running and only the configuration is wrong, this is the whole recovery. A reload re-reads prometheus.yml and every matched rule file and rebuilds the scrape pools without restarting: no WAL replay, no readiness gap, no hole in any series.

There is one asymmetry worth carrying around. A bad config on reload is not fatal - Prometheus logs the error and keeps running the previous configuration. A bad config on restart is fatal, because the process exits at load. That is why promtool check config belongs in front of every restart and why reload is always the first thing to try.

The restore branch

Data-loss riskstop, preserve, restore

# 1. Stop, and prove it stopped. The lock is held until the process exits.
sudo systemctl stop prometheus
systemctl is-active prometheus
pgrep -x prometheus                # expect no output

# 2. Preserve. This directory is evidence and it is your way back.
sudo mv /var/lib/prometheus/data \
"/var/lib/prometheus/data.broken.$(date +%Y%m%dT%H%M)"

# 3. Restore into a fresh directory.
sudo install -d -o prometheus -g prometheus -m 0750 /var/lib/prometheus/data
sudo tar -C /var/lib/prometheus/data -xzf /srv/restore/prom-snapshot.tar.gz --strip-components=1

# 4. Inspect before starting. A read-only tool is a cheaper place to fail.
sudo promtool tsdb list /var/lib/prometheus/data

# 5. Ownership last, and recursively. A root-owned restore does not start.
sudo chown -R prometheus:prometheus /var/lib/prometheus
sudo chmod 0750 /var/lib/prometheus

Step 4 is the step people skip and the one that saves the night. promtool tsdb list walks the block inventory without a running server, so a snapshot that was already damaged when it was taken - the classic outcome of a backup job that tarred a live data directory instead of calling the snapshot API - fails here, in a read-only command, rather than as a service that will not come up after you have already moved the original aside.

Step 5 is the commonest self-inflicted second failure. The restore runs as root, every restored file is owned by root, and the service user cannot write into its own database. The symptom is a permission error partway through startup, which reads like corruption and is not.

Alerting first, history second

This is the decision that most distinguishes a good recovery from a slow one, and it is a decision, so make it out loud.

Restoring a snapshot restores the history. It also takes as long as the copy takes - tens of minutes on a large TSDB - and during all of it this server is not scraping, not evaluating rules, and not able to page anyone. Starting empty takes about a minute and gets scraping, rule evaluation and alerting back immediately, at the cost of an empty graph until the restore is done separately or the history is simply accepted as lost.

Service impact possiblerestore the alert source now, the history later

sudo install -d -o prometheus -g prometheus -m 0750 /var/lib/prometheus/data
sudo systemctl start prometheus

curl -sf http://localhost:9090/-/ready
curl -s http://localhost:9090/api/v1/targets \
| jq -r '[.data.activeTargets[] | select(.health != "up")] | length'

The rule of thumb: an estate with no history still protects production; an estate with no alerting does not. If a long restore is standing between production and any alerting at all, start empty, say so in the incident channel, and treat the history as a separate piece of work done without a clock running.

Two caveats keep this honest. If a remote-write backend holds the same series, history is not actually lost - it is queryable from the backend, and starting empty costs much less than it appears to. And if this Prometheus is one half of an HA pair, its partner has been covering alerting the whole time, which changes the urgency completely and makes the careful restore the right call.

Start, and follow the replay

Service impact possiblestart and watch

sudo systemctl start prometheus
journalctl -u prometheus -f

WAL replay on a restored head takes as long as it takes, and /-/ready staying silent during it is the expected behaviour rather than a second failure. Watch for the replay completing and the configuration loading; those two lines together are the point at which the recovery is real.

If the server starts by hand as the service user but not under systemd, the problem is the unit - sandboxing, limits or identity - and not the data:

Read-only / Safeisolate systemd from Prometheus

sudo -u prometheus /usr/local/bin/prometheus \
--config.file=/etc/prometheus/prometheus.yml \
--storage.tsdb.path=/var/lib/prometheus/data

On RHEL with SELinux enforcing, a data directory restored to a path without a matching file-context rule produces denials that look exactly like Unix permission errors. Check ls -lZ on the directory and ausearch -m avc -ts recent before spending an hour on modes that are already correct.

Where you stop and think

The evidence does not clearly support one shape. Keep gathering. Every destructive step is below this point and none of them get cheaper by being reached sooner.

The most recent readable snapshot is older than the RPO. That is now a business fact, not an operational one. It needs escalating while the recovery proceeds, not afterwards, because other people are about to make decisions based on data that will not exist.

The cause was an OOM kill or a full disk. Restoring changes nothing about the cause, and the failure will repeat - within hours for memory, within days for disk. Recover, then hand the cause to the capacity or storage owner as a separate item with its own owner.

An hour has gone by with no working alert source. Stop optimising for history. Start empty, restore paging, and recover the data as unhurried work.

Common patterns

Symptom	Likely cause	Resolution
`opening storage failed: lock DB directory`	A second Prometheus holds the data directory	`pgrep -a prometheus`, `docker ps`; stop the extra one. Nothing is corrupt
`read-only file system` on the data path	`ProtectSystem=strict` without `ReadWritePaths`	Fix the unit, `daemon-reload`, restart. The data is fine
Starts, then killed repeatedly	OOM. `journalctl -k` shows the kill	Sizing or cardinality; a restore only postpones it
Queries empty, targets all `up`	Data directory was wiped or the volume did not mount	`findmnt`; if unmounted, the writes went to `/`
Permission denied partway through start	Restore ran as root; files not owned by `prometheus`	`chown -R prometheus:prometheus`, then start
`promtool tsdb list` fails on the restore	The backup tarred a live directory instead of using the snapshot API	Use an older snapshot; fix the backup job as a separate item
Rules missing after a successful restore	Configuration restored from a different commit than the snapshot	Check out the matching commit, `promtool check`, reload
Volume fills again days later	A `snapshots/` directory was left in place, pinning old blocks	Remove it once the off-host copy is confirmed

Why this matters

The reason this runbook spends more space on diagnosis than on repair is that the repairs are short and the wrong repair is expensive. A reload costs nothing. A restore costs everything between the snapshot and now, permanently, and it is entirely possible to run one for a problem that was a bad line in a YAML file.

The second reason is that the pressure runs the wrong way. This is the system that tells you whether everything else is healthy, so while it is down the whole estate is flying blind and every instinct says move faster. That is precisely the state in which people delete a data directory to “start clean”, and lose both the newest hours of history and the only evidence of what happened.

Moving rather than deleting costs one mv and no time at all. It is the single highest-value habit in this procedure, and it is also the one most likely to be skipped at 03:00 by somebody who is sure they know what went wrong. The whole structure here - read-only triage first, an explicit line before anything destructive, preserve before restore, and alerting ahead of history - exists to make the fast wrong move harder than the correct one.