medium riskservice affecting~120 min

Runbook: Upgrade Prometheus

1 · Prerequisites

Confirm every item is in place before any state change.

Prometheus Upgrade Basics
TSDB Migration
Prometheus Config Compatibility
Prometheus Rule Compatibility
Rollout Validation
Reading Release Notes
Rollback Strategy
The current version read from the running process (/api/v1/status/buildinfo) and the target version as an exact string - the package manager and the wiki are both hearsay
The release notes for every version between the two, not only the target
Somewhere off this host to put a TSDB snapshot, and enough free space on the data volume to create one
A witness outside this server for the window - a blackbox probe, a dead-man switch, or a named person - because the thing being restarted is the thing that would otherwise tell you it broke

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· The running version comes from the process, not from the host: curl -fsS http://localhost:9090/api/v1/status/buildinfo | jq -r .data.version. An estate where the binary on disk and the running process disagree already has a half-applied upgrade, and this runbook is not the fix for that.
· The release notes for every version being crossed have been read. A server three minors behind meets three sets of breaking changes in one window, and the removed configuration field is usually in the middle one rather than the newest.
· The new tarball is downloaded and its digest verified against the published sha256sums.txt before the window opens. Verifying an artefact is not an emergency task and does not belong inside a change window.
· The configuration and the rule files parse under the promtool that shipped with the new release: promtool check config and promtool check rules, run from the unpacked new tarball. The currently installed promtool agrees with the currently installed binary and tells you nothing about the next one.
· Rule behaviour, not only rule syntax, has been checked where tests exist: promtool test rules from the new release replays synthetic samples and compares against the expected series. A rule can parse cleanly under the new binary and still produce different output.
· A baseline has been captured to a file: version, count by (job) (up), active target count, rule group count, prometheus_tsdb_head_series, and the Alertmanager list from /api/v1/alertmanagers. Every verification step below is a comparison against that file, and a number nobody wrote down cannot be compared.
· The data volume has free space for a snapshot plus normal compaction scratch. The snapshot is created by hard-linking blocks, so it costs almost nothing at creation - but it holds those blocks against deletion until it is removed, and the copy you take off-host costs their full size.
· Alertmanager is already on a version the target Prometheus supports. Alertmanager moves before Prometheus, never after: the side that has to parse the newer request goes first. If both need to move, that is the full-stack upgrade runbook, not this one.
· The window has an owner, an end time, and a decision recorded about the scrape gap: every restart in this procedure puts a hole in every series, and somebody will ask later which minutes are missing.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Capture the baseline to a file and confirm the configuration on disk matches the repository. curl -fsS http://localhost:9090/api/v1/status/config prints what the process actually loaded. An upgrade that lands on undeclared configuration has a rollback that restores the wrong file.
2Unpack the new release beside the running one and validate with its promtool, not the installed one: promtool check config /etc/prometheus/prometheus.yml and promtool check rules /etc/prometheus/rules/*.yml. Any failure here is a configuration change that must merge and be reviewed before the binary moves - not during.
3Take the TSDB snapshot with curl -XPOST http://localhost:9090/api/v1/admin/tsdb/snapshot (requires --web.enable-admin-api). It lands under snapshots/ inside the data directory and is a hard-linked, consistent view - cheap to create and useless as a rollback artefact until it is copied somewhere else.
4Copy the snapshot off this host. A hard-linked snapshot on the same volume survives a bad upgrade; it does not survive the volume. Until the copy lands elsewhere and its size has been checked, this procedure has no rollback and should not proceed.
5Preserve the outgoing binary under a versioned name: sudo cp -a /usr/local/bin/prometheus /usr/local/bin/prometheus.OLDVERSION. Rollback then costs a file copy and a restart rather than a download from a host whose monitoring is currently down.
6Stop the service and let it stop cleanly: sudo systemctl stop prometheus, then confirm with systemctl is-active prometheus and pgrep -x prometheus. On SIGTERM Prometheus flushes the head and exits; a kill -9 here leaves the whole write-ahead log to replay on the next start, which is the difference between a short restart and a long one.
7Install the new binaries with install -o root -g root -m 0755 over /usr/local/bin/prometheus and /usr/local/bin/promtool. Both, always: a new server with an old promtool is a validation tool that quietly disagrees with the thing it is validating.
8Start the service and watch the journal through WAL replay, not just the exit code of systemctl start. /-/healthy answers as soon as the process is alive; /-/ready only answers once the TSDB is open and the configuration is valid, and on a large head that gap is minutes.
9Run the version and flags check first. /api/v1/status/buildinfo must report the target version and /api/v1/status/flags must still report the paths and retention you expect. A flag that a new release renamed or removed shows up as a failure to start, but a flag whose default changed shows up only here.
10Walk the five validation surfaces against the baseline file: targets, rules, alerting chain, remote write if configured, and one dashboard per data source. Each has its own pass criterion; "the UI loads" is not one of them.
11Hold before declaring done. The gate proves the process started; the hold proves it survives a compaction cycle, a full scrape rotation and a rule evaluation cycle. Most slow-path regressions are invisible for ten minutes and obvious by the fourth hour.
12Close the change: record the new version, the snapshot location and its expiry, the exact gap window with start and end times, and remove the on-host snapshot directory once the off-host copy is confirmed.

4 · Verification

Confirm the procedure actually fixed the problem.

✓/api/v1/status/buildinfo reports the target version, and prometheus --version on the host agrees with it. Two different answers means the process is still the old binary and the restart did not take.
✓/-/ready returns 200 and the journal shows WAL replay completed with no storage error. Readiness after an upgrade is the claim that the new binary opened the existing data directory successfully, which is the single most important thing this procedure needs to be true.
✓/api/v1/status/flags reports the same config path, storage path and retention values as the baseline file. A retention default that changed between releases silently deletes history, and this is the only check that catches it before the deletion happens.
✓The active target count matches the baseline and count by (job) (up) returns the same jobs with the same counts. A count that is lower but still plausible is exactly the failure this runbook exists to catch, because nothing about it looks like an error.
✓/api/v1/rules lists every rule group from the baseline, each with health: ok and a lastEvaluation timestamp inside the last evaluation interval. A rule with health: err carries its lastError in the same response - read it rather than the log.
✓prometheus_rule_group_iterations_missed_total is flat rather than climbing, and prometheus_tsdb_head_series is within normal variation of the baseline. A head series count that dropped sharply means targets stopped being scraped, not that the upgrade tidied anything up.
✓/api/v1/alertmanagers lists every Alertmanager replica the baseline named. One entry where there should be two means half the alerting chain is silently not receiving, and nothing else on the page will say so.
✓A query that spans the restart returns data from both sides of it, with a gap no longer than the restart plus WAL replay. If the gap is longer, the server was not scraping for a period nobody has accounted for yet.
✓The off-host snapshot copy still exists and its size is plausible, and the on-host snapshots/ directory has been removed. A snapshot left in place holds old blocks against retention deletion and will quietly consume the volume.

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶Decide first whether this is a rollback or a fix-forward. A configuration or rule problem is usually fix-forward: correct the file, promtool check, systemctl reload. A binary that will not start, or one that starts and behaves wrongly, is a rollback.
↶Stop the service cleanly before touching anything: sudo systemctl stop prometheus, then confirm with pgrep -x prometheus. Never move or replace files in the data directory while the process holds the lock on it.
↶Restore the binary: sudo install -o root -g root -m 0755 /usr/local/bin/prometheus.OLDVERSION /usr/local/bin/prometheus, and restore the matching promtool the same way. Start, then check /-/ready and the journal.
↶If the old binary starts cleanly on the existing data directory, the rollback is done and no data was lost. This is the common case for a minor upgrade caught within the first minutes, before the new version has compacted anything.
↶If the old binary refuses to open the data directory - the journal says opening storage failed - the new version has already written something the old one cannot read. Move the data directory aside rather than deleting it (mv /var/lib/prometheus/data /var/lib/prometheus/data.postupgrade.$(date +%Y%m%dT%H%M)), restore the snapshot copy into a fresh directory, fix ownership, and start.
↶Everything between the snapshot and the rollback is gone from local storage. That window is the real cost of a rollback and the real argument for taking the snapshot as late as possible before the change rather than as part of last night backup.
↶Configuration and rules roll back by reverting the repository to the tagged commit and reloading. That is the only genuinely symmetrical part of this procedure; the binary and the data are recovery rather than reversal.
↶Re-run the full verification list after any rollback. An unvalidated rollback is just another untested change, made faster and under more pressure.

6 · Escalation

When the runbook isn't enough, contact:

· Escalate to the platform owner if the release notes describe a migration, a format change or a removed flag on any version being crossed. That is a project with its own window and its own approval, not a step inside a version bump.
· Escalate to the service owners before starting if this server is the only Prometheus covering targets that page. The restart is a period with no scraping and no alert evaluation, and whether that is acceptable is a business decision rather than an operator one.
· Escalate to a second operator before restoring a data directory. The command takes a path as an argument, it is one-way, and it has been run at 03:00 against the wrong one.
· Escalate to the storage owner if the volume cannot hold a snapshot alongside normal compaction scratch. Do not proceed without the snapshot: an upgrade with no rollback artefact is a one-way change, whatever the change record calls it.
· Escalate to security if the release being applied is a CVE fix. The window is then bounded by the exposure rather than by convenience, and "hold on the current version" may not be one of the available outcomes.
· Escalate to the consuming teams if the server is held mid-window on a version nobody planned for. A half-upgraded estate is a supported state only while a named person owns it and knows it is half-upgraded.

Swapping the Prometheus binary takes about four seconds. Everything else in this runbook exists because of what is underneath it: a data directory the new version is about to open, a set of rules the new version is about to parse, and a scrape loop that is the only reason anyone knows whether the rest of the estate is healthy.

The asymmetry that shapes every decision here is simple. The binary is trivial to put back. The data directory is not. Once the new version has opened it and written to it, “roll back the package” may no longer be a thing you can do, and the fallback is a restore. So the whole procedure is arranged around one question: at what point does this stop being reversible, and what did I capture before that point.

What this runbook is, and what it is not

It is a single Prometheus server, moving one version step, on a host built by the install runbook: tarball binaries in /usr/local/bin, configuration in /etc/prometheus from version control, data on its own filesystem, systemd holding it all together.

It is not the stack. Alertmanager, exporters, Loki, Tempo and Grafana each move on their own cadence and the order between them matters; that sequencing is the full-stack upgrade runbook, and this one deliberately assumes Alertmanager has already moved if it needed to. It is also not a migration: if the release notes describe a format change or a removed flag, that is a separate project with its own approval, not a step inside a version bump.

Blast radius, by step

Read this before the window. It is the line between the steps you can undo and the steps you can only recover from.

Step	Reversible?	What it costs if it goes wrong
Validate config and rules with the new promtool	Yes	Nothing. This is the point of doing it early
Take the snapshot	Yes	Disk: the snapshot pins old blocks against retention deletion
Copy the snapshot off-host	Yes	Time and network. Skipping it is what makes everything below one-way
Preserve the outgoing binary	Yes	Nothing. One file
Stop the service	Yes	The gap starts here. No scraping, no rule evaluation, no alerting
Install the new binary	Yes	A file copy back, while the service is still stopped
Start the new version	Not cleanly	The new binary opens and may write to the existing data directory
First compaction under the new version	No	A block in a format the previous binary may refuse to read

The bold row is the one that matters. Everything above it is a file operation. Everything from there on has your live data underneath it.

Step 1: Baseline, to a file

Read-only / Safecapture the pre-upgrade baseline

PROM=http://localhost:9090
OUT=/var/tmp/prom-upgrade-baseline
mkdir -p "$OUT"

curl -fsS "$PROM/api/v1/status/buildinfo" | jq -r '.data.version' > "$OUT/version.txt"
curl -fsS "$PROM/api/v1/status/flags"     | jq '.data'            > "$OUT/flags.json"
curl -fsS "$PROM/api/v1/status/config"    | jq -r '.data.yaml'    > "$OUT/config.yml"
curl -fsS "$PROM/api/v1/targets"          | jq '[.data.activeTargets[]
| {job: .labels.job, health}] | group_by(.job)
| map({job: .[0].job, n: length})'                              > "$OUT/targets.json"
curl -fsS "$PROM/api/v1/rules"            | jq '[.data.groups[]
| {name, rules: (.rules | length)}]'                            > "$OUT/rules.json"
curl -fsS "$PROM/api/v1/alertmanagers"    | jq '.data'            > "$OUT/alertmanagers.json"

Write it to a file, not to a terminal you will scroll back through. Every check after the restart is a comparison against these numbers, and the regression this procedure most often catches is not an error - it is a count that came back slightly lower and entirely plausible.

status/config deserves its own note: it prints the configuration the process loaded, not the file on disk. Diff it against the repository now. An upgrade that lands on undeclared configuration has a rollback that restores the wrong file, and you will not discover that until the rollback fails.

Step 2: Validate with the new promtool

Read-only / Safevalidate against the incoming release

VER=2.55.1
tar xzf "prometheus-$VER.linux-amd64.tar.gz"
NEW="./prometheus-$VER.linux-amd64"

"$NEW/promtool" check config /etc/prometheus/prometheus.yml
"$NEW/promtool" check rules /etc/prometheus/rules/*.yml

# Where rule test files exist, this is the behavioural check, not just syntax.
"$NEW/promtool" test rules /etc/prometheus/rules/tests/*.yml

The promtool already installed on this host agrees with the binary already installed on this host. That tells you the current state is consistent, which you knew. The question in front of you is whether the next binary accepts this configuration, and only the promtool that shipped alongside it can answer.

check and test answer different questions and both are worth running. check rules parses: it catches a field that was deprecated three releases ago and removed in this one, which fails as unknown field at startup and takes the whole server down with it. test rules replays synthetic samples through the rule and compares the output against the expected series: it catches a rule that parses perfectly and now produces something different. The second failure is far worse, because the server starts, the dashboards render, and the numbers are wrong.

Steps 3 to 5: Build the rollback before you need it

Service impact possiblesnapshot, copy off-host, preserve the binary

PROM=http://localhost:9090
DATA=/var/lib/prometheus/data

# Consistent, hard-linked snapshot. Needs --web.enable-admin-api.
# It lands in a snapshots/ directory inside the TSDB path, on the data volume.
SNAP=$(curl -fsS -XPOST "$PROM/api/v1/admin/tsdb/snapshot" | jq -r '.data.name')
echo "snapshot: $SNAP"
du -sh "$DATA/snapshots/$SNAP"

# The step that turns it into a rollback artefact.
tar -C "$DATA/snapshots" -czf "/var/tmp/$SNAP.tar.gz" "$SNAP"
scp "/var/tmp/$SNAP.tar.gz" backup-host:/srv/backup/prometheus/
ssh backup-host ls -l "/srv/backup/prometheus/$SNAP.tar.gz"

# Keep the outgoing binary. Rollback should be a copy, not a download.
OLD=$(prometheus --version 2>&1 | head -1 | awk '{print $3}')
sudo cp -a /usr/local/bin/prometheus "/usr/local/bin/prometheus.$OLD"
sudo cp -a /usr/local/bin/promtool   "/usr/local/bin/promtool.$OLD"

The snapshot call briefly pauses head writes while it creates the links, then returns a directory name. It is consistent by construction, which is the whole reason to use it rather than tar over a live data directory - a tar of blocks mid-compaction restores into a Prometheus that will not start.

Take the snapshot as late as you can before the change. Everything between the snapshot and a rollback is gone from local storage, so a snapshot taken at the top of the window costs you less history than last night backup does.

Preserving the outgoing binary is thirty seconds of work that changes the character of the rollback completely. Without it, rolling back at 03:00 means downloading and verifying a tarball from a host whose monitoring is currently down, on a network you are no longer able to observe.

Steps 6 to 8: The change itself

Service impact possiblestop, install, start

sudo systemctl stop prometheus
systemctl is-active prometheus
pgrep -x prometheus          # expect no output

VER=2.55.1
NEW="./prometheus-$VER.linux-amd64"
sudo install -o root -g root -m 0755 "$NEW/prometheus" /usr/local/bin/prometheus
sudo install -o root -g root -m 0755 "$NEW/promtool"   /usr/local/bin/promtool

sudo systemctl start prometheus
journalctl -u prometheus -f

Let it stop cleanly. On SIGTERM, Prometheus compacts the head, flushes and exits, which is why a controlled stop restarts quickly. A kill -9 because the stop “was taking too long” leaves the entire write-ahead log to replay on the next start, and on a large server that turns a thirty-second gap into several minutes - during the one window where you most want the server back.

Install both binaries. A new server with an old promtool is a validation tool that quietly disagrees with the thing it validates, and the disagreement surfaces during the next incident rather than during this change.

Then watch the journal rather than the exit code. systemctl start returning 0 means systemd forked a process. The interesting lines come after: the storage path it opened, the WAL replay progress, the configuration load, and finally that it is ready to receive web requests.

Steps 9 and 10: Validate against the baseline

Read-only / Safepost-upgrade validation

PROM=http://localhost:9090
OUT=/var/tmp/prom-upgrade-baseline

# 1. The process is the new version, and it is past WAL replay.
curl -fsS "$PROM/api/v1/status/buildinfo" | jq -r '.data.version'
curl -fsS "$PROM/-/ready"

# 2. Flags survived the upgrade - including the ones you never set.
diff <(curl -fsS "$PROM/api/v1/status/flags" | jq '.data') "$OUT/flags.json"

# 3. Targets: same jobs, same counts.
curl -fsS "$PROM/api/v1/targets" | jq '[.data.activeTargets[]
| {job: .labels.job, health}] | group_by(.job)
| map({job: .[0].job, n: length})'

# 4. Rules: every group present, every rule healthy.
curl -fsS "$PROM/api/v1/rules" | jq -r '.data.groups[].rules[]
| select(.health != "ok") | "\(.name) \(.health) \(.lastError)"'

# 5. The alerting chain still has both ends.
curl -fsS "$PROM/api/v1/alertmanagers" | jq '.data.activeAlertmanagers | length'

Check 2 is the one that catches what nothing else does. A flag the release removed stops the server, which is loud. A flag whose default changed while you never set it explicitly is silent, and if it happens to be a retention default, the consequence is history being deleted on a schedule nobody chose. Diffing the flag dump against the baseline is a second of work.

Check 4 reads health and lastError straight out of the API rather than grepping the journal. A rule that fails to parse under the new binary reports its error there on every evaluation, and the series it used to produce simply stops existing - which downstream looks like an empty dashboard panel with no error anywhere near it.

Step 11: Hold

The gate proves the process started. The hold proves it survives the slow paths: a compaction cycle, a full scrape rotation, a complete rule evaluation interval, and enough time for prometheus_rule_group_iterations_missed_total to start climbing if evaluation is now too slow for the interval.

Most upgrade regressions worth catching are invisible in the first ten minutes. If the estate has more than one Prometheus, hold this one as a canary through a real compaction before touching the second.

Where you stop and think

The release notes describe a migration or a removed flag. Stop. The configuration change has to merge, be reviewed and pass the new promtool before the binary moves, because the new binary will refuse to start on the old file. Finding this at the restart converts a version bump into an outage with a config change in the middle of it.

The snapshot cannot be copied off-host. Stop. Without that copy the change is one-way and the change record is describing something other than what is about to happen. This is worth saying out loud rather than proceeding quietly.

A rule fails test rules but passes check rules. That is the behavioural regression the tests exist to find. Fix the expression before the window, not after - and be suspicious of the urge to delete the failing test.

The gate passed but the hold is not clean. The correct response is to roll this server back, not to reason about whether it would settle. A canary that passed every immediate check and degraded at hour four has told you something true about the new version.

Hold is a first-class outcome

Stopping half-way is the runbook working, but it is a decision with the same paperwork as proceeding: a named owner for the mixed-version estate, an end time by which this server is either finished or returned to the previous version, and a written note of which version is actually live so the next incident is not diagnosed against an assumption.

Common patterns

Symptom	Likely cause	Resolution
Service will not start; journal says `unknown long flag`	A flag was removed in a version being crossed	Read the release note for the middle version, not just the target
Service will not start; `opening storage failed`	The data directory is held, or the new version rejects it	Check `pgrep -x prometheus` first; the lock is the common case
Starts, `/-/healthy` OK, `/-/ready` slow for minutes	WAL replay after an unclean stop	Expected. Wait, and stop cleanly next time
Target count slightly lower, no errors	A scrape job stopped parsing after a config or exporter change	Diff `/api/v1/status/config` against the baseline
A recording rule stopped producing rows	Stricter parsing, or a changed default	`/api/v1/rules` reports `health` and `lastError` per rule
History quietly shortening after the upgrade	A retention default changed and was never set explicitly	Diff `/api/v1/status/flags` against the baseline; set both retention limits
Old binary refuses the data directory on rollback	The new version already wrote a newer on-disk format	Restore the snapshot copy into a fresh data directory
Disk fills days after a successful upgrade	The snapshot was never removed and pins old blocks	Remove `snapshots/` once the off-host copy is confirmed

Why this matters

Almost everything in this runbook is preparation, and that ratio is the point. The irreversible part of a Prometheus upgrade is one command - starting the new binary against the existing data directory - and it takes a fraction of a second. Everything before it exists so that the second after it is survivable.

The two failures that make this expensive are both failures of preparation rather than of execution. The first is the snapshot that was never taken, or was taken onto the same volume that later died, or was taken by cron three weeks ago and has been silently failing since: the rollback path was a belief rather than an artefact. The second is the validation that was run with the wrong tool - the installed promtool, agreeing with the installed binary, confirming a state nobody was asking about.

And the specific reason this matters more for Prometheus than for most services is that Prometheus is the thing that would tell you. While it is restarting, every dashboard is flat and every alert is quiet, and both of those look identical to a healthy estate. The witness outside the stack, the baseline written to a file, and the gap recorded with real timestamps are what let you tell the difference afterwards.