The release notes have been read starting with security, then breaking changes, then deprecations, then anything touching schema or storage — before the new features. That ordering is not tidiness: the first four sections decide whether this upgrade is reversible and what it costs, and the section people read first is the one that decides nothing. A note describing a storage change as a performance improvement is a migration wearing a friendly label.
If you are skipping versions, the notes for every release in between have been read, not only the target's. Breaking changes and removals are announced in the release that makes them and are not repeated afterwards, so a two-minor jump silently contains everything the intermediate release said. This is the single most common way a documented breaking change reaches production undocumented. curl -fsS https://api.github.com/repos/prometheus/prometheus/releases | jq -r '.[] | "\(.tag_name) \(.published_at)"' | head -20The project's security advisories have been checked as their own source, not inferred from the changelog. Advisories follow a different disclosure timeline and are sometimes described in a changelog entry without being flagged as security at all. A fix you did not know was a fix changes the priority of the whole upgrade, and finding that out afterwards means you were exposed for the interval.
Every deprecation in the notes has been matched against what your configuration and rules actually use. A deprecated setting still works in this release and disappears in a later one, which means this upgrade is the cheap moment to remove it and the next one is where it becomes an outage. Deprecation warnings are a schedule, and the only way to use the schedule is to know which entries apply to you.
The running version of every component on every wire has been written down, including the ones you are not upgrading. Compatibility is a property of a pair, not of a release: Grafana against each datasource, Prometheus against Alertmanager, the collectors against every backend they write to. You cannot check a pairing you have not enumerated, and the version people believe is deployed is regularly not the one that is. curl -fsS http://prometheus.internal:9090/api/v1/status/buildinfo | jq -r '.data.version'; curl -fsS http://alertmanager.internal:9093/api/v2/status | jq -r '.versionInfo.version'; curl -fsS -u "$GRAFANA_AUTH" http://grafana.internal:3000/api/health | jq -r '.version'For every pair that changes, the new version is documented as supported against the version on the other side. Supported means named in one project's notes, not inferred from both being current — the components move on independent cadences and a query API or datasource plugin that expects the newer response shape returns an empty result rather than an error against the older one. Empty is the failure mode, which is why this is worth checking rather than discovering.
Grafana plugins, collector distributions and any other pluggable component have been checked against the target version. A plugin is compiled against an SDK and declares only a minimum host version, so a host upgrade can leave it unable to load with nothing to warn you in advance. The panels that depend on it then break while every other panel works, which reads as a dashboard problem rather than an upgrade problem. grafana cli plugins lsThe existing configuration has been checked with the tooling from the version you are moving to, not the one installed on the host. This is the whole point: the current binary accepts the current config by definition. Running the new release's checker against the old file is the cheapest possible preview of what the upgrade will make of it, and it exits non-zero, so it belongs in CI rather than in someone's memory. docker run --rm -v /etc/prometheus:/etc/prometheus:ro prom/prometheus:v2.55.0 promtool check config /etc/prometheus/prometheus.ymlEvery rule file parses under the new binary and the rule unit tests still pass under it. Parsing catches a removed function; only the tests catch a function whose behaviour changed — an aggregation that now returns a different value for an empty input, for instance, which parses perfectly and quietly changes what your alerts mean. Rules that have no tests are rules whose semantics you are upgrading blind. docker run --rm -v /etc/prometheus:/etc/prometheus:ro prom/prometheus:v2.55.0 promtool test rules /etc/prometheus/tests/*.ymlThe environment you rehearse in matches production in the dimensions this upgrade touches. A staging Grafana on a different database backend proves nothing about a schema migration; a staging Prometheus with a thousand series proves nothing about a replay of forty million. Naming the dimension the upgrade actually stresses, and matching that one, is worth more than matching everything else.
The full sequence has been executed in that environment end to end, and so has the reversal. A rehearsal that stops at the upgrade validates the half you were already fairly confident about. The reversal is the half that will be attempted under pressure by somebody who has never done it, and the rehearsal is the only place it is cheap to find out that it does not work.
The order the components will be touched in is written down and follows the direction of the wire contracts. Upgrade the destination before the sender: notification receivers, then Alertmanager, then Prometheus, then the exporters, then the dashboards. Wrong order leaves a newer sender speaking to an older receiver for the length of the rollout, and the resulting incident is fan-out shaped — pages that never arrive, or arrive twice.
How long the estate will run mixed versions is a number somebody chose, especially where clustered components gossip with each other. A ring holding two versions that disagree on the wire format splits, and a split ring stops deduplicating and stops applying silences uniformly. "As long as the rollout takes" is not a bound; it is what the bound turns into when nobody sets one.
A snapshot exists for every component that holds state, taken through the mechanism the project supports rather than by copying a data directory under load. A copy taken while the process is writing is a copy of a moment that never existed, and it will restore, start, and be subtly wrong. Record where the snapshot is and how long it takes to restore, because both are inputs to the rollback decision. curl -fsS -XPOST http://prometheus.internal:9090/api/v1/admin/tsdb/snapshot | jq -r '.data.name'Each snapshot is stored alongside the exact version that produced it, and that version's package is still obtainable. On-disk formats are not portable across versions in both directions, so a snapshot without its matching binary is an archive you cannot open. The moment you need this is the moment the package cache has been cleaned and the release has been pulled.
Grafana's database has a fresh backup, and it has been inspected for the data a new constraint could reject. A Grafana upgrade is a database migration first and a binary change second: the migration runs automatically at start, and if existing rows violate a constraint the new schema introduces, the process refuses to start and the schema is left half-migrated — which blocks the rollback as well as the upgrade. sqlite3 /var/lib/grafana/grafana.db ".backup '/var/backups/grafana/grafana-pre-upgrade.db'"The buckets behind the log and trace stores have versioning on, or a copy somewhere else. Schema, index and block-format changes are the class of upgrade that can make existing objects unreadable or delete them outright, and the object store is the only layer where that is recoverable. This is cheap to turn on now and impossible to turn on retroactively. aws s3api get-bucket-versioning --bucket prod-loki-chunks --profile loki-storageThe specific moments after which rollback stops being possible are named in the plan. The first block written in a new on-disk format, a completed database schema migration, an appended schema entry that is now the active one: each of these is a point after which the previous binary cannot read the current state. Knowing where they are converts "can we roll back" from a debate at 03:00 into a lookup.
The reversal is a checked-in document with the exact commands in order, written for an engineer who did not perform the upgrade and is reading it at 03:00. It lives next to the upgrade plan and is written before the upgrade, because a rollback improvised under pressure is where teams discover what their on-disk format actually was. A wiki page nobody can find is not this.
The runbook says how you will know the rollback worked, not only how to perform it. A partial reversal — the binary rolled back but not the config, or the config but not the rules — produces a half-old, half-new stack that no release note covers and no health check objects to. The validation is what distinguishes a completed rollback from one that merely appeared to complete.
The conditions that trigger an immediate revert are quantitative and agreed before the change starts. A named threshold on active series, on rule evaluation, on notification delivery, means the decision is a reading rather than a judgement made by a tired person who has just spent two hours on the upgrade and would prefer it to have worked. Deciding this in advance is the point.
The canary host is the one whose workload looks most like the fleet's, not the one whose failure would matter least. Those are usually different hosts, and picking the second gives you a green canary that exercised none of the paths the fleet runs — an idle instance will not compact, will not replay anything large, and will not meet the cardinality that breaks things.
The canary must pass named checks and stay healthy for a stated period before the fleet follows, and the gates map to the risks the release notes actually raised. The window has to be long enough to include the slow paths — compaction, replay, retention enforcement — because those run on their own schedules and are exactly where a regression hides from a five-minute check.
Target count, active series, ingestion rate, query latency, rule count and notification volume have been recorded while the platform is known good. Most post-upgrade checks are comparisons, and without these numbers they degrade into asking whether the current value looks about right. Capturing them takes a minute now and is impossible to do afterwards. for q in "count(up)" "prometheus_tsdb_head_series" "rate(prometheus_tsdb_head_samples_appended_total[5m])"; do printf "%s " "$q"; curl -fsS -G http://prometheus.internal:9090/api/v1/query --data-urlencode "query=$q" | jq -r '.data.result[0].value[1]'; doneWhatever monitors the platform being upgraded is not itself being upgraded in the same window, and it is confirmed working first. Upgrading both together removes your ability to see the upgrade you are performing, which is the one circumstance where being blind costs the most. If the monitoring path runs through the component under change, that is the finding, and it is worth fixing before the upgrade rather than during it. curl -fsS -G http://meta-prom:9090/api/v1/query --data-urlencode 'query=up' | jq '.data.result | length'Any silence created for the change is scoped to what is being touched and expires on its own. An estate-wide open-ended silence is how a real incident during a maintenance window goes unnoticed, and silences reliably outlive the change that justified them. Check what is already silenced before adding more, because the leftovers from previous windows are usually still there. amtool --alertmanager.url=http://alertmanager.internal:9093 silence queryOne named person decides to proceed, against a written criterion, at a stated time — and the same applies to abandoning it. An upgrade with no decision point runs until it is finished or until somebody is too tired to continue, and neither of those is a decision. "Stop and revert at 05:00 regardless of progress" is a first-class outcome and needs to be named as one in advance.