Runbook: Upgrade the Full Stack

1 · Prerequisites

Confirm every item is in place before any state change.

• Upgrade Basics
• Reading Release Notes
• Compatibility
• Upgrade Order
• Canary Upgrade
• Upgrade Test
• The current and the target version of every component, written down as exact strings - never "latest"
• The deployment mechanism for each component (package manager, systemd unit, or the image pin in the manifest repository) and the access to drive it
• Write access to the repository that owns the stack configuration, and a way to merge under change control during the window
• Somewhere to put a rollback artefact that is not the host being upgraded
• A witness outside the stack - a probe, a dead-man switch or a person - because the stack cannot be the only observer of its own upgrade

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

• · The release notes for every version being crossed have been read, not only the target. A team three minors behind meets three release notes worth of breaking changes in one window, and the removed flag is usually in the middle one.
• · The plan names ONE component for this window, with one owner and one rollback artefact. Two components in the same window means the first regression cannot be attributed to either of them, and attribution is the whole reason for the sequence.
• · The compatibility matrix has been updated with the pairings actually running, read from the processes themselves (/api/v1/status/buildinfo, /api/v2/status, /api/health) rather than from the wiki. Release notes document supported pairings; the matrix documents observed ones.
• · The rollback artefact for this component exists, has been checked for size and readability, and lives off the host: a Prometheus TSDB snapshot from POST /api/v1/admin/tsdb/snapshot, a dump of the Grafana database, an export of the Alertmanager silences, and the configuration repository at a tagged commit.
• · The running configuration matches the repository. curl /api/v1/status/config prints what the Prometheus process actually loaded; an upgrade that lands on undeclared configuration has a rollback that restores the wrong file.
• · The target version has been started in staging against production-shaped configuration, and the dry-run parse passed for this component: promtool check config, promtool check rules, amtool check-config, loki -verify-config, otelcol validate --config=....
• · A witness outside the stack is in place for the whole window. Upgrading the thing that would page you means arranging in advance for something else to page you - a blackbox probe from another network, a dead-man switch, or a named person watching.
• · The window is one where a regression is both detectable and attributable: enough traffic that a failure shows, enough staffing that somebody sees it, and no unrelated change in flight from another team.
• · The hold decision has an owner and a clock: who is allowed to call "stop here and stay on the current step", and the wall-clock time at which the window closes whether or not the sequence is finished.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1. 1Freeze and tag. Merge nothing else into the stack configuration repository for the duration of the window, and tag the commit that describes the currently running state. That tag is the rollback target for configuration; binaries and data have their own artefacts and their own, worse, rollback stories.
2. 2Capture the baseline into a file, not a screenshot. Component versions, count by (job) (up), the active target count, the rule group count, the Alertmanager peer count and cluster status, the Grafana datasource list. Every verification step later in this runbook is a comparison against that file, and a number nobody wrote down cannot be compared.
3. 3Confirm the destinations before any binary moves. The paging provider, the chat webhook, the remote-write or long-term backend: these sit outside your change control and are the one part of the chain a git revert does not reach. If a newer Alertmanager will send them a different payload shape, that conversation happens now, not at 03:00.
4. 4Alertmanager first. amtool check-config against the new binary, then roll the whole cluster to the new version in one pass - never leave two Alertmanager versions gossiping across a boundary the release notes flagged. Gate: /-/ready answers, /api/v2/status reports the new version with cluster status ready and the full peer count, and a deliberately fired test alert arrives at a real receiver.
5. 5Hold, then Prometheus. Take the TSDB snapshot, run promtool check config and promtool check rules with the NEW promtool, then restart. Gate: /-/ready, /api/v1/status/buildinfo shows the target version, /api/v1/alertmanagers lists every Alertmanager on the /api/v2 path, the active target count matches the baseline, and prometheus_rule_group_iterations_missed_total is not climbing.
6. 6Exporters and node agents after Prometheus, not before. They emit the shape Prometheus parses, so the parser moves first. Roll one host, diff its /metrics output against the baseline for renamed metrics and changed units, and only then roll the rest. Gate: every target up, and no recording rule that consumed a changed series has gone empty.
7. 7Loki. Configuration compatibility, schema compatibility and storage compatibility are three separate decisions; take at most one of them per window. loki -verify-config, then roll. Gate: writes are accepted, and a LogQL query over a range that spans the cutover returns rows from BOTH sides of it.
8. 8Tempo. Same shape as Loki, same rule about migrations. Gate: a trace ID that was written before the restart is still retrievable after it, and the ingest path accepts new spans.
9. 9OTel Collector or Grafana Alloy, after the backends they write into. otelcol validate --config=/etc/otelcol/config.yaml or alloy fmt --check, then restart. The sending queue held in memory is lost across the restart in both directions, so drain it or accept the gap deliberately and write down which.
10. 10Grafana last. Dump the database first: the schema migration that runs on the first start of a new Grafana is the one step in this runbook with no reverse. Gate: /api/health reports the new version and "database": "ok", every datasource passes its health check, and one dashboard per backend renders real data rather than an empty panel.
11. 11Hold each component for its own window before starting the next. The gate proves the process started; the hold proves it survives a compaction, a full scrape rotation and a rule evaluation cycle. Most slow-path regressions are invisible for the first ten minutes and obvious by the fourth hour.
12. 12Close the change. Reconcile the baseline file line by line, update the compatibility matrix with the observed pairing, and record every gap window with a start and end time. Each restart in this sequence produced a hole in the data, and somebody will eventually ask which minutes are missing.

4 · Verification

Confirm the procedure actually fixed the problem.

• ✓Every component reports the intended version from its own API rather than from the package manager: /api/v1/status/buildinfo (Prometheus), /api/v2/status (Alertmanager), /api/health (Grafana), loki -version, tempo -version, otelcol --version. A package manager reports what is installed on disk, not what the running process loaded.
• ✓The counts match the baseline file: active targets, rule groups, Alertmanager peers, Grafana datasources. A count that is lower but still plausible is exactly the failure this runbook exists to catch, because nothing about it looks like an error.
• ✓/api/v1/alertmanagers on Prometheus lists every Alertmanager replica, on the /api/v2 path. One entry where the deployment has two replicas means half the alerting chain is silently not receiving.
• ✓A real alert has traversed the entire chain and arrived at a real receiver since the window opened. "Alertmanager reports ready" is a different and much weaker claim; the only proof of delivery is a delivery.
• ✓Queries spanning the cutover return data from both sides of it for metrics, logs and traces alike. The gap should be the length of a restart, not the length of the retention.
• ✓No recording rule has gone silent: each rule group still produces rows, and prometheus_rule_group_iterations_missed_total is flat rather than climbing.
• ✓Every Grafana datasource passes its health check and at least one dashboard per backend renders real data. A datasource that saves without error and returns nothing is the commonest post-upgrade shape.
• ✓The compatibility matrix records the observed pairing, and every gap window is written into the change record with start and end times.

5 · Rollback

If verification fails, undo the procedure in reverse order.

• ↶Configuration rolls back by reverting to the tagged commit and reloading. That is the only genuinely symmetrical part of this runbook; treat everything below as recovery rather than reversal.
• ↶Alertmanager: reinstall the previous version across the WHOLE cluster in one pass. A mixed-version gossip ring is the failure you are rolling back from - do not create one on the way out. Re-import the exported silences; any silence created during the window is otherwise lost, and a lost silence becomes a page.
• ↶Prometheus: reinstall the previous binary, and if the new version has already written blocks, restore the data directory from the pre-upgrade snapshot. A newer on-disk format is not readable by the older binary, so a rollback without the snapshot is a rollback that starts with an empty database.
• ↶Exporters: reinstall the previous package. Series whose names or units changed during the window keep both shapes on disk, so dashboards that span the window stay split until retention rolls past it. That is a cosmetic defect with an expensive-looking graph, and it is worth telling the dashboard owners rather than letting them find it.
• ↶Loki and Tempo: reverting the binary is safe only while no schema entry has been added and the old index files still exist. Once a migration has run and the old files have been removed, there is no rollback - only a restore from object storage, at object-storage speed.
• ↶Grafana: restore the database dump. The schema migration on first start of the new version is one-way, and the previous binary will either refuse the migrated database or misread it. Rolling back the package alone, without the dump, is how a dashboard estate is lost.
• ↶OTel Collector and Alloy: revert the configuration and the image pin. Whatever was in the sending queue at the moment of the restart is gone, in both directions, and no rollback recovers it.
• ↶If the sequence is several components deep, roll back in reverse order of the upgrade and re-run each gate on the way out. An unvalidated rollback is just another untested change, made faster and under more pressure.

6 · Escalation

When the runbook isn't enough, contact:

• · Escalate the moment two components are on a pairing the compatibility matrix does not cover and the window has closed. The stack is now running a combination nobody tested, and every incident diagnosed against it inherits that doubt.
• · Escalate to the service owners before any step that stops alert delivery, however briefly. During the Alertmanager step nothing pages; whether that is acceptable is a business decision and not an operator one.
• · Escalate to every consuming team if the sequence has to be held mid-way. A half-upgraded stack is a supported state only for as long as a named person owns it and knows it is half-upgraded.
• · Escalate to a second operator before restoring a Prometheus data directory or a Grafana database over a live one. Both are one-way, both take a path as an argument, and both have been run at 03:00 against the wrong one.
• · Escalate to the storage or schema owner before running any Loki or Tempo migration that rewrites or removes index files. That is a migration, not a version bump, and it does not belong in the same window as one.
• · Escalate to security if the release being applied is a CVE fix. The window is then bounded by the exposure rather than by convenience, and holding on the current version may not be one of the available options.

References

1. Prometheus management API (reload, snapshot, lifecycle)
2. Prometheus HTTP API (status, targets, rules, alertmanagers)
3. Prometheus storage: TSDB layout and on-disk format
4. promtool command-line reference
5. Alertmanager configuration and clustering
6. Grafana upgrade guide
7. Loki upgrade notes
8. Tempo upgrade notes
9. OpenTelemetry Collector configuration
10. Semantic Versioning 2.0.0