Rollback Strategy

A Loki 3.x upgrade moves the compactor to a new index format. The canary looks clean. The fleet rollout proceeds. Two hours later, the compactor starts emitting out of range errors on every chunk. The team has the previous version’s binary in the package cache. It does not have a snapshot of the old index. Restoring the old binary against the new index does not work — the old binary cannot read the new layout. The team is now in a forward-only state with no documented path back.

The upgrade was reversible in principle. The rollback was not prepared in practice. The strategy is what closes that gap.

What a rollback strategy is

A rollback strategy is the plan, the artefacts, and the verification steps that allow a failed upgrade to be reversed without data loss and within a documented time bound. It has four primitives:

1. Snapshot. A point-in-time copy of every artefact the upgrade touches: configuration files, on-disk data, remote-write buffers, query caches, dashboards.
2. Runbook. A checked-in document that names the exact commands, in order, for the reversal. Written for an engineer who did not perform the original upgrade.
3. Validation. A list of commands that confirm the rollback has actually restored the prior state — not just appeared to.
4. Communication. The contact list, the change ticket, and the stop-the-bleed criteria that escalate from “rollback now” to “wider incident.”

The strategy is not a wiki page. It is a checked-in artefact next to the upgrade plan, written before the upgrade is performed.

Why a sysadmin cares

Every upgrade has a failure probability. The discipline of upgrading is to lower that probability; the discipline of rollback is to absorb the remainder. Three failure shapes that a prepared rollback prevents:

• Schema-only forward path. The on-disk format changed; the old binary cannot read the new data; the operator cannot go back without the pre-upgrade snapshot.
• Configuration drift on rollback. The old configuration references a flag that the new binary deprecated. The rollback to the old binary fails because the configuration has already been migrated.
• Silent partial rollback. The operator rolls back the binary but not the configuration, or the configuration but not the rules. The result is a half-old, half-new stack that no release note covers.

A prepared rollback absorbs each of these. An unprepared operator improvises under pressure at 03:00.

How it works: the rollback flow

  Decision to roll back
          |
          v
  +-------------------+   No   +-------------------+
  | Snapshot present?  |------>| Forward-only path |
  +-------------------+       +-------------------+
          | Yes
          v
  +-------------------+   No   +-------------------+
  | Runbook present?   |------>| Improvise         |
  +-------------------+       +-------------------+
          | Yes
          v
  +-------------------+
  | Execute runbook   |
  | (in order)        |
  +-------------------+
          |
          v
  +-------------------+   No   +-------------------+
  | Validation passes? |------>| Investigate       |
  +-------------------+       +-------------------+
          | Yes
          v
  +-------------------+
  | Communicate close |
  +-------------------+

The flow has three decision points where the rollback can stall. Each one is a place where the operator should have prepared before the upgrade.

How to configure it: a rollback runbook

The runbook is a markdown file, checked in next to the upgrade plan. It is written for a tired engineer at 03:00 who is not the original author:

# Rollback: prometheus 2.55.0 -> 2.55.1 (host prom-prod-03)

**Stop-the-bleed criteria**: any of the following triggers
immediate rollback without further investigation:
- `prometheus_tsdb_head_series` drops by more than 5% within
  ten minutes of upgrade.
- Rule evaluator returns `unsupported API version` for any
  rule group.
- Scrape failures exceed 1% of total scrape targets.

## Pre-conditions
- Snapshot exists at `s3://prom-snapshots/2026-09-11-pre/`.
- Binary `prometheus-2.55.0.linux-amd64.tar.gz` available in
  `/var/cache/prometheus/`.
- Configuration under `/etc/prometheus/` is the version
  that was running at 02:00 UTC.

## Steps
1. Stop the new binary:
   `systemctl stop prometheus`
2. Restore the prior binary:
   `tar -xzf /var/cache/prometheus/prometheus-2.55.0.linux-amd64.tar.gz -C /opt/prometheus/`
3. Restore the prior configuration:
   `cp -r /etc/prometheus/backup-2026-09-11/* /etc/prometheus/`
4. Restore the TSDB snapshot:
   `promtool tsdb restore s3://prom-snapshots/2026-09-11-pre/ /var/lib/prometheus/data`
5. Start the old binary:
   `systemctl start prometheus`
6. Wait for `/-/ready` to return `ready`.
7. Validate the metric pipeline:
   `curl -fsSG --data-urlencode 'query=up{job="prometheus"}' http://localhost:9090/api/v1/query`

## Validation
- `up{}` for every scrape target equals the count that was
  running at 02:00 UTC.
- `prometheus_tsdb_head_series` is within 1% of the value at
  02:00 UTC.
- Rule groups evaluate without errors for ten minutes.

## Communication
- Notify the change ticket `CHG-1234` that rollback has
  completed.
- Notify the on-call rotation in `#observability-oncall`.
- Post a brief summary in `#observability-incidents`.

The runbook is executable by anyone who can ssh to the host. It does not require the original upgrade author.

How to validate it

The validation step is the part operators skip when they are confident the rollback worked. The cost of skipping it is a half-rolled-back state that surfaces at the next incident.

# READ-ONLY: confirm the running binary is the rollback
# target, not the upgrade.
prometheus --version 2>&1 | head -1
# Prometheus version 2.55.0

# READ-ONLY: confirm the TSDB head is back to its prior
# cardinality.
curl -fsSG --data-urlencode \
    'query=prometheus_tsdb_head_series' \
    http://prom-1:9090/api/v1/query \
    | jq '.data.result[0].value[1]'
# "412381"

# READ-ONLY: confirm the rule evaluator is healthy.
promtool check rules /etc/prometheus/rules/*.yml
# SUCCESS: 27 rules found
curl -fsS -X POST http://prom-1:9090/-/reload

# READ-ONLY: confirm the scrape pipeline is whole.
curl -fsSG --data-urlencode 'query=count(up==1)' \
    http://prom-1:9090/api/v1/query \
    | jq '.data.result[0].value[1]'
# "412"

# READ-ONLY: cross-check against the upgrade-time baseline.
diff <(curl -fsSG --data-urlencode 'query=count by (job)(up)' \
            http://prom-1:9090/api/v1/query | jq -S) \
     <(cat upgrades/baselines/prom-2026-09-11.json | jq -S)

The last command — diff against the upgrade-time baseline — is the cheapest confirmation that the rollback restored the prior state rather than appearing to.

How it can fail

Five failure modes recur around rollback.

1. No snapshot. The upgrade proceeded without a snapshot. The on-disk format has changed; the rollback to the old binary is not possible without the old data. Symptom: the operator is in a forward-only state.
2. Snapshot taken after the bad state. The snapshot was taken during the upgrade window, after the new format was partially written. Symptom: restoring the snapshot restores the bad state, not the prior state.
3. Configuration drift on rollback. The configuration was migrated to the new format during the upgrade. The old binary cannot read the migrated configuration. Symptom: the old binary refuses to start; the operator must manually restore the prior configuration as well.
4. Incomplete notification flush. Alertmanager rolled back while alerts were in-flight. Symptom: duplicate notifications are sent; the on-call rotation receives the same alert twice.
5. No validation step. The rollback appears to succeed; the operator moves on; a partial rollback (old binary, new config) surfaces at the next rule evaluation. Symptom: rule evaluator errors six hours after the rollback was declared successful.

How to troubleshoot it

When a rollback misbehaves, the diagnostic order is from the data path back to the configuration:

1. What does the running binary say? prometheus --version, loki -version. Confirm the binary is actually the rollback target.
2. What does the data path say? Read the most recent log lines from the storage engine. Look for “unable to read”, “unsupported format”, “out of range”.
3. What does the configuration say? Confirm the configuration files on disk are the pre-upgrade versions. diff against the configuration backup.
4. What does the metric say? up{}, prometheus_tsdb_head_series, loki_request_duration_seconds. Compare against the upgrade-time baseline.
5. Form a hypothesis. Is the binary rolled back but the configuration not? Is the configuration rolled back but the data not?
6. Test the hypothesis. Apply the missing piece of the rollback.
7. Validate the fix. Re-run the validation steps from the runbook.

Security implications

Rollback has two security implications:

• Credential state. A credential that was rotated during the upgrade window needs to be re-rotated after rollback if the rotation was tied to the new version’s behaviour.
• Authentication protocol. A rollback to an older version may down-negotiate the authentication protocol. Symptom: the wire is no longer mTLS-protected without any log line announcing it.

Performance implications

Rollback has a performance cost beyond the obvious restart time:

• Replay cost. A rolled-back Prometheus must re-read the WAL; the cost is bounded by the WAL size but is not zero.
• Cache cold-start. The query engine’s cache is empty after the restart; the first queries are slower.
• Index rebuild. Loki’s index may need to re-compact after a rollback if the compactor was mid-flight during the upgrade.

Production guidance

• The snapshot is part of the change. A change that has no snapshot is not a change; it is a wish.
• The runbook is checked in. A wiki page is lost; a markdown file is reviewed.
• The validation is automated. The validation commands live in the runbook; they are runnable by anyone.
• The communication list is current. The on-call rotation, the change ticket, the incident channel — all three must point at the current state.

Verification

You should now be able to answer:

• What four primitives must a rollback strategy provide?
• Why is a snapshot taken after the upgrade useless?
• What does a half-rolled-back state look like in the metrics six hours later?
• Which component’s rollback is the most expensive to reverse, and why?

Quiz