Upgrade rollback and the backup that replaces it

Advanced⏱ ~18 mincephcephadm

What you'll learn

State what is reversible in an upgrade
Prepare the backup that substitutes for rollback
Handle a partially completed upgrade
Decide between completing and restoring

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Ceph has no downgrade, so the plan must account for that rather than assuming a reverse gear exists.

What is reversible

Element	Reversible
The upgrade itself	no
Stopping partway	yes — the cluster runs mixed-version
Configuration changes made alongside	yes
Feature flags enabled after upgrading	generally not
`require-min-compat-client` raised	lowering it does not undo feature enablement
Individual daemon versions	no; older daemons may not start against upgraded state

ceph orch upgrade stop
ceph versions

Stopping leaves the cluster mixed-version, which is a supported
transitional state but not a rollback.

The backup that substitutes for rollback

What a restore would need:
  the monitor store, from before the upgrade
  the OSD data, which is far larger
  the configuration and CRUSH map

In practice, restoring a whole cluster to a pre-upgrade state is a disaster recovery operation rather than an upgrade rollback. The realistic protections are different:

Protection	Value
Application-level backups of the data	restores the data to a new or rebuilt cluster
A tested restore procedure	the backup is only as good as the restore
Upgrading a test cluster first	finds the problem before production
Upgrading one cluster of several first	same, at production scale
Reading the release notes	finds documented problems before encountering them

# what a restore would actually involve
ceph df
# the data volume determines the restore duration

Handling a partially completed upgrade

ceph orch upgrade status
ceph versions
ceph -s

Situation	Action
Stalled on a health gate	resolve the condition; the upgrade continues
A daemon failing on the new version	investigate; it is the specific problem
A regression affecting the cluster	complete the upgrade, then address it
Mixed-version state, upgrade stopped	restart or complete it; do not remain

Completing is almost always better than remaining mixed. The mixed state
is transitional by design and is not tested as a long-term configuration.

ceph orch upgrade resume

Deciding between completing and restoring

Complete when:
  the problem is a specific daemon or condition
  the upgrade is most of the way through
  the regression is tolerable and addressable afterwards

Consider restoring when:
  data integrity is affected
  the cluster is unavailable and the cause is the upgrade
  the release notes document the problem as a known blocker

Restoring is a last resort with a duration proportional to the data
volume, which is why the test-first practice matters so much.

Quiz

Knowledge check · 4 questions

Q1. What is the practical protection against an upgrade problem, given there is no rollback?

Stopping the upgrade partway, since the daemons still on the old release preserve the previous behaviour for the pools they serve Upgrading a test cluster — or the least critical production cluster — first, so the problem is found before it matters A monitor store backup taken beforehand, which restores the maps and returns each OSD to its previous version on the next restart Lowering require-min-compat-client afterwards, which disables the features the new release enabled and restores the earlier behaviour
Q2. Mixed-version operation is tested only for the hours an upgrade takes, not as a configuration a cluster runs for weeks.

True False
Q3. Decide how to handle an upgrade problem.

An upgrade is 80% complete when a regression appears affecting performance but not availability or data integrity.
Q4. Why is mixed-version operation supported during an upgrade but not as a long-term state?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Upgrade a test cluster or the least critical production cluster first — there is no rollback, and finding the problem before it matters is the only real protection. If an upgrade encounters a tolerable problem partway, complete it: a mixed-version state is worse than either endpoint.

Cross-course references

Kubernetes: canary upgrades exist precisely because rollback is imperfect
Linux: irreversible changes require testing rather than reversal planning