Rolling back a maintenance change

Advanced⏱ ~17 mincephgrubby

What you'll learn

Identify what is reversible in a maintenance
Preserve the rollback path
Execute a rollback
Decide between rolling back and going forward

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Some maintenance actions are reversible and some are not, and knowing which before starting determines what the plan can rely on.

What is reversible

Action	Reversible
Kernel upgrade	yes, by booting the previous kernel
Package upgrade	usually, by downgrading
Configuration change	yes
Firmware upgrade	often not
Ceph version upgrade	not straightforwardly
A drain	yes, by marking the OSDs back in
An OSD removal	no
A device zap	no
A CRUSH change	yes, at the cost of moving the data back

The pattern: changes to the host are usually reversible;
changes to the cluster's data or identity are not.

Preserving the rollback path

# the previous kernel must remain installed
ls /boot/vmlinuz-*
grep installonly_limit /etc/dnf/dnf.conf

# configuration captured before changing it
ceph config dump > /tmp/config-before-$(date +%F).txt
ceph osd getcrushmap -o /tmp/crush-before-$(date +%F).bin
ceph osd pool ls detail > /tmp/pools-before-$(date +%F).txt

# and the package state
rpm -qa --last | head -20 > /tmp/packages-before.txt

Capturing the before state costs seconds and is the difference between a rollback and a reconstruction.

Executing a rollback

# the kernel to return to, from the `ls /boot/vmlinuz-*` output above:
PREV_KERNEL=5.14.0-503.15.1.el9_5.x86_64

# kernel
ssh ceph-03 "grubby --set-default /boot/vmlinuz-$PREV_KERNEL && systemctl reboot"
ssh ceph-03 uname -r

# a drain
for id in $(ceph osd ls-tree ceph-03); do ceph osd in "$id"; done
ceph -s

# the date stamp on the capture taken before the change:
CAPTURE_DATE=2026-08-18

# a CRUSH change
ceph osd setcrushmap -i "/tmp/crush-before-$CAPTURE_DATE.bin"
ceph -s | grep misplaced

A CRUSH rollback moves the same volume back, so it is a planned operation
rather than an instant reversal.

Deciding between rolling back and going forward

Situation	Decision
The change caused a clear regression	roll back
The change is partially applied and the fix is known	go forward
The rollback itself carries risk	weigh both
Rolling back means repeating the window later	usually still correct
The change cannot be rolled back	go forward; there is no choice
Uncertain whether the change is the cause	roll back to establish it

The last is worth stating: rolling back is also a diagnostic. If the
problem disappears, the change caused it.

Quiz

Knowledge check · 4 questions

Q1. What pattern distinguishes reversible from irreversible maintenance actions?

Changes to the host are usually reversible; changes to the cluster's data or identity are not Actions the orchestrator performs are reversible, and ones done by hand on a host are not Configuration changes are reversible and everything touching a device or a package is not Anything completed inside an hour is reversible, because the cluster retains that much history
Q2. Rolling back is only useful when the change is known to be the cause.

True False
Q3. Decide whether to roll back a kernel upgrade.

A kernel upgrade was applied to one host. Its OSDs now benchmark 30% below their previous figures. The cause is not established.
Q4. What should be captured before a maintenance to preserve the rollback path?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Capture the configuration, CRUSH map, pool details, and package state before every maintenance — a rollback needs something to roll back to, and reconstructing it from memory during an incident is not the same thing. Use rollback as a diagnostic when a change is a suspect but not proven.

Cross-course references

Kubernetes: capturing manifests before applying changes preserves the same option
Linux: any change without a captured prior state is not reversible in practice