The post-upgrade review

Intermediate⏱ ~17 minceph

What you'll learn

Structure a post-upgrade review
Capture the timings and the deviations
Update the runbook from what was learned
Feed findings into the next upgrade

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Most organisations upgrade several clusters, and the second one benefits enormously from a written record of the first.

What to capture

Timings:
  wall clock, start to completion
  per-daemon-type duration
  the longest single daemon
  time spent stalled

Deviations:
  every point where reality differed from the plan
  what was done about it
  whether it recurred

Outcomes:
  health at completion
  performance against the baseline
  deep scrub results over the following days
  anything still open

# per-daemon-type timing, from the orchestrator log
ceph log last 5000 cephadm | grep -i 'upgrade' | head -60

ceph orch ps --format json | python3 -c '
import sys,json,collections
c = collections.Counter(d["daemon_type"] for d in json.load(sys.stdin))
print("daemon counts for the next estimate:")
for k,v in sorted(c.items()): print("  %-8s %d" % (k,v))'

The comparison that matters

BK=/backups/ceph-pre-upgrade
diff <(ceph config dump) "$BK/config.txt"
diff <(ceph osd pool ls detail) "$BK/pools.txt"
diff <(ceph versions) "$BK/versions.txt"

Comparison	What a difference means
Config dump	an option changed, added, or removed by the release
Pool detail	a pool parameter default moved
Performance	a regression or improvement worth recording
Health check set	a new check the release introduced

ceph osd perf
# against the baseline latency.json

Updating the runbook

From every upgrade, the runbook gains:
  a corrected duration estimate for this cluster size
  any step that was missing
  any check that would have caught a problem earlier
  any communication that was inaccurate
  the specific commands that were actually used

Finding	Runbook change
Duration exceeded the estimate	raise the per-daemon figure
A module failed	add a module check to the pre-upgrade list
A client version was missed	make `ceph features` a mandatory gate
A stall was misdiagnosed	add the diagnostic sequence
A communication was wrong	fix the template

Feeding the next upgrade

Before the next cluster:
  read this review
  apply the runbook changes
  check whether the same conditions exist there
  allow the duration this one actually took, scaled

# what differs between this cluster and the next
diff <(ceph mgr module ls --format json | python3 -c '
import sys,json; print("\n".join(sorted(json.load(sys.stdin)["enabled_modules"])))') \
     /path/to/next-cluster-modules.txt

The comparison of enabled modules and service types between clusters is
the one that predicts whether the next upgrade will resemble this one.

Quiz

Knowledge check · 4 questions

Q1. What is the most useful number a post-upgrade review produces?

The count of health warnings raised in the window, which measures how disruptive it was The actual duration divided by daemon count, which makes the next window estimate honest The peak recovery throughput observed, which sizes the cluster network for the next upgrade The number of daemons that completed without needing any manual intervention at all
Q2. Two clusters sharing a Ceph version and an upgrade path can still be substantially different upgrades.

True False
Q3. Conduct a post-upgrade review.

A cluster upgrade completed. It took 40% longer than estimated, a manager module failed and was disabled, and performance matches the baseline.
Q4. Which comparison between two clusters predicts whether their upgrades will resemble each other?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Record the actual upgrade duration divided by the daemon count — it is almost always larger than planned, and it is what makes the next estimate honest. Compare module and service sets between clusters to judge whether one upgrade’s experience genuinely predicts the next.

Cross-course references

Kubernetes: upgrade retrospectives correct estimates the same way
Linux: a post-change review is worth only what its honesty about deviations is worth