The upgrade order and why it exists

Advanced⏱ ~17 minceph

What you'll learn

State the upgrade order and its rationale
Explain the dependency at each stage
Recognise the consequences of violating it
Handle a manually-upgraded daemon out of order

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

The order is not a convention; each stage depends on the previous one having completed.

The order

1. Managers
2. Monitors
3. Crash and auxiliary daemons
4. OSDs
5. MDS
6. RGW and other services

ceph orch upgrade status
ceph versions

The dependency at each stage

Stage	Depends on
Managers first	the orchestrator driving the upgrade should be current
Monitors second	they hold the maps every other daemon reads
OSDs after monitors	they must be able to read the map format the monitors write
MDS after OSDs	they store metadata in pools the OSDs serve
RGW after OSDs	same

The monitors write the authoritative maps.
An OSD on a newer version may write map structures an older monitor
cannot process, which is why monitors go first.

Consequences of violating it

Violation	Consequence
OSDs before monitors	OSDs may write structures the monitors cannot read
Monitors before managers	the orchestrator may not understand the new monitor behaviour
MDS before OSDs	metadata operations against older OSDs may fail
A mixed monitor quorum for an extended period	map commits may be constrained by the oldest

ceph versions | python3 -c '
import sys,json
d = json.load(sys.stdin)
mon = list(d.get("mon", {}).keys())
osd = list(d.get("osd", {}).keys())
print("mon versions:", len(mon))
print("osd versions:", len(osd))
if len(mon) > 1: print("WARNING: mixed monitor versions")'

Manually upgraded daemons

A daemon upgraded outside cephadm — a package update on one host, or a
container image changed directly — leaves the cluster in a state cephadm
did not create and may not expect.

ceph orch ps --refresh
ceph versions

# reconcile by letting cephadm redeploy it
DAEMON_NAME=osd.12
ceph orch daemon redeploy ${DAEMON_NAME}

The correct handling:
  if the manual upgrade was to the same target, let cephadm continue
  if it was to a different version, redeploy the daemon to the intended one
  do not leave a daemon on a version cephadm does not know about

ceph orch ps --format json | python3 -c '
import sys,json
for d in json.load(sys.stdin):
    print("%-24s %s" % (d.get("daemon_name"), d.get("version")))' | sort -k2

Quiz

Knowledge check · 4 questions

Q1. Why must monitors be upgraded before OSDs?

Monitors take longer to restart than OSDs, so doing them first keeps the slowest stage inside the earliest part of the window OSDs read their configuration from the manager rather than the monitors, so the orchestrator must be current before any OSD restarts Monitors write the authoritative maps, and an OSD on a newer version may write structures an older monitor cannot process The order matters only for the health gate, since cephadm needs a stable quorum to judge whether the cluster returned to its prior health
Q2. A Ceph package upgrade on a cephadm-managed host changes the host tooling while the daemons keep running the image they were deployed with.

True False
Q3. Handle a daemon upgraded out of order.

An operator ran a package update on one Ceph host, and `ceph versions` now shows a mixed state that cephadm did not create.
Q4. Why does a mixed monitor quorum constrain the cluster?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Exclude Ceph packages from host update automation on cephadm-managed hosts — daemons run from container images and a package update changes host tooling rather than the cluster, producing a mismatch. Let ceph orch upgrade drive every version change.

Cross-course references

Kubernetes: control plane before nodes exists for the identical map-compatibility reason
Linux: upgrading a coordinator before its participants is a general distributed-systems rule