Running a cephadm upgrade

Advanced⏱ ~18 mincephcephadm

What you'll learn

Start and monitor a cephadm upgrade
Understand what it does at each stage
Pause and resume it
Handle a stall

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

cephadm handles the sequencing and the health gating, and knowing what it is doing at each stage is what makes a stall diagnosable.

Starting

ceph orch upgrade check --ceph-version 19.2.1
ceph orch upgrade start --ceph-version 19.2.1

# or by image, for a specific build
ceph orch upgrade start --image quay.io/ceph/ceph:v19.2.1

ceph orch upgrade status

{
    "target_image": "quay.io/ceph/ceph:v19.2.1",
    "in_progress": true,
    "which": "Upgrading daemons of type mgr",
    "services_complete": [],
    "progress": "3/103 daemons upgraded",
    "message": ""
}

What it does at each stage

1. pulls the target image on every host
2. upgrades the managers, standby first, then failing over the active
3. upgrades the monitors, one at a time
4. upgrades the crash and other minor daemons
5. upgrades the OSDs, respecting availability
6. upgrades MDS, RGW, and other services

Between each daemon, cephadm waits for the cluster to return to its
pre-restart health before proceeding.

ceph -s
ceph orch ps --refresh | grep -v running

Monitoring

watch -n 30 'ceph orch upgrade status; echo; ceph -s | grep -E "health|osds"'

# and the version distribution as it progresses
ceph versions | python3 -c '
import sys,json
d = json.load(sys.stdin)
for k, v in d.items():
    if k == "overall": continue
    print(k, {ver.split()[2]: n for ver, n in v.items()})'

Signal	Meaning
Progress advancing	normal
Progress static for a long period	the health gate is not being satisfied
`message` populated	cephadm is reporting a problem
Health degrading	a daemon did not come back
Daemons in mixed versions	expected during the upgrade

Pausing and stopping

ceph orch upgrade pause
ceph orch upgrade resume
ceph orch upgrade stop

Pausing leaves the cluster in a mixed-version state, which is supported
for the duration of an upgrade but is not a state to remain in
indefinitely.

ceph versions

Stopping leaves it mixed permanently until the upgrade is restarted or completed, so it is for pausing to investigate rather than for abandoning.

Handling a stall

ceph orch upgrade status
ceph -s
ceph health detail
ceph orch ps --refresh | grep -v running

Cause	Check
Health not returning between daemons	`ceph health detail`
A daemon failing to start on the new version	`ceph orch ps`, its logs
Image pull failing on a host	`ceph orch upgrade status` message
An OSD not rejoining	its daemon log
A PG stuck peering	`ceph pg query`

kubectl() { :; }  # not applicable
ceph orch ps --daemon-type osd --refresh | grep -v running

# OSD_ID is the daemon named by the line above; the fsid comes from the cluster
FSID=$(ceph fsid)
OSD_ID=84

journalctl -u "ceph-$FSID@osd.$OSD_ID" --since '30 min ago' | tail -50

Quiz

Knowledge check · 4 questions

Q1. Why are managers upgraded before monitors?

Managers hold no quorum and can be restarted without affecting the cluster, so they go first to prove the target image pulls and starts correctly The manager runs the orchestrator performing the upgrade, so upgrading it first means newer code that understands the target version drives the rest A monitor restart forces a Paxos re-election that takes far longer than a manager restart, so monitors are deferred until the window is under way The manager holds the only copy of the module state, which must be rewritten in the new format before the monitors can commit an upgraded map
Q2. A stalled cephadm upgrade usually indicates a problem with cephadm.

True False
Q3. Handle a stalled upgrade.

A cephadm upgrade shows "47/103 daemons upgraded" and has not advanced in 40 minutes. The cluster reports HEALTH_WARN.
Q4. What state does `ceph orch upgrade stop` leave the cluster in?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Read ceph health detail when an upgrade stalls — cephadm gates on the cluster returning to its prior health between daemons, so the stall names its own cause. Use pause rather than stop; stopping leaves the cluster in a mixed-version state that should not persist.

Cross-course references

Kubernetes: rolling updates gated on readiness stall for the same reason
Linux: staged upgrades with health gates diagnose themselves through the gate