Purpose
The dangerous window in a Ceph upgrade is not the upgrade, it is the hours after it, when the cluster is running a mixed set of daemon versions and everybody has gone home believing it is finished. This checklist is what converts “the packages installed” into “the upgrade is done”, and it is run before the maintenance window is closed.
Order of work
The list runs in three passes. The state pass is versions, health and
modules: confirm that every daemon actually moved, that the cluster settled,
and that the manager modules came back - modules are the ones that fail
silently, taking the dashboard and the metrics exporter with them while
ceph -s still reads HEALTH_OK. The assurance pass is scrub and
performance, which are the only two items that look at the data rather than
at the daemons. The record pass is document, which is what makes the next
upgrade cheaper than this one.
Before you start
Have the pre-upgrade performance baseline, the version you were upgrading from,
and the list of manager modules that were enabled beforehand. Without the
baseline the performance item is a guess.
Verification
The upgrade is complete when every critical item is checked. Leave the maintenance window open until they are. Mark an item N/A rather than leaving it unchecked when it does not apply to this cluster.
Escalation
Daemons stuck on the old version after the orchestrator reports success is a stop-and-investigate condition. Do not start the next stage on top of it.