Purpose
Ceph upgrades are well engineered and mostly uneventful, which is exactly why they get started without preparation. The cost of the missing preparation is not felt during the upgrade; it is felt in the twenty minutes afterwards when something is wrong and there is no MON store backup, no exported CRUSH map and no agreed rollback. This checklist is run before the first daemon is touched.
Order of work
The list runs in four passes. The reading pass is release-notes, which is
first because it can change every decision below it. The state pass is
health and recovery: upgrading a cluster that is already moving data is
how a routine upgrade becomes an incident. The backup pass is backup-mon,
backup-config, backup-keys and backup-crush - four separate items
because they live in four separate places and people reliably capture two of
them and assume the rest. The rehearsal pass is staging, window and
rollback, which is where the upgrade stops being a technical question and
becomes a scheduling one.
Before you start
Have the current and target versions, the release notes for every version in between, the staging cluster’s result, and somewhere off-cluster to put the backups.
Verification
Do not begin until every critical item is checked. The backup items in particular are checked by restoring one of them somewhere else, not by confirming the file exists. Mark an item N/A rather than leaving it unchecked when it does not apply to this cluster.
Escalation
If the cluster will not reach HEALTH_OK before the window opens, postpone. That call belongs to the change owner.