Skip to main content
RunBook Academy

← All checklists in Ceph

As neededceph-node-maintenance

Node Maintenance Checklist

7 items ·5 critical ·2 warn ·0 info

Purpose

Taking a Ceph host out of service is safe if the cluster is told first and told again afterwards. It is dangerous when it is not: pulling a host without setting flags starts a full backfill of every OSD on it, and forgetting to clear those flags afterwards leaves the cluster unable to self-heal the next time something dies. This checklist is the wrapper around whatever the actual maintenance is.

Order of work

Unlike the readiness checklists, this list is strictly sequential and the frontmatter is in execution order. window and flag are the entry gate. out is the wait - the step people skip, and the reason a host gets pulled while data is still only on it. work is the maintenance itself, whatever it happens to be. rejoin, unset-flags and verify are the exit gate, and all three are required: a host that is back up with flags still set looks fine in ceph -s and is not fine at all.

Before you start

Have the change record, the expected duration, the list of OSDs on the host, and the current cluster health. Do not start from a cluster that was already degraded before you touched it.

Verification

Maintenance is complete when the flags are cleared and every PG is back to active+clean - not merely when the host is reachable again. Mark an item N/A rather than leaving it unchecked when it does not apply to this cluster.

Escalation

If backfill has not completed by the end of the window, extend the window rather than proceeding. Bring the on-call storage engineer in.

Critical5 items

  1. ceph progress
  2. ceph -s
  3. ceph osd dump
  4. ceph -s

Warning2 items

  1. ceph osd dump