The safe order for OSD operations

Why this matters in production

The removal sequence is short and each step protects against a specific outcome. Skipping one does not usually fail immediately, which is why the consequences are discovered later.

The sequence

# 1. exclude from placement — data starts moving, daemon keeps serving
ceph osd out 13

# 2. confirm the drain completed
ceph osd safe-to-destroy 13

# 3. stop the daemon
ceph orch daemon stop osd.13

# 4. remove from the cluster and clean the device
ceph osd purge 13 --yes-i-really-mean-it

What each step protects

What skipping each costs

Stopping the daemon before marking out. The OSD goes down/in, its PGs become degraded, and the drain that would have been non-disruptive becomes a recovery from reduced redundancy. On a size-3 pool that means running at two copies for the duration rather than three.

Purging before the drain completes. The OSD’s data is gone and the PGs that needed it must be reconstructed from the remaining copies — which is possible above min_size and is not the operation you intended.

Manual removal instead of purge. Leaves the CRUSH entry, the auth key, and the OSD map slot:

ceph osd crush remove osd.13
ceph auth del osd.13
ceph osd rm 13

That is what purge does in one command. Doing it partially leaves residue that appears in ceph osd tree as a phantom entry.

When the OSD has already failed

A device that has already failed cannot be drained — there is nothing to read from it.

# the OSD is down and its data is inaccessible
ceph osd safe-to-destroy 13
# Error EBUSY: OSD(s) 13 have 41 pgs currently mapped to them

# let recovery reconstruct from the surviving copies first
ceph osd out 13
ceph -s                        # wait for active+clean

# then remove
ceph osd purge 13 --yes-i-really-mean-it

The order is the same; the difference is that recovery reconstructs from the other copies rather than copying from this OSD. Waiting for active+clean before purging is what ensures the reconstruction finished.

Verifying afterwards

DEVICE=/dev/sdb
ceph osd tree | grep -w 13          # should return nothing
ceph auth ls | grep osd.13          # should return nothing
ceph orch device ls | grep ${DEVICE} # should show available

Quiz

Production discipline

Follow the out, verify, stop, purge order without shortcuts; each step protects against a specific outcome and skipping one fails later rather than immediately. Use safe-to-destroy as the drain progress meter as well as the safety check — its refusal message counts the remaining dependent PGs.

Cross-course references

• Kubernetes: cordon, drain, verify, delete follows the identical protective ordering
• Linux: removing a member from a cluster in the wrong order leaves the same kind of residue