The removal order and why it exists

Why this matters in production

The removal order is not convention — each step protects against a specific failure, and skipping one produces that failure.

The order

1. out       → placement is recomputed without the OSD
2. wait      → the data actually moves
3. verify    → confirm zero PGs and HEALTH_OK
4. stop      → the daemon stops with nothing depending on it
5. purge     → all three registries are cleaned
6. host      → the host is removed once no daemons remain
7. CRUSH     → empty buckets are cleaned

What each step protects

The second and third are the ones that protect data. The rest protect consistency.

Consequences of skipping steps

Removing three OSDs from a size=3 pool without draining:
  every PG that had copies on all three loses all of them
  those PGs are permanently lost

This is the case the order exists to prevent, and it is reachable in a few commands.

Recovering from a wrong removal

# an OSD removed but its data still on the device
ceph-volume lvm activate --all
ceph osd tree

If the device still holds the OSD’s data and it was purged rather than zapped, reactivating it may restore the OSD:

OSD_ID=12
OSD_FSID=3e0b2c14-9f3a-4d21-8a77-1c9f0e2b5d64
ceph-volume lvm list
ceph-volume lvm activate ${OSD_ID} ${OSD_FSID}

ceph pg dump pgs | awk '$10 ~ /incomplete|down/ {print $1, $10}'

incomplete PGs after a removal indicate copies that no longer exist anywhere.

Quiz

Production discipline

Follow the order — out, wait, verify, stop, purge, host, CRUSH — and recognise that the wait and the verification are the two steps protecting data. If OSDs were removed without draining, preserve the devices immediately; reactivation is the only recovery path and zapping closes it.

Cross-course references

• Kubernetes: forced deletion of resources bypasses the protections that ordering provides
• Linux: removing array members without confirming resync completed loses data the same way