Removing a host that cannot be drained

Why this matters in production

Every documented removal procedure assumes the host answers. When it does not, following that procedure leaves a drain that never completes and a CRUSH map that still contains a machine which no longer exists.

Why the normal drain does not apply

ceph orch host drain stor-07
ceph orch osd rm status

OSD  HOST     STATE     PGS  REPLACE  FORCE  DRAIN STARTED AT
12   stor-07  draining  118  False    True   2026-08-18 09:14:02
13   stor-07  draining  121  False    True   2026-08-18 09:14:02

The orchestrator drains by reweighting, waiting for PGs to move, then stopping the daemon and zapping the device. Two of those four steps require an agent on the host. The entries stay in draining forever.

# cancel the stuck removals before doing anything else
ceph orch osd rm stop 12
ceph orch osd rm stop 13

Mark them out in one command

ceph osd out 12 13 14 15 16 17 18 19

Marking them out individually produces one osdmap epoch per OSD, and
each epoch is a new CRUSH mapping. Data that starts moving toward a
target excluded by the next epoch moves twice.

ceph osd safe-to-destroy 12 13 14 15 16 17 18 19

safe-to-destroy reports whether any PG would lose data if those OSDs vanished. On a healthy size=3 pool with the rest of the cluster intact it returns safe immediately, because the two surviving copies are enough.

The manual teardown

for id in 12 13 14 15 16 17 18 19; do
  ceph osd purge "$id" --yes-i-really-mean-it
done

# purge refuses while the OSD is up; confirm first if it errors
ceph osd tree down | grep -E 'osd\.(1[2-9])'

# the empty host bucket survives the OSD removals
ceph osd crush remove stor-07
ceph osd tree | grep -c stor-07

Clearing the orchestrator

ceph orch host rm stor-07 --offline --force
ceph orch host ls

Without --offline, cephadm tries to reach the host to remove its daemons and refuses when it cannot. --force permits removal while daemons are still recorded against it.

Verifying nothing is left

ceph osd tree | grep -i stor-07 || echo "absent from CRUSH"
ceph auth ls 2>/dev/null | grep -E 'osd\.(1[2-9])$' || echo "no stale keys"
ceph orch host ls | grep -i stor-07 || echo "absent from orchestrator"
ceph config-key ls | grep -i stor-07 | head
ceph -s

Quiz

Production discipline

Cancel stuck orchestrator removals before starting a manual teardown, and mark every OSD on the dead host out in one command rather than one at a time. Finish by removing the empty CRUSH host bucket and the cephadm host entry — the OSDs going away does not remove either, and both cause failures later that name the wrong cause.

Cross-course references

• Kubernetes: deleting the pods on a dead node does not delete the Node object
• Linux: removing a device from an array leaves its superblock unless you wipe it