Removing the OSDs

Why this matters in production

Several commands remove an OSD and they leave different residue behind.

The commands

# the orchestrator path, which handles all of it
ceph orch osd rm 84
ceph orch osd rm status

# the manual path, in order
FSID=$(ceph fsid)

ceph osd out 84
# wait for the drain
systemctl stop "ceph-$FSID@osd.84"
ceph osd purge 84 --yes-i-really-mean-it

ceph osd purge is the single command that removes the OSD from the CRUSH map, the auth registry, and the OSD map together — using the individual commands risks leaving one behind.

Preserving or releasing the ID

# preserve, for a device replacement in the same slot
ceph orch osd rm 84 --replace

# release, for a permanent removal
ceph orch osd rm 84

--replace marks the ID as destroyed rather than removing it
  → the ID is reserved
  → a new OSD created on that host adopts it
  → CRUSH placement is preserved

ceph osd tree | grep destroyed

For a node being decommissioned the IDs should be released; for a device being swapped they should be preserved.

Verifying removal

# gone from the OSD map
ceph osd tree | grep -c 'osd.84' || echo "removed"

# gone from CRUSH
ceph osd crush tree | grep -c 'osd.84' || echo "removed from crush"

# gone from auth
ceph auth ls | grep -c 'osd.84' || echo "auth removed"

# and the daemon is gone
ceph orch ps --daemon-type osd --hostname ceph-07

# residue left by a partial removal
ceph osd tree | grep -E 'DNE|destroyed'
ceph auth ls 2>/dev/null | grep 'osd\.' | while read e _; do
  id="${e#osd.}"
  ceph osd tree | grep -q "osd.$id " || echo "orphan auth entry: $e"
done

The dangerous variant

ceph orch osd rm 84 --force

This removes the OSD without waiting for the drain, so any PG for which it holds a copy the cluster needs loses that copy. It exists for OSDs that cannot drain — a failed device with no readable data — and it must never be used as a way to speed up a removal.

Quiz

Production discipline

Use ceph orch osd rm or ceph osd purge rather than composing the individual removal commands — an OSD lives in three registries and partial removal leaves residue including untracked auth credentials. Reserve --force for OSDs whose device has failed and which have nothing to drain.

Cross-course references

• Kubernetes: finalizers exist to ensure multi-registry cleanup completes
• Linux: removing a device from an array must update every record of it