Skip to main content
RunBook Academy

CephLVII · Replacing Failed OSDsReplacing Failed OSDs

Stopping and purging before the swap

Intermediate⏱ ~15 minceph

What you'll learn

  • Stop and purge in the correct order
  • Verify the cluster records are clear
  • Prepare the device for reuse
  • Handle a device that cannot be zapped

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Not yet marked complete on this device.

Why this matters in production

Purging before the physical swap means the new disk arrives to a cluster with no stale record of the old one. Doing it afterwards frequently means discovering the residue when the new OSD will not deploy.

The sequence

# confirmed drained
ceph osd safe-to-destroy 13

# stop
ceph orch daemon stop osd.13
ceph orch ps --daemon-type osd | grep osd.13

# purge
ceph osd purge 13 --yes-i-really-mean-it

# verify
ceph osd tree | grep -w 13
ceph auth ls | grep 'osd\.13'

All verification commands should return nothing.

Preparing the device

Where the old device is still readable — a preventive replacement rather than a failure — zap it before removal so no metadata leaves the datacentre on it:

ceph orch device zap ceph-osd-01 /dev/sdg --force

For a failed device this is not possible and not necessary; the device is being disposed of and its contents are unreadable in any case. Where the data was sensitive and the device is being returned under warranty, physical destruction or the vendor’s secure-erase process applies.

Handling a device that cannot be zapped

ceph orch device zap ceph-osd-01 /dev/sdg --force
# Error: device is busy

ssh ceph-osd-01 'lsof /dev/sdg; dmsetup ls | grep ceph'
ssh ceph-osd-01 'ceph-volume lvm list /dev/sdg'

A device still mapped through LVM by a stopped OSD needs the mapping removed:

ssh ceph-osd-01 'ceph-volume lvm zap --destroy /dev/sdg'

Verifying the cluster is clean

ceph -s
ceph osd tree
ceph orch device ls ceph-osd-01

The device should appear as available, and the OSD id should be absent from the tree. If the id still appears with no daemon, the purge did not complete:

ceph osd crush remove osd.13
ceph osd rm 13
ceph auth del osd.13

Quiz

Knowledge check · 4 questions

  1. Q1. `ceph orch device zap` reports the device is busy after the OSD has been stopped and purged. Why?

  2. Q2. Purging should be done after the physical disk swap.

  3. Q3. Prepare for a preventive disk replacement.

    A disk is being replaced preventively — SMART predicts failure but the device is still readable and the OSD is running. The old disk will be returned to the vendor under warranty.

  4. Q4. Why is device cleanup separate from purging the OSD?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Purge before the physical swap so the new disk deploys into a cluster with no stale record; doing it afterwards produces deployment failures that are hard to attribute. Remember that device cleanup is host-side LVM work separate from the cluster-side purge.

Cross-course references

  • Kubernetes: removing a node object separately from cleaning the host is the same two-sided operation
  • Linux: LVM state persisting after a service stops is standard behaviour and a standard surprise