The physical replacement

Why this matters in production

The physical step is brief and the verification around it is what makes it safe. Most replacement problems are discovered at this point and are cheaper to resolve before the new OSD is deployed.

Before the swap

# the OSD is purged and the bay is identified
DEVICE_ID=12
ceph osd tree | grep -w 13          # nothing
ceph device light on ${DEVICE_ID}    # or ledctl

Confirm the lit bay with whoever is at the chassis.

The swap

Hot-swap bays allow removal and insertion with the host running. Non-hot-swap connections — some SATA configurations, some NVMe — require the host to be powered down, which turns a disk replacement into a host outage and should be planned as one.

# check whether the bay is hot-swappable before assuming
ssh ceph-osd-01 'cat /sys/block/sdg/device/../../../../removable' 2>/dev/null

After the swap

ssh ceph-osd-01 'dmesg -T | tail -20'
# [Mon Aug 18 11:04:12 2026] scsi 0:0:6:0: Direct-Access SEAGATE ST16000NM
# [Mon Aug 18 11:04:12 2026] sd 0:0:6:0: [sdg] 31251759104 512-byte logical blocks

ssh ceph-osd-01 'lsblk -o NAME,SIZE,SERIAL,MODEL | grep sdg'
ssh ceph-osd-01 'smartctl -i /dev/sdg'

Confirm the serial is the new one, the size is as expected, and the model matches what was ordered.

When the device does not appear

ssh ceph-osd-01 'dmesg -T | tail -30'
ssh ceph-osd-01 'lsblk'
ssh ceph-osd-01 'echo "- - -" > /sys/class/scsi_host/host0/scan'
ceph orch device ls ceph-osd-01 --refresh

A rescan of the SCSI host prompts enumeration where the hot-plug event was missed.

Health-checking the new device

ssh ceph-osd-01 'smartctl -H /dev/sdg'
ssh ceph-osd-01 'smartctl -a /dev/sdg | grep -E "Reallocated|Pending|Power_On_Hours"'
ssh ceph-osd-01 'smartctl -t short /dev/sdg'
# wait, then
ssh ceph-osd-01 'smartctl -l selftest /dev/sdg'

A replacement disk with reallocated sectors or a nonzero power-on hours count in the thousands is not new, and deploying it means doing this again sooner than planned.

Quiz

Production discipline

Run a short SMART self-test on every replacement disk before deploying an OSD on it; catching a bad device now costs minutes and catching it later costs another drain and rebalance. Confirm the size and model against the order — a mismatched part produces a permanently uneven host.

Cross-course references

• Kubernetes: verifying node hardware matches the pool specification before scheduling is the same check
• Linux: burn-in verification of replacement hardware is standard practice in any storage environment