Replacing a failed disk

Why this matters in production

Replacing a disk is routine and the step that goes wrong is identifying which physical device to pull.

Identifying the device

ceph osd metadata 44 | python3 -c '
import sys,json; d=json.load(sys.stdin)
for k in ("hostname","bluestore_bdev_dev_node","devices","device_ids","device_paths"):
    print("%-28s %s" % (k, d.get(k)))'

hostname                     ceph-03
bluestore_bdev_dev_node      /dev/sdf
device_ids                   sdf=ATA_HGST_HUH721616ALE600_2EJXYZ1A
device_paths                 sdf=/dev/disk/by-path/pci-0000:18:00.0-sas-...

The device_ids field carries the serial number, which is what is printed on the drive and what identifies it unambiguously.

# and light the locate LED where the enclosure supports it
DEVICE_ID=12
ceph device light on ${DEVICE_ID}
ceph device ls-by-host ceph-03

# on the host, as a cross-check
ls -l /dev/disk/by-id/ | grep sdf
lsblk -o NAME,SERIAL,SIZE | grep sdf

Serial number plus locate LED plus slot path — three independent identifiers, because pulling the wrong drive from a degraded cluster is a second failure.

Replacing

# 1. remove the OSD, preserving its ID
ceph orch osd rm 44 --replace
ceph orch osd rm status

# 2. wait for the drain, if the device is readable
watch -n 30 'ceph orch osd rm status'

# 3. for a device that has failed completely, it cannot drain
ceph orch osd rm 44 --replace --force

--force here is correct: the device has failed and holds nothing readable,
so there is nothing to drain and waiting achieves nothing.

# 4. confirm the ID is reserved
ceph osd tree | grep destroyed

# 5. physically replace the drive
# 6. the orchestrator detects and deploys, if a spec matches
ceph orch device ls ceph-03 --refresh
ceph -s

Preserving the ID

With --replace:
  the ID is marked destroyed and reserved
  the new OSD adopts it
  CRUSH position is preserved
  movement is drain + refill

Without:
  the ID is released
  the new OSD gets a new ID and position
  movement is drain + redistribute + refill

The difference is roughly half the data movement for a replacement.

Verifying

ceph osd tree | grep -A3 ceph-03
ceph osd metadata 44 | grep -E 'device_ids|bluestore_bdev'
ceph osd df | awk '$1==44'
ceph -s

The new OSD:
  has the same ID
  is up and in
  has the correct device class
  is receiving data
  the serial matches the new drive, not the old

The last check confirms the right drive was replaced.

Quiz

Production discipline

Identify a failed drive by its serial from ceph osd metadata, and light the locate LED where available — pulling the wrong drive from a degraded cluster is a second failure. Use --replace to preserve the ID, which halves the data movement the replacement costs.

Cross-course references

• Kubernetes: identifying the correct node or device before action is the same discipline
• Linux: device nodes are unstable identifiers; serials and WWNs are not