Identifying the physical disk behind a failed OSD

Why this matters in production

Pulling the wrong disk from a running chassis takes a healthy OSD offline and turns a single failure into a double one. The identification chain from OSD number to physical bay has several links and each is worth confirming.

The chain

osd.13 → host → device path → serial/WWN → physical bay

Every step should be verified rather than assumed.

From OSD to host and device

ceph osd tree | grep -B5 'osd\.13'
ceph osd metadata 13 | jq -r '{hostname, devices, device_ids, device_paths}'

{
  "hostname": "ceph-osd-01",
  "devices": "sdg",
  "device_ids": "SEAGATE_ST16000NM_ZR51ABCD",
  "device_paths": "/dev/disk/by-path/pci-0000:18:00.0-sas-phy6-lun-0"
}

device_ids carries the serial and device_paths the physical topology. Both are stable in ways /dev/sdg is not.

Confirming on the host

ssh ceph-osd-01 lsblk -o NAME,SIZE,SERIAL,WWN,MODEL
ssh ceph-osd-01 smartctl -i /dev/sdg | grep -E 'Serial|Model'
ssh ceph-osd-01 ls -l /dev/disk/by-id/ | grep sdg

Compare the serial against what ceph osd metadata reported. If they disagree, the device path has changed and the metadata is stale.

Locating the physical bay

# via the device health module
DEVICE_ID=12
ceph device ls-by-daemon osd.13
ceph device light on ${DEVICE_ID}
ceph device light off ${DEVICE_ID}

# directly, where the enclosure supports it
ssh ceph-osd-01 'ledctl locate=/dev/sdg'
ssh ceph-osd-01 'ledctl locate_off=/dev/sdg'

The locate LED is the reliable confirmation: turn it on, have someone confirm which bay is lit, then act.

Without locate LEDs

# enclosure and slot from the device path
ssh ceph-osd-01 'ls -l /dev/disk/by-path/ | grep sdg'
ssh ceph-osd-01 'lsscsi -v | grep -A2 sdg'

# from the SAS topology
ssh ceph-osd-01 'sas3ircu 0 display' 2>/dev/null | grep -B5 -A5 ZR51ABCD

The by-path link encodes the controller, the PHY, and the LUN, which maps to a physical position through the chassis documentation.

Where none of this is available, the serial number on the drive label is the last resort — which means reading it, and therefore pulling drives to look, which is exactly what the earlier steps avoid.

Verifying before removal

# the OSD should be stopped and purged
ceph osd tree | grep -w 13
ceph orch device ls ceph-osd-01 | grep sdg

# the serial matches
ssh ceph-osd-01 smartctl -i /dev/sdg | grep Serial

Quiz

Production discipline

Confirm the serial number against ceph osd metadata before any physical removal; the device path is not stable across reboots and pulling the wrong disk converts one failure into two. Use the locate LED where the enclosure supports it — visual confirmation is worth the minute it takes.

Cross-course references

• Kubernetes: identifying the physical node behind a logical name has the same verification requirement
• Linux: identifying disks by serial or WWN is standard practice in any storage operation