Preparing disks for OSD use

Why this matters in production

Preparing a disk means destroying whatever is on it, and device paths are not stable. Identifying the device by something durable before zapping it is the difference between provisioning a new disk and wiping a production one.

What makes a device usable

ceph orch device ls --wide
lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINT,MODEL,SERIAL

A device is available when it has no partition table, no filesystem signature, no LVM metadata, is not mounted, and is large enough.

Identifying a device durably

ls -l /dev/disk/by-id/
lsblk -o NAME,SERIAL,WWN
smartctl -i /dev/sdb | grep -E 'Serial|Model'

/dev/sdb can be a different physical disk after a reboot. Serial numbers and WWNs do not move.

# confirm before acting
smartctl -i /dev/sdb | grep 'Serial Number'
# compare against the disk you physically intend

Clearing a device

ceph orch device zap ceph-osd-01 /dev/sdb --force

Or manually:

wipefs -a /dev/sdb
sgdisk --zap-all /dev/sdb
dd if=/dev/zero of=/dev/sdb bs=1M count=100 oflag=direct
blkdiscard /dev/sdb        # SSD/NVMe: fast and thorough

blkdiscard on flash is both faster and better for the device than writing zeros.

Controller configuration

RAID under Ceph hides device failures from CRUSH, wastes capacity on redundancy Ceph already provides, and adds a failure domain the cluster does not model. Where a controller cannot do passthrough, single-disk RAID-0 volumes are the workaround — with the caveat that the controller’s cache behaviour then applies.

Verifying health before deployment

smartctl -a /dev/sdb | grep -E 'Reallocated|Pending|Uncorrectable|Power_On_Hours'
smartctl -t short /dev/sdb
smartctl -l selftest /dev/sdb

A new disk with reallocated sectors or a failed self-test should be returned rather than deployed. Catching it now costs a return; catching it later costs a rebuild.

DEVICE_ID=12
ceph device ls
ceph device get-health-metrics ${DEVICE_ID}

Quiz

Production discipline

Confirm the serial number of any device before zapping it; paths move and the mistake is unrecoverable. Verify SMART health on new disks before deployment — a device with reallocated sectors out of the box costs a return now and a rebuild later.

Cross-course references

• Kubernetes: local persistent volume provisioning faces the same device identification problem
• Linux: identifying disks by WWN or serial is standard practice for exactly this reason