Pre-check: devices and capacity

Why this matters in production

A device that fails in its first month costs a replacement and a rebalance. Most such devices were detectably marginal before deployment.

Devices present and available

# the host whose new devices are being checked:
HOST=ceph-07

lsblk -o NAME,SIZE,TYPE,ROTA,MODEL,SERIAL
ceph orch device ls "$HOST" --wide

HOST     PATH      TYPE  SIZE  AVAILABLE  REJECT REASONS
ceph-07  /dev/sdb  hdd   16T   Yes
ceph-07  /dev/sdc  hdd   16T   No         LVM detected

# clear a device with stale metadata
ceph orch device zap ceph-07 /dev/sdc --force

Health before deployment

for d in /dev/sd{b..m}; do
  printf '%-10s ' "$d"
  smartctl -H "$d" | grep -i 'overall-health' || echo "no SMART"
done

# the attributes that matter, on every device
for d in /dev/sd{b..m}; do
  echo "== $d"
  smartctl -A "$d" | awk '$1 ~ /^(5|187|197|198|199)$/ {print "   ", $2, $10}'
done

# NVMe
for d in /dev/nvme{0..3}n1; do
  nvme smart-log "$d" | grep -iE 'critical_warning|percentage_used|media_errors'
done

A new drive with non-zero endurance consumed or existing reallocations was not new, and deploying it starts its service life already degraded.

Device class detection

# after adding, but worth anticipating
lsblk -o NAME,ROTA

ROTA=1 → detected as hdd
ROTA=0 → detected as ssd or nvme by transport

# correct a misdetection after adding
ID=12
ceph osd crush set-device-class ssd osd.${ID}

Devices behind certain controllers report rotational incorrectly, so checking ROTA beforehand predicts whether a correction will be needed.

Planning the OSD layout

service_type: osd
service_id: ceph-07-hdd
placement:
  host_pattern: 'ceph-07'
spec:
  data_devices:
    rotational: 1
  db_devices:
    rotational: 0
  db_slots: 6

DB device sizing:
  a common guideline is a few percent of the data device per OSD
  undersized DB spills over to the data device, negating the benefit
  one NVMe serving too many OSDs becomes a bottleneck and a failure domain

# check for spillover on existing OSDs to validate the sizing
ceph health detail | grep -i BLUEFS_SPILLOVER
ceph osd df tree | head

Quiz

Production discipline

Read SMART on every device before deployment and reject any with pending, reallocated, or uncorrectable sectors — the failure is already determined and deploying it costs a repair and a replacement. Verify DB device sizing against the OSD count; spillover produces worse tail latency than having no DB device.

Cross-course references

• Kubernetes: node readiness checks before joining serve the same purpose
• Linux: burn-in and SMART verification before deploying drives is long-standing practice