Creating the replacement OSD

Why this matters in production

The replacement OSD should match its peers — the same DB placement, the same device class, the same weight. An OSD deployed without the DB device its neighbours have performs differently for the rest of its life, and the cause is not obvious later.

Automatic replacement through drive groups

Where a drive group specification covers the host, the orchestrator deploys on the new device automatically:

ceph orch ls --service-type osd
ceph orch device ls ceph-osd-01 --refresh
ceph orch ps --daemon-type osd --hostname ceph-osd-01

This is the reason to use drive groups: a replacement disk becomes an OSD with the correct configuration without anyone specifying it.

# confirm the specification would cover it
ceph orch apply -i osdspec.yaml --dry-run

Manual deployment

# simple case, DB on the data device
ceph orch daemon add osd ceph-osd-01:/dev/sdg

# with a DB device, matching the peers
ceph orch daemon add osd ceph-osd-01:data_devices=/dev/sdg,db_devices=/dev/nvme0n1

The DB device must match what the OSD’s peers use, and there must be space on it — the purged OSD’s DB partition should have been freed by the zap.

ssh ceph-osd-01 'lsblk /dev/nvme0n1'
ssh ceph-osd-01 'vgs; lvs'

If the old DB logical volume persists, remove it before deploying:

ssh ceph-osd-01 'ceph-volume lvm zap --destroy /dev/nvme0n1/osd-db-13'

Verifying the new OSD

ceph osd tree | grep -A2 -B2 'osd\.13'
ceph osd metadata 13 | jq -r '{bluestore_bdev_type, devices, bluefs_db_devices, osd_objectstore}'
ceph osd df | grep '^ *13 '

Compare against a peer OSD on the same host:

for i in 12 13; do
  echo "== osd.$i"
  ceph osd metadata $i | jq -r '{devices, bluefs_db_devices}'
done

The new OSD should report the same DB device as its peers. If bluefs_db_devices is empty where peers have a value, the DB was not placed and the OSD will perform differently.

Weight and class

ceph osd tree | grep 'osd\.13'
# 13    hdd  16.00000          osd.13    up   1.00000

The weight should match the device size and the class should match its peers. Both are set automatically from the device, so a discrepancy means the device differs from what was expected.

Quiz

Production discipline

Use drive group specifications so replacements are automatic and configured identically to their peers; the manual DB placement is the error that costs performance silently for the life of the OSD. Compare bluefs_db_devices against a peer after every manual deployment.

Cross-course references

• Kubernetes: a DaemonSet provisioning replacements automatically is the same declarative benefit
• Linux: templated provisioning avoids the per-instance configuration errors that manual builds produce