Adding OSDs to an existing cluster

Why this matters in production

Adding OSDs is routine and produces a rebalance whose size and timing are under your control if the sequence is right.

Before starting

ceph -s
ceph health detail
ceph osd df | sort -k17 -rn | head -3
ceph orch device ls --wide

Do not start if:
  the cluster is not HEALTH_OK
  a recovery is already running
  another expansion is in progress

Adding capacity during an existing recovery compounds the movement and extends both.

The sequence

# 1. verify the devices are available
# the host being expanded:
HOST=ceph-04

ceph orch device ls "$HOST" --wide

HOST      PATH      TYPE  SIZE  AVAILABLE  REJECT REASONS
ceph-04   /dev/sdb  hdd   16T   Yes
ceph-04   /dev/sdc  hdd   16T   No         Insufficient space, LVM detected

The reject reason names exactly what prevents a device being used.

# 2. clear a device if it has stale data
ceph orch device zap ceph-04 /dev/sdc --force

# 3. add all the host's OSDs together
ceph orch daemon add osd ceph-04:/dev/sdb
ceph orch daemon add osd ceph-04:/dev/sdc
# or, from a spec
ceph orch apply -i osd-spec.yaml

# 4. verify they are up and in
ceph osd tree | grep -A10 ceph-04
ceph -s

Controlling the rebalance

# before adding, if the impact must be bounded
ceph osd set norebalance

# add the OSDs; they come up but no data moves
ceph orch daemon add osd ceph-04:/dev/sdb

# release the movement when ready
ceph osd unset norebalance

# and pace it
ceph config set osd osd_max_backfills 2
ceph config set osd osd_mclock_profile high_client_ops

Adding with norebalance set lets the OSDs be verified as healthy before any data moves onto them.

Watching it

watch -n 30 'ceph -s | grep -E "misplaced|recovery"'
ceph osd df | sort -k17 -n | head -5

The new OSDs filling and the existing ones emptying is the expected pattern.

The common mistakes

ID=12
ceph osd tree | grep -E 'hdd|ssd|nvme'
ceph osd crush get-device-class osd.${ID}

Quiz

Production discipline

Set norebalance before adding OSDs so they come up and can be verified before any data moves onto them, then clear it when ready. Add every OSD of an expansion together — individual additions produce a rebalance each, most moving data to intermediate placements.

Cross-course references

• Kubernetes: batching node additions avoids repeated rescheduling waves
• Linux: combining storage changes into one commit avoids intermediate movement