Bringing a replacement node into service

Why this matters in production

A replacement node restores a designed topology rather than extending it, and the verification is against that design.

Differences from an expansion

# what the topology should look like
ceph osd crush tree
ceph osd tree | grep -c 'host '

Adding it

# if the old host was removed
ceph orch host add ceph-03 10.0.2.23 --labels osd
ceph osd crush move ceph-03 rack=rack1

# if the old host bucket remains, the new host takes its place
ceph osd crush tree | grep -A2 rack1

# add the OSDs
ceph orch apply -i osd-spec-ceph-03.yaml
ceph osd tree | grep -A15 ceph-03

The pre-checks from a normal addition all apply: network, devices, CRUSH position, capacity.

Reserved identities

ceph osd tree | grep destroyed

If the OSDs were removed with --replace, their IDs are reserved and the new ones adopt them:

ceph orch apply -i osd-spec-ceph-03.yaml
ceph osd tree | grep -A15 ceph-03

IDs reserved: the new OSDs take them, CRUSH position preserved
IDs released: new IDs assigned, placement recomputed

For a host replacement where the same number of devices return to the same position, reserving the IDs means the data returns to where it was.

Restoring the topology

# the host count per failure domain
ceph osd tree | awk '/rack/ {r=$NF} /host/ {c[r]++} END {for (k in c) print k, c[k]}'

# the OSD count per host
ceph osd df tree | awk '/host/ {h=$NF; c=0} /osd\./ {c++} /^$/ {if (h) print h, c}'

# and the weights
ceph osd crush tree

The cluster should return to:
  the same host count per rack
  the same OSD count per host
  the same total weight per host
  the same failure domain coverage

Verifying the designed state

ceph -s
ceph osd df | awk 'NR>1 {if($17+0>m)m=$17+0; if(mn==""||$17+0<mn)mn=$17+0}
  END {printf "spread %.1f\n", m-mn}'

# host failure absorption restored
ceph osd df tree | awk '/host/ {h=$NF} /osd\./ {s[h]+=$5} END {for (k in s) print k, s[k]}'

The check that matters is whether the cluster can again absorb a host failure, which is the property the replacement restores.

Quiz

Production discipline

Use --replace when removing a failed host’s OSDs even if the replacement is weeks away — the reserved IDs mean the data returns to its positions rather than the cluster computing a new distribution. Verify host failure absorption is restored, not just that the node is in service.

Cross-course references

• Kubernetes: replacing a node into its previous topology position avoids rescheduling churn
• Linux: an array member returning to its slot resyncs rather than triggering a reshape