Bringing the replacement into the cluster

Why this matters in production

Losing a node costs one full re-replication. Replacing it costs a second one, in the other direction. Whether you pay once or twice is decided in the first hour, before any hardware arrives.

Two migrations, or one

ceph osd set noout          # only against a committed, short delivery date
ceph osd unset noout

Holding noout is a bet that hardware arrives when someone said it would.
For a permanent loss the bet is usually wrong, and the cost of losing it
is running on two copies for as long as the slip lasts.

Parity checks before it takes placement

ceph cephadm check-host stor-07r 10.20.0.27
ceph orch host add stor-07r 10.20.0.27
ceph orch device ls stor-07r --refresh

ceph orch device ls stor-07r --format json | python3 -c '
import sys,json
for h in json.load(sys.stdin):
    for d in h.get("devices", []):
        print("%-12s %-10s %6.1f TiB  available=%s" %
              (d.get("path"), d.get("human_readable_type"),
               d.get("sys_api", {}).get("size", 0)/1024**4, d.get("available")))'

Adding the OSDs

ceph orch apply -i /root/osd-spec.yaml --dry-run
ceph orch apply -i /root/osd-spec.yaml
ceph orch ps --daemon-type osd --hostname stor-07r

# or one device at a time when you want to watch each one land
ceph orch daemon add osd stor-07r:/dev/sdb

ceph osd tree | grep -A9 stor-07r
ceph osd metadata 48 | python3 -c '
import sys,json
d = json.load(sys.stdin)
print(d.get("hostname"), d.get("bluestore_bdev_type"), d.get("osd_objectstore"))'

Phasing in the weight

# create them with zero weight so nothing moves until you say so
ceph config set osd osd_crush_initial_weight 0

for id in 48 49 50 51 52 53 54 55; do
  ceph osd crush reweight "osd.$id" 3.0
done
ceph -s | grep misplaced

ceph config rm osd osd_crush_initial_weight
ceph balancer status
ceph osd df tree | tail -12

Quiz

Production discipline

Do not hold noout against a delivery estimate — a range is not a date, and the cost of the bet is two-copy exposure for its whole duration. Bring replacement OSDs in at zero weight and raise them in stages, so a misconfigured device class or a slow network path shows up while only a fraction of the data has moved.

Cross-course references

• Kubernetes: a replacement node with different resources changes scheduling across the cluster
• Linux: adding a larger member to a striped set redistributes more than the new member