Executing the node addition

Why this matters in production

The pre-checks establish readiness; the execution is a short sequence where each step is verified before the next.

The sequence

# 1. cluster is ready
ceph -s | grep HEALTH_OK || { echo "not ready"; exit 1; }

# 2. throttles set
ceph config set osd osd_max_backfills 2
ceph config set osd osd_mclock_profile high_client_ops

# 3. hold the movement
ceph osd set norebalance

# 4. add the host
ceph orch host add ceph-07 10.0.2.27 --labels osd
ceph orch host ls | grep ceph-07

# 5. place it in the hierarchy
ceph osd crush move ceph-07 rack=rack4
ceph osd crush tree | grep -A2 rack4

# 6. verify devices
ceph orch device ls ceph-07 --wide

# 7. add the OSDs, all together
ceph orch apply -i osd-spec-ceph-07.yaml

# 8. verify they are up and in with the right class
ceph osd tree | grep -A15 ceph-07

# 9. release the movement
ceph osd unset norebalance

# 10. monitor
watch -n 60 'ceph -s | grep -E "misplaced|health"'

Verification at each step

ID=12
ceph osd tree | awk '/ceph-07/,/^host|^rack/' | head -20
ceph osd crush get-device-class osd.${ID}

Handling a failure partway

# a partially created OSD
# ID is the numeric OSD id from `ceph osd tree`; substitute your own:
ID=12

ceph orch osd rm "$ID" --force
ceph orch device zap ceph-07 /dev/sdX --force

With norebalance set throughout, nothing has committed data and any step can be undone.

Confirming the result

# capacity increased
ceph df

# the new OSDs are receiving data
ceph osd df | grep -A15 ceph-07

# the spread is narrowing
ceph osd df | awk 'NR>1 {if($17+0>m)m=$17+0; if(mn==""||$17+0<mn)mn=$17+0}
  END {printf "spread %.1f\n", m-mn}'

# and the cluster returns to health
ceph -s

# restore the throttles when complete
ceph config set osd osd_max_backfills 1
ceph config set osd osd_mclock_profile balanced

Quiz

Production discipline

Hold norebalance from before the host is added until every OSD is verified — device class, CRUSH position, and count are all correctable at no cost while nothing has moved. Clearing the flag is the single irreversible step and belongs last.

Cross-course references

• Kubernetes: cordoning a new node until it is verified follows the same pattern
• Linux: staging a change and committing once verified is standard practice