Evacuating a node

Why this matters in production

There are two ways to drain a node and they differ in what happens if the process is interrupted.

The two methods

# method 1: mark the OSDs out
for id in $(ceph osd ls-tree ceph-07); do ceph osd out "$id"; done

# method 2: reduce the CRUSH weight
for id in $(ceph osd ls-tree ceph-07); do ceph osd crush reweight osd.$id 0; done

# the orchestrator's method, which handles the sequence
ceph orch host drain ceph-07
ceph orch osd rm status

ceph orch host drain marks the OSDs out and removes them once drained, which is the intended path for a permanent removal.

Which to use

# a gradual drain
for w in 0.75 0.5 0.25 0; do
  for id in $(ceph osd ls-tree ceph-07); do
    ceph osd crush reweight osd.$id $(echo "16.0 * $w" | bc)
  done
  while ! ceph health | grep -q HEALTH_OK; do sleep 60; done
done

Stepping the weight down means each step’s movement is bounded and the cluster reaches health between them.

Executing an evacuation

# verify readiness
ceph -s
ceph osd df | sort -k17 -rn | head -3
ceph osd df tree | grep -A15 ceph-07

The remaining OSDs must be able to hold this node's data
without any of them passing backfillfull.

# start
ceph orch host drain ceph-07

# monitor
ceph orch osd rm status
watch -n 60 'ceph -s | grep -E "misplaced|health"'

Controlling the pace

ceph config set osd osd_max_backfills 2
ceph config set osd osd_mclock_profile high_client_ops

# pause if needed
ceph osd set norebalance
# and resume
ceph osd unset norebalance

An evacuation is a backfill like any other and responds to the same throttles.

Quiz

Production discipline

Verify the remaining OSDs can absorb the node’s contents below backfillfull before starting an evacuation — a stalled drain leaves the cluster in a worse state than not having started. Use ceph orch host drain for permanent removal and a stepped CRUSH reweight where each step’s movement must be bounded.

Cross-course references

• Kubernetes: draining a node requires the remaining ones to have capacity
• Linux: removing an array member requires the survivors to hold its data