Rack failure domain — aligning replicas with the building

Why this matters in production

Rack-level failure domains are what turn “we have three copies” into “we survive losing a rack”. They are also where most designs discover that their rack count and rack capacities do not support the rule they wrote down.

What it requires

rule rack_replicated {
  step take default
  step chooseleaf firstn 0 type rack
  step emit
}

• At least size racks with usable OSDs. Three racks for size 3.
• The CRUSH map must contain rack buckets with hosts correctly assigned.
• Practically, size + 1 racks so a rack can be worked on.

ceph osd crush add-bucket rack-a rack
ceph osd crush move rack-a root=default
ceph osd crush move ceph-01 rack=rack-a
ceph osd crush rule create-replicated rack_rule default rack
ceph osd pool set rbd-vms crush_rule rack_rule

The migration

Moving a populated pool from host to rack failure domain moves data — potentially a large fraction, since the constraint changes for every PG.

# 1. verify the topology
ceph osd tree

# 2. build and test the rule offline
ceph osd getcrushmap -o /tmp/cm.bin
crushtool -d /tmp/cm.bin -o /tmp/cm.txt
# add rule
crushtool -c /tmp/cm.txt -o /tmp/cm.new
crushtool -i /tmp/cm.new --test --rule 1 --num-rep 3 --show-bad-mappings

# 3. throttle recovery before applying
ceph config set osd osd_max_backfills 1
ceph config set osd osd_recovery_sleep 0.05

# 4. apply
ceph osd setcrushmap -i /tmp/cm.new
ceph osd pool set rbd-vms crush_rule rack_rule

# 5. watch
ceph -s

Expect it to take hours to days on a large pool. The cluster serves throughout, at reduced performance.

Verifying it works

for pg in $(ceph pg ls-by-pool rbd-vms | awk 'NR>1 {print $1}' | head -20); do
  osds=$(ceph pg map $pg -f json | jq -r '.up[]')
  racks=$(for o in $osds; do ceph osd find $o -f json | jq -r '.crush_location.rack'; done | sort -u | wc -l)
  echo "$pg racks=$racks"
done

Every PG should report three distinct racks. Any reporting fewer means the rule is not doing what it claims.

Quiz

Production discipline

Compare rack weights before adopting a rack rule — the smallest rack sets the ceiling, and unbalanced racks strand capacity in the larger ones. Aim for size + 1 racks, because with exactly size any rack maintenance degrades the entire pool for its duration. Validate the rule offline with crushtool --test and estimate the movement with osdmaptool before applying, and throttle recovery first, since changing the failure domain relocates a large fraction of the data while clients are being served.

Cross-course references

• Ceph: Part XCIII (Changing CRUSH Topology) for the live migration.
• Ceph: Part IV (Failure Domains) for choosing rack over host.
• Ceph: Part XCVI (Node Maintenance) for working within these constraints.