Pre-check: CRUSH map preparation

Why this matters in production

CRUSH changes move data, and the amount can be computed before anything is committed.

Preparing the buckets

# the bucket must exist before the host moves into it
ceph osd crush add-bucket rack4 rack
ceph osd crush move rack4 root=default
ceph osd crush tree

# and the host, once added, moves in
ceph orch host add ceph-07 10.0.2.27 --labels osd
ceph osd crush move ceph-07 rack=rack4

Ordering: bucket, host, move, then OSDs. Each step before the OSDs exist is free.

Verifying rules place correctly

ceph osd getcrushmap -o /tmp/crush.bin
crushtool -d /tmp/crush.bin -o /tmp/crush.txt

# test a rule against the map, without applying anything
crushtool -i /tmp/crush.bin --test --show-mappings \
  --rule 1 --num-rep 3 --min-x 0 --max-x 10

# and check the rule can be satisfied at all
crushtool -i /tmp/crush.bin --test --rule 1 --num-rep 3 \
  --show-bad-mappings

--show-bad-mappings reports PGs the rule cannot place, which is the check that catches a rule requiring more failure domains than exist.

Estimating the movement

# build the intended map offline
crushtool -c /tmp/crush-new.txt -o /tmp/crush-new.bin

# compare against the current one
crushtool -i /tmp/crush.bin --compare /tmp/crush-new.bin \
  --rule 1 --num-rep 3

rule 1 had 0/10240 mappings that changed

The comparison reports exactly how many PG mappings differ, which is the volume that will move.

Confirming before committing

Before applying:
  the bucket exists and is under the correct parent
  the rule can place every PG (--show-bad-mappings clean)
  the movement volume is known and acceptable
  the cluster is healthy and has capacity for the movement

ceph -s
ceph osd df | sort -k17 -rn | head -3

# apply
ceph osd setcrushmap -i /tmp/crush-new.bin

# or, for simple moves, the direct commands
ceph osd crush move ceph-07 rack=rack4

For a simple host move the direct command is fine; for rule changes the offline test-then-apply cycle is what makes the movement predictable.

Quiz

Production discipline

Run crushtool --compare before any rule change — placement is pseudo-random and the movement volume must be computed rather than reasoned about. Check --show-bad-mappings to catch a rule that cannot place, which otherwise produces permanently undersized PGs with no obvious cause.

Cross-course references

• Kubernetes: dry-run and diff before applying is the same discipline
• Linux: testing a configuration offline before committing it is universal practice