Moving buckets in the hierarchy

Why this matters in production

Moving a bucket changes placement for every PG that could map through it, and the volume depends on what the move changes about the selection.

The move commands

# a host into a rack
ceph osd crush move ceph-07 rack=rack4

# a rack under a different root
ceph osd crush move rack4 root=default

# an individual OSD to a different host — rarely correct
ceph osd crush move osd.84 host=ceph-08

The third is almost always wrong: an OSD’s CRUSH host should match its physical host, and moving it makes the map describe something untrue.

ceph osd crush tree
ceph osd find 84

Predicting the movement

# compute it before committing
ceph osd getcrushmap -o /tmp/crush.bin
crushtool -d /tmp/crush.bin -o /tmp/crush.txt
# edit the move into /tmp/crush.txt
crushtool -c /tmp/crush.txt -o /tmp/crush-new.bin
crushtool -i /tmp/crush.bin --compare /tmp/crush-new.bin --rule 1 --num-rep 3

The comparison gives the exact PG count that changes.

Restructuring the hierarchy

Introducing a rack level to a host-only hierarchy:
  create the rack buckets
  move each host into its rack
  change the rule to use the rack failure domain

ceph osd crush add-bucket rack1 rack
ceph osd crush add-bucket rack2 rack
ceph osd crush add-bucket rack3 rack
for r in rack1 rack2 rack3; do ceph osd crush move $r root=default; done

ceph osd crush move ceph-01 rack=rack1
ceph osd crush move ceph-02 rack=rack2
ceph osd crush move ceph-03 rack=rack3

Moving hosts into racks alone moves little — the hosts are still under the
same root and the rule still selects by host.
Changing the rule to rack is what moves the data.

Separating the two means the structural change and the placement change can be staged.

# Substitute the pool you are moving before running:
POOL=rbd-vms

ceph osd crush rule create-replicated rack-rule default rack
ceph osd pool set "$POOL" crush_rule rack-rule

Verifying

ceph osd crush tree
ceph osd tree
ceph -s | grep misplaced

# every host is where it should be
ceph osd crush tree --format json | python3 -c '
import sys, json
def walk(n, parent=None):
    if n.get("type") == "host": print(n["name"], "under", parent)
    for c in n.get("children", []):
        if isinstance(c, dict): walk(c, n["name"])
for r in json.load(sys.stdin)["nodes"]:
    if r.get("type") == "root": walk(r)'

Quiz

Production discipline

Stage a hierarchy restructure: move the hosts into their buckets first, which moves little, then change the rule, which moves a lot. Never place an OSD in a CRUSH host different from its physical one — every failure domain guarantee becomes false and nothing detects it.

Cross-course references

• Kubernetes: topology labels must correspond to physical reality to mean anything
• Linux: staging structural and behavioural changes separately makes each verifiable