Physical topology — mapping racks, rows, and chassis into CRUSH

Why this matters in production

CRUSH acts on the hierarchy it is given. If that hierarchy is a flat list of hosts under default, Ceph can only protect against host failure regardless of how the hardware is arranged. Describing the physical layout is the prerequisite for protecting against anything larger.

The bucket types available

Ceph ships a standard set of bucket types, from osd at the leaves upward:

0  osd
1  host
2  chassis
3  rack
4  row
5  pdu
6  pod
7  room
8  datacenter
9  zone
10 region
11 root

Not all need to be used. Use the ones that correspond to real shared failure causes in your facility and skip the rest — a hierarchy with empty intermediate levels adds nothing and makes rules harder to read.

Building the hierarchy

Create the buckets and move hosts into them:

ceph osd crush add-bucket rack-a1 rack
ceph osd crush add-bucket rack-a2 rack
ceph osd crush move rack-a1 root=default
ceph osd crush move rack-a2 root=default
ceph osd crush move ceph-node-01 rack=rack-a1
ceph osd crush move ceph-node-02 rack=rack-a2

Then verify:

ceph osd tree

The output is the authoritative statement of what Ceph believes the topology is. Read it against the physical layout, physically, once — this is the check that catches the errors nothing else will.

Choosing the depth

The rule of thumb is to stop at the largest boundary you can actually have size independent instances of. Defining rows when you have two rows does not help a 3-replica pool; it just makes the tree taller.

Verifying against reality

Three checks worth doing before trusting the tree:

ceph osd tree                                # the claimed topology
ceph osd crush rule dump                     # what the rules separate on
ceph osd getcrushmap -o /tmp/cm && crushtool -d /tmp/cm -o /tmp/cm.txt

Then, physically: does rack-a1 in the tree contain the hosts that are in rack A1 in the room? This has to be checked by a person once. Nothing in Ceph can do it, and a transposed pair of hosts produces a cluster that looks correct and is not.

Quiz

Production discipline

Decide the hierarchy at build time and set each host’s location before its OSDs are created, because doing it afterwards is a data-movement event proportional to the data held. Stop the hierarchy at the largest boundary you have at least size instances of. Set osd_crush_update_on_start to false, or use a crush location hook, so a rebooting OSD cannot silently reassign itself. And walk the room once against ceph osd tree — that manual check is the only thing that catches a transposition.

Cross-course references

• Ceph: Part XV (CRUSH Maps and Rules) for editing the map directly.
• Ceph: Part XCIII (Changing CRUSH Topology) for doing this on a live cluster.
• Ceph: Part LX (Recovery Tuning) for throttling the resulting movement.