Chassis, room, and row — extending the hierarchy

Intermediate⏱ ~15 minceph

What you'll learn

Identify when chassis-level separation is required
Choose an appropriate hierarchy depth for a facility
Build multi-level hierarchies correctly
Avoid unnecessary hierarchy depth

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

The bucket types between host and root exist for real physical boundaries. Using the right ones makes a map that describes the building; using all of them makes a map nobody can read.

The types worth using

Type	Real boundary	Use when
`chassis`	shared backplane, PSUs, fans	blade or multi-node enclosures
`rack`	PDU, top-of-rack switch, cooling	almost always, above 3 racks
`pdu`	one power strip	power does not align with racks
`room`	cooling plant, fire suppression	multi-room facilities
`row`	distribution power, aggregation switching	very large facilities
`datacenter`	building, utility feed	multi-site

The chassis case

Multi-node chassis — four nodes in 2U sharing a backplane and a pair of PSUs — are common and are the clearest case for a bucket level that people forget.

ceph osd crush add-bucket chassis-1 chassis
ceph osd crush move chassis-1 rack=rack-a
ceph osd crush move ceph-01 chassis=chassis-1
ceph osd crush move ceph-02 chassis=chassis-1
ceph osd crush move ceph-03 chassis=chassis-1
ceph osd crush move ceph-04 chassis=chassis-1

Then a rule at chassis level, or above:

step chooseleaf firstn 0 type chassis

Without this, four hostnames describe one failure boundary and a size 3 host rule places all three copies inside one enclosure for some fraction of PGs.

Stop at the deepest level you have enough of

A hierarchy is only useful at levels where you have at least size instances. Defining rooms when you have two rooms adds a level no rule can use for a size 3 pool.

ceph osd tree | grep -c '^-.*rack'
ceph osd tree | grep -c '^-.*room'

Count the buckets at each level before defining it. If the count is below size, the level exists only for documentation — which is a legitimate reason, but it should be a conscious one rather than an assumption that deeper is safer.

A map with unused intermediate levels is harder to read, harder to validate, and adds nothing to placement.

Building multi-level hierarchies

Order matters: create buckets, place them in their parents from the top down, then move hosts in.

ceph osd crush add-bucket row-1 row
ceph osd crush move row-1 root=default
ceph osd crush add-bucket rack-a1 rack
ceph osd crush move rack-a1 row=row-1
ceph osd crush add-bucket chassis-1 chassis
ceph osd crush move chassis-1 rack=rack-a1
ceph osd crush move ceph-01 chassis=chassis-1
ceph osd tree

Each move of a populated bucket causes data movement, which is why this belongs at build time.

Why a rule at a high level implicitly satisfies lower ones

A rule with chooseleaf firstn 0 type rack selects distinct racks and then descends to an OSD in each. Since a host belongs to exactly one rack, distinct racks implies distinct hosts — and distinct chassis, if chassis sit inside racks.

So a rack-level rule provides host and chassis separation for free. There is no need to write a rule that separates on several levels at once for the common case; picking the highest level you can satisfy gives everything below it.

Where multi-level selection is genuinely needed is when the counts differ: “two racks, three hosts within each” for an erasure-coded pool that has more chunks than racks. That is a different requirement from separation, and it is why choose and chooseleaf can be combined.

The practical rule is: for replicated pools, one select step at the highest satisfiable level. For EC pools with more chunks than domains, multi-level selection with the tolerance arithmetic done explicitly.

Choosing depth

3-10 hosts, one rack: root → host.
Multi-node chassis: add chassis.
3+ racks: root → rack → host.
Power not aligned to racks: add pdu.
Multi-room or multi-row at scale: add those levels only if you have at least size of them.

Quiz

Knowledge check · 4 questions

Q1. A cluster uses a rack-level failure domain. Does it also need a chassis-level rule to protect against chassis failure?

Yes, a rule constrains only the level it names, so each boundary needs its own select step Yes, but only where chassis buckets exist in the map, since CRUSH ignores levels it cannot see No, since a chassis sits inside one rack, distinct racks implies distinct chassis No, because host separation already covers it — a chassis failure is just several host failures
Q2. Defining more bucket levels in a CRUSH hierarchy makes a cluster more resilient.

True False
Q3. A cluster of 32 hosts in eight 4-node chassis across four racks uses a host-level rule. Recommend the hierarchy.

Eight multi-node chassis, four per rack in two racks, and eight standalone 2U servers spread across two further racks — 32 hosts total in four racks. Pool size 3, min_size 2, currently type host. The team wants protection against both chassis and rack failure without over-complicating the map.
Q4. Explain why multi-node chassis are a commonly missed failure domain and what the consequence is.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Count the buckets you have at each level before defining it, and stop at the deepest level with at least size instances — unused levels make maps harder to read and add nothing. Write one select step at the highest satisfiable level for replicated pools, since separation there implies separation below. Define chassis buckets whenever multi-node enclosures are present, because four hostnames sharing a backplane are one failure boundary and only you can tell CRUSH that.

Cross-course references

Ceph: Part IV (Failure Domains) for identifying the boundaries.
Ceph: Part XIII (CRUSH Fundamentals) for multi-level select steps.
Ceph: Part XV (CRUSH Maps and Rules) for writing the rules.