Switches as failure domains

Why this matters in production

CRUSH protects against the failure domains you tell it about. A top-of-rack switch that takes twenty hosts offline is a failure domain whether or not it appears in ceph osd tree, and the mismatch between physical topology and CRUSH hierarchy is one of the more common ways a well-designed cluster fails badly.

The three topologies

Single ToR per rack. Simple and cheap. The switch is a single point of failure for every host in the rack. Ceph must treat the rack as a failure domain — which it should anyway for power reasons.

Dual ToR with MLAG. Two switches presenting as one to LACP bonds. A switch failure removes one bond member per host and Ceph notices nothing. This is the standard production design.

Dual ToR, independent, no MLAG. Each host has separate links to two switches, in active-backup or with separate addressing. More complex, avoids MLAG’s shared control plane, requires more care in Ceph configuration.

Aligning CRUSH with the network

The rule: every level of your physical topology that can fail as a unit should be a CRUSH bucket.

ceph osd crush add-bucket rack1 rack
ceph osd crush add-bucket rack2 rack
ceph osd crush add-bucket rack3 rack
ceph osd crush move rack1 root=default
ceph osd crush move ceph-osd-01 rack=rack1

ceph osd crush rule create-replicated by-rack default rack
ceph osd pool set rbd-vms crush_rule by-rack

ceph osd tree

If a switch serves exactly one rack, the rack bucket covers it. If a switch spans racks — a leaf serving half of two racks — then the switch is a failure domain that the rack hierarchy does not represent, and you need a bucket type that matches the switch boundary.

Counting domains

For size 3 with failure_domain=rack you need at least 3 racks, and meaningfully more for recovery headroom. For EC k+m you need k+m.

A common mistake: two racks with failure_domain=rack on a size-3 pool. CRUSH cannot place three replicas in two racks, so the pool never goes clean. The rule silently produces undersized PGs rather than an error at pool creation.

Spine-leaf

At scale, leaf switches serve racks and spine switches interconnect them. The failure domains become:

• Leaf — takes out its rack; covered by a rack bucket
• Spine — reduces inter-rack bandwidth; with several spines, degradation rather than failure
• Uplink oversubscription — not a failure, but a throughput ceiling that recovery traffic will find

Quiz

Production discipline

Draw the physical topology and the CRUSH hierarchy side by side during design review and confirm every unit that can fail together appears in both. Review uplink oversubscription against recovery traffic whenever the failure domain changes — the two are coupled and are usually reviewed by different teams.

Cross-course references

• Kubernetes: zone and region topology labels serve the same alignment purpose
• Linux: cluster manager fencing domains have the same requirement to match physical reality