What a failure domain is, and why the answer is never "the disk"

Why this matters in production

Three replicas means the cluster survives two failures — but only if those three replicas cannot fail at the same time for the same reason. A failure domain is the boundary of a shared cause. Everything inside one can die together; that is the definition.

Most real data-loss incidents in replicated storage are not “three disks failed at once”. They are one event that took out three copies because all three lived behind the same power feed, the same switch, or the same hypervisor.

The domains in a single rack

Walk up a rack and the boundaries are visible:

device      one SSD or HDD
host        one server: shares PSU, motherboard, NIC, kernel, OS
chassis     multi-node enclosure: shares backplane and fans
rack        shares PDUs, top-of-rack switch, and often cooling
row         shares distribution power and aggregation switching
datacentre  shares utility feed, cooling plant, physical building
region      shares nothing but weather and network peering

Each level up is a larger blast radius and a rarer event. The design question is which level the cluster must survive, and that answer is determined by the business, not by Ceph.

How CRUSH enforces it

A CRUSH rule names the type of bucket that replicas must be spread across:

rule replicated_rule {
  id 0
  type replicated
  step take default
  step chooseleaf firstn 0 type host
  step emit
}

type host is the failure domain. CRUSH will choose OSDs such that no two replicas of the same PG land on the same host. Change it to type rack and no two replicas share a rack.

flowchart TD
  R[root default] --> R1[rack A]
  R --> R2[rack B]
  R --> R3[rack C]
  R1 --> H1[host a1] --> O1[(osd.0)]
  R2 --> H2[host b1] --> O2[(osd.7)]
  R3 --> H3[host c1] --> O3[(osd.14)]
  O1 -.->|"replica 1"| P[PG 3.1f]
  O2 -.->|"replica 2"| P
  O3 -.->|"replica 3"| P

Where stated redundancy exceeds real redundancy

CRUSH only knows the topology you describe. It has no way to discover that two hosts share a power feed. Common gaps:

• Virtualised hosts on one hypervisor. Three OSD nodes, one physical box.
• Blade or multi-node chassis. Four “hosts” sharing one backplane and one pair of PSUs.
• Single top-of-rack switch. Hosts survive independently; their network does not.
• One PDU. Or two PDUs both fed from the same UPS.
• Multiple OSDs per NVMe device. Four OSDs, one device, one failure.

Each of these produces a cluster that reports the redundancy you configured and delivers less.

The design question

Ask it in this order:

1. What single event must the cluster survive without data loss?
2. Which physical boundary contains that event?
3. Do we have at least size independent instances of that boundary?
4. Does the CRUSH rule name that boundary?

Most clusters answer “a server” and use type host, which is correct for them. The failures come from answering “a rack” in the design document and type host in the rule.

Quiz

Production discipline

Write the failure-domain decision down explicitly: what event the cluster must survive, which physical boundary contains it, whether there are at least size instances of that boundary, and whether the CRUSH rule names it. Audit the map against the physical layout rather than trusting it — CRUSH cannot discover shared power, shared switches, or shared hypervisors, and every one of those turns stated redundancy into a number that is larger than the real one.

Cross-course references

• Ceph: Part XIV (CRUSH Failure Domains) for the rule syntax in depth.
• Ceph: Part CXXII (Small Cluster Risks) for what to do when boundaries are scarce.
• Kubernetes: topology spread constraints solve the same problem for pods.