When more domains fail than the rule allows

Advanced⏱ ~18 mincephcrushtool

What you'll learn

Compute the fraction of PGs affected by a two-domain loss
Prioritise recovery of the pools that matter
Sequence partial restoration for maximum effect
Test a CRUSH rule against the expected failure

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Losing two failure domains out of three is a total outage and losing two out of six is a 20% outage, and the difference is decided years earlier by how many domains were built.

The arithmetic

python3 -c '
from math import comb
print("racks   PGs left at one copy after losing two racks")
for r in range(3, 11):
    print("%5d   %6.1f%%" % (r, 100.0 * (r - 2) / comb(r, 3)))'

racks   PGs left at one copy after losing two racks
    3    100.0%
    4     50.0%
    5     30.0%
    6     20.0%
    8     10.7%
   10      6.7%

Racks	Guaranteed tolerance	Two-rack loss
3	one rack	every PG inactive
4	one rack	half the PGs inactive
6	one rack	a fifth inactive
10	one rack	a fifteenth inactive

The guaranteed tolerance never changes. What changes is whether the
second failure is an outage or an incident.

While most PGs are inactive

ceph osd pool set cephfs-metadata recovery_priority 5
ceph pg force-recovery 4.a1 4.b7
ceph pg cancel-force-recovery 4.a1 4.b7

Priority	Reason
CephFS or RGW metadata pools	one small pool gates an entire service
The `.mgr` pool	dashboard, orchestrator, and modules depend on it
Pools backing running workloads	shortest path to restored service
Cold or archival pools	last, deliberately

Partial restoration beats waiting

ceph osd tree down
ceph osd in 61 62 63
ceph pg dump_stuck inactive | wc -l

Every OSD returned from a failed domain takes its share of PGs from one
available copy to two, which is at or above min_size. Those PGs serve
again immediately, before any object is copied.

Testing the design afterwards

ceph osd getcrushmap -o /tmp/crush.map
crushtool -i /tmp/crush.map --test --rule 1 --num-rep 3 \
  --show-utilization --min-x 0 --max-x 1023

crushtool -i /tmp/crush.map --test --rule 1 --num-rep 3 \
  --show-mappings --min-x 0 --max-x 9

Why returning one OSD from a failed domain helps far more than its capacity share

When two domains are down, the affected PGs each hold one available copy and sit below min_size, refusing client I/O while their data is fully intact. Restoring one OSD does not need to copy anything to help: as soon as it peers, every PG that had a replica on it has two available members, which is at or above min_size, and those PGs go active immediately with recovery running behind them. A single returned OSD in a rack of twenty therefore restores roughly a twentieth of that rack’s inactive PGs within seconds. This is why powering up part of a failed rack, or moving a handful of disks into working chassis, buys service back far faster than waiting for the rack to be repaired properly — availability returns with peering, not with backfill.

Quiz

Knowledge check · 4 questions

Q1. A cluster has six racks, size 3, and a rack failure domain. Two racks are lost. Roughly what fraction of PGs is left at a single copy?

All of them Half A third A fifth
Q2. Adding a fourth and fifth rack to a three-rack cluster with size 3 converts a two-rack failure from a total outage into a partial one.

True False
Q3. Restore service after losing two of five racks.

Two racks are down on a power fault. About 30% of PGs are inactive. CephFS is unavailable to users and the electrical work will take six hours.
Q4. Why does building more failure domains than the replica count matter if tolerance stays the same?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Design for the second domain failure, not only the first — the guaranteed tolerance is unchanged by extra domains but the blast radius falls sharply. During the incident, restore any part of a failed domain first; PGs return to service at peering time rather than waiting for backfill.

Cross-course references

Kubernetes: three zones with three replicas has no partial failure mode either
Linux: redundancy sized exactly to the replica count leaves no graceful degradation