Choosing the failure domain a second copy must survive

Advanced⏱ ~18 mincephcrushtool

What you'll learn

Enumerate the failure domains a design can span
Compare in-cluster, stretch and external architectures
Place monitor quorum against the site failure you plan for
Identify the failures no in-cluster design survives

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

A copy inside the failure domain you are planning against is not a second copy. Most Ceph DR designs pass the physical test and fail the administrative one.

The domains a design can span

Domain	Spanned by
device	replication or erasure coding
host	a CRUSH rule with a host failure domain
rack, room, power feed	a CRUSH rule with that bucket type
datacentre	stretch mode, or a second cluster
region	a second cluster, or an offsite copy
administrative	only a copy outside this cluster’s control plane

ceph osd crush tree
ceph osd crush rule ls
ceph osd crush rule dump replicated_rule

ceph osd crush rule create-replicated rack-rule default rack
ceph osd pool set rbd crush_rule rack-rule
ceph osd pool get rbd crush_rule

What each architecture buys

Architecture	Survives	RPO	RTO	Crosses the admin boundary
one cluster, host failure domain	drives, hosts	0	seconds	no
one cluster, rack or room domain	a rack, a power feed	0	seconds	no
stretch cluster, two sites plus tiebreaker	a datacentre	0	seconds to minutes	no
two clusters, asynchronous replication	a datacentre, a cluster	minutes	minutes to hours	partly
copy to an independently operated target	all of the above, plus mistakes	hours to days	hours to days	yes

Stretch mode

rule stretch_rule {
  id 1
  type replicated
  step take default
  step choose firstn 0 type datacenter
  step chooseleaf firstn 2 type host
  step emit
}

ceph osd getcrushmap -o /tmp/cm.bin
crushtool -d /tmp/cm.bin -o /tmp/cm.txt
# add the rule, then compile and load it
crushtool -c /tmp/cm.txt -o /tmp/cm.new
ceph osd setcrushmap -i /tmp/cm.new

ceph mon set_election_strategy connectivity
ceph mon enable_stretch_mode mon.e stretch_rule datacenter
ceph mon dump | grep -iE 'stretch|tiebreaker|election'

ceph osd pool get rbd size       # 4
ceph osd pool get rbd min_size   # 2

Two monitors per site plus a tiebreaker in a third location, two data copies per site, and a documented degradation path — healthy stretch mode, degraded stretch mode while a site is down, recovery stretch mode as it returns.

Where every in-cluster design ends

Event	Any CRUSH rule	Stretch mode	External copy
a rack loses power	survives	survives	survives
a datacentre is lost	depends on the rule	survives	survives
a pool is deleted	no	no	survives
a CRUSH edit maps every PG onto one site	no	no	survives
an upgrade corrupts the OSD store format	no	no	survives
ransomware with cluster credentials	no	no	if the target is immutable

Why stretch mode uses size 4 and min_size 2 rather than three copies

With three replicas across two sites, one site necessarily holds two copies and the other holds one — and which site that is varies per PG, because CRUSH distributes independently. Losing the site with two copies leaves a single copy for those PGs, below any safe min_size, so one of the two site failures is survivable and the other is not, on a per-PG basis nobody can enumerate in advance. Four copies, two per site, makes both site failures identical: the survivor holds exactly min_size copies and IO continues without operator intervention. The price is a fourth copy of everything, a third more raw capacity than a three-replica cluster, plus the inter-site bandwidth every write consumes twice — which is the real reason stretch mode is a deliberate architecture rather than a configuration flag.

Quiz

Knowledge check · 4 questions

Q1. Which failure does no in-cluster replication design protect against?

Loss of a rack, unless the CRUSH rule names rack as its failure domain and enough racks exist to satisfy the replica count Loss of a single OSD, because recovery cannot begin until the monitors have marked it out, which the down-out interval delays by ten minutes Loss of a datacentre, which stretch mode addresses for reads only, since a write still requires both sites to acknowledge it An administrative action — a pool deletion, a CRUSH edit or a bad upgrade — which every copy inside the cluster faithfully obeys
Q2. Placing replicas in two datacentres with a CRUSH rule is enough to survive the loss of one datacentre.

True False
Q3. Choose a DR architecture for a two-site estate.

A team has two data centres 40 km apart with a 10 ms round trip, three monitors in site A, two in site B, and a rack-level CRUSH rule. They want to survive the loss of either site.
Q4. What does an externally operated copy protect against that stretch mode does not?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Place monitor quorum against the site failure you are planning for before worrying about data placement — a perfect CRUSH rule cannot help a cluster that cannot reach quorum. Keep an external copy alongside any in-cluster design; stretch mode answers site loss and nothing else.

Cross-course references

Kubernetes: a multi-zone cluster still has one API server and one blast radius for a bad manifest
Linux: redundancy inside a system never protects against the system being told to do the wrong thing