Stretch mode: what enabling it actually changes

Advanced⏱ ~18 mincephcrushtool

What you'll learn

State the prerequisites stretch mode enforces
Enable stretch mode with the required CRUSH rule and tiebreaker
Interpret degraded stretch mode during a site outage
Return the cluster to healthy stretch mode after recovery

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

A datacentre failure domain alone does not stop both sites from serving independently after a partition. Stretch mode is the machinery that does.

What a plain two-site CRUSH rule leaves unsolved

Problem	Plain CRUSH rule	Stretch mode
Monitors split evenly, no quorum	unsolved	tiebreaker monitor in a third location
PG goes active with all copies on one side	possible	blocked by the peering barrier
Surviving site refuses I/O at `min_size`	likely	`min_size` drops automatically
Both sites believe they are authoritative	possible	prevented

ceph mon dump | grep -E 'election_strategy|tiebreaker|stretch'

Enabling it

# 1. label the monitors and OSD hosts with their site
ceph mon set_location ceph-a-mon1 datacenter=dc-a
ceph mon set_location ceph-b-mon1 datacenter=dc-b
ceph mon set_location ceph-t-mon1 datacenter=dc-tiebreaker
ceph osd crush move ceph-a-01 datacenter=dc-a

# 2. connectivity-based elections, which stretch mode requires
ceph mon set election_strategy connectivity

# 3. a rule that takes two hosts from each datacentre
ceph osd getcrushmap -o /tmp/cm && crushtool -d /tmp/cm -o /tmp/cm.txt

rule stretch_rule {
        id 1
        type replicated
        step take default
        step choose firstn 0 type datacenter
        step chooseleaf firstn 2 type host
        step emit
}

crushtool -c /tmp/cm.txt -o /tmp/cm.new && ceph osd setcrushmap -i /tmp/cm.new
ceph mon enable_stretch_mode ceph-t-mon1 stretch_rule datacenter

What it forces

Property	Value under stretch mode
Monitors	5: two per site plus the tiebreaker
Pool type	replicated only — erasure coding is not supported
Pool size	4
Pool `min_size`	2, one commit from each site
Peering	a PG needs OSDs from both sites to go active
Write latency	includes the inter-site round trip, always

ceph osd pool get vms size min_size crush_rule
ceph -s

Losing a site

Monitors lose contact with dc-b. The tiebreaker plus dc-a form quorum.
The cluster enters degraded stretch mode: min_size drops to 1 for the
surviving site and I/O continues on two copies in dc-a.

ceph -s | grep -E 'HEALTH|stretch|osds'
ceph health detail | head -20

Signal	Meaning
`degraded stretch mode`	one site is out; I/O continues on the survivor
`recovery stretch mode`	the site returned and backfill is running
PGs `undersized` but active	expected while a site is down
Quorum of 3 of 5	tiebreaker plus the surviving site

Coming back

# when the failed site returns, recovery starts once its OSDs are up
ceph osd tree | grep -c 'up'
ceph -s | grep -E 'recovery|backfill'

# force the transitions only if the cluster does not make them itself
ceph osd force_recovery_stretch_mode --yes-i-really-mean-it
ceph osd force_healthy_stretch_mode --yes-i-really-mean-it

# replace a lost tiebreaker
ceph mon set_new_tiebreaker ceph-t-mon2

Why the peering barrier matters more than the monitor tiebreaker

The tiebreaker prevents two monitor quorums; it does not prevent a placement group from going active with every copy on one side. Without the barrier, a PG whose acting set happened to hold two copies in dc-a could serve reads and writes during a partition while dc-b, holding the other two, did the same — producing two divergent histories for the same PG that no recovery can merge. Stretch mode makes peering require OSDs from both sites in normal operation and relaxes that requirement only when the monitors have collectively agreed a site is gone. The agreement is what makes the relaxation safe, which is why forcing the mode transitions by hand while the peer site is merely unreachable is the one action that can genuinely split the data.

Quiz

Knowledge check · 4 questions

Q1. What does the stretch mode peering barrier prevent that a tiebreaker monitor does not?

Monitors forming two independent quorums, one on each side of the failed link, each believing it is authoritative Clients writing to the wrong site, since each client session is pinned to the OSDs in its own data centre by the rule Excessive inter-site bandwidth use, by keeping recovery and backfill traffic local to whichever site holds the primary A placement group going active with every copy on one side of a partition, which would produce two unmergeable histories
Q2. An erasure-coded pool can be used under stretch mode to reduce its 4x raw cost.

True False
Q3. Bring a stretch cluster back to healthy after a site outage.

dc-b lost power for six hours. The cluster ran in degraded stretch mode on dc-a plus the tiebreaker. Power is restored and the dc-b OSDs are booting.
Q4. What independence does a stretch cluster tiebreaker monitor require, and what does it not require?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Verify the tiebreaker’s independence against the real power and transit paths, not the rack diagram — a tiebreaker sharing a dependency with either site removes the protection it exists to provide. Let the cluster make its own stretch mode transitions; forcing them while a site is merely unreachable is the action that can split the data.

Cross-course references

Kubernetes: an etcd quorum member in the same failure domain provides no arbitration
Linux: fencing decisions belong to the cluster consensus, not to the operator at the console