Synchronous cross-site replication and its latency ceiling

Advanced⏱ ~18 mincephradosping

What you'll learn

Explain why a CRUSH rule spanning sites is already synchronous
Compute the write-rate ceiling imposed by inter-site latency
Predict behaviour under a partition
Choose between blocking and losing writes

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Synchronous cross-site replication in Ceph is not a missing feature — it is a CRUSH rule, and its cost is one inter-site round trip on every write.

Where the synchrony comes from

A RADOS write is acknowledged to the client only after every OSD in the
acting set has committed it. Place one acting-set member in another
datacentre and the write is synchronous across sites by construction.

ceph osd crush tree --show-shadow | head -20
ceph osd pool ls detail | grep -E 'pool|crush_rule'

# a rule whose failure domain is the datacentre
ceph osd crush rule create-replicated cross-site default datacenter
ceph osd pool set vms crush_rule cross-site

RBD does not replicate between clusters synchronously — but RBD is not
the layer that replicates. The pool is. An image on a pool whose acting
sets span two datacentres is already written synchronously to both.

The latency ceiling

Separation	One-way	RTT	Ceiling per serialised writer
Same rack	0.05 ms	0.1 ms	~10000 writes/s
Same campus	0.1 ms	0.2 ms	~5000 writes/s
Metro, 40 km	0.4 ms	0.8 ms	~1200 writes/s
Metro, 100 km	1.0 ms	2.0 ms	~500 writes/s
Regional, 500 km	3.5 ms	7 ms	~140 writes/s
Continental, 3000 km	20 ms	40 ms	~25 writes/s

ping -c 100 -q ceph-b-01.example.net | tail -2

# measure it as Ceph experiences it: one outstanding 4 KiB write
rados bench -p vms 60 write -t 1 -b 4096 --no-cleanup
rados -p vms cleanup

Fibre carries light at roughly 200 000 km/s, so 100 km costs 0.5 ms each
way before any switch touches the packet. No tuning changes that number.
Concurrency raises aggregate throughput; it never lowers the latency a
single synchronous writer sees.

Behaviour under a partition

Design	Partition behaviour	Data loss
Synchronous, both sites in the acting set	writes block until quorum and `min_size` are met	none
Asynchronous mirroring	writes continue at the primary	whatever had not shipped
Synchronous with `min_size` lowered to 1	writes continue on one copy	none yet, but no redundancy

ceph osd pool get vms size min_size
ceph -s | grep -E 'health|pgs'

Ceph blocks rather than diverges. A partitioned synchronous pool goes
inactive on the minority side and stays consistent; an asynchronous pair
stays available on both sides and loses the unshipped tail. That is the
whole trade, and it is a choice about which failure you prefer.

Choosing

Requirement	Choose
Zero RPO, tolerant of blocked writes	synchronous, sites under ~2 ms RTT
High write rate per client thread	asynchronous mirroring
Sites more than about 100 km apart	asynchronous mirroring
Database with synchronous commit	measure first; the ceiling is usually binding
Archive or bulk object data	asynchronous, the latency is irrelevant

Quiz

Knowledge check · 4 questions

Q1. What sets the write-rate ceiling for a single serialised writer on a cross-site pool?

The inter-site round-trip time, since each write must commit at the remote OSD before it is acknowledged The aggregate link bandwidth, which bounds how many writes per second can cross between the two sites at peak The number of OSDs in the pool, since each additional OSD adds a parallel path for the write to take The replication factor, because every extra copy adds one more commit the primary has to wait for
Q2. A CRUSH rule that places replicas in two datacentres already makes writes synchronous across both sites.

True False
Q3. Evaluate a proposed cross-site synchronous pool.

Two datacentres 180 km apart, measured RTT 2.4 ms. The proposal is a single replicated pool with a datacentre failure domain, hosting database VMs whose commits are serialised.
Q4. What does Ceph do to a synchronous cross-site pool during a network partition, and how does that differ from asynchronous mirroring?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Measure inter-site RTT and benchmark with one outstanding operation before committing to a synchronous topology — the concurrent number describes the link, and the serialised number describes the workload that will complain. Treat a lowered min_size as an incident action with an owner and a reversal, never as standing configuration.

Cross-course references

Kubernetes: synchronous cross-region control loops inherit the same round-trip floor
Linux: the speed of light in fibre is the one parameter no tuning reaches