Eventual consistency across zones

Expert⏱ ~18 minradosgw-admin

What you'll learn

Explain the consistency model across zones
Measure and monitor replication lag
Describe conflict resolution in active-active
Design applications against the model

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Within a zone RGW is strongly consistent. Across zones it is not, and applications written against the single-zone behaviour break in ways that are intermittent and load-dependent. The model has to be a design input, not a discovery.

The model

Scope	Consistency
Within a zone	strong — a write is visible immediately after acknowledgement
Across zones	eventual — visible after replication completes
Metadata across a zonegroup	eventual, but usually fast
Bucket creation	eventual across zones

An object written at site-a is durable at site-a immediately and appears at site-b after replication — seconds normally, longer under load or after an outage.

Measuring lag

radosgw-admin sync status
radosgw-admin data sync status --source-zone=site-a
radosgw-admin bucket sync status --bucket=data

# a direct measurement
aws --endpoint-url $SITE_A s3 cp /tmp/probe s3://synctest/probe-$(date +%s)
# poll site-b until it appears, and record the delay

A synthetic probe object written periodically at one site and timed for arrival at the other is the most useful lag metric, because it measures end-to-end what applications experience.

Conflict resolution

In active-active, two clients can write the same key at different sites before either replicates. Resolution is last-writer-wins by timestamp — one version survives and the other is discarded, silently.

The practical implications:

Do not write the same key at two sites concurrently
Route writes for a given key or bucket to one site where possible
Applications requiring coordination must implement it themselves

Designing against the model

Requirement	Approach
Read-after-write	read from the site you wrote to
Global uniqueness	generate keys with a site component
Consistent listing	list at the writing site
Cross-site coordination	do not — use a single site for that workload
Disaster recovery	acceptable lag defines the RPO

The read-after-write case is the one that catches applications: a client writing at site-a and reading through a global DNS name that resolves to site-b sees a 404 for a bounded period.

Quiz

Knowledge check · 4 questions

Q1. Two clients write the same key at different zones before replication occurs. What happens?

Last-writer-wins by timestamp — one version survives and the other is discarded silently Both versions are retained as separate object versions, and the application chooses between them The write fails at the second site with a 409 once the sync log detects the concurrent key The zones diverge permanently, since sync only ever applies a key it has not already seen
Q2. A client writing at one zone and immediately reading through a global DNS name may receive a 404.

True False
Q3. Design an application for an active-active multisite deployment.

An application writes objects and immediately reads them back to verify. It will run at two sites with a global DNS endpoint balancing between them. The team assumes the storage behaves as a single system.
Q4. Why can replication lag be stable for a long period and then grow rapidly?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Treat the cross-zone consistency model as a design input for any application spanning sites, and measure real lag with a synthetic probe rather than inferring it from sync status. Alert on lag as a trend rather than an absolute, since the growth accelerates once sync throughput is exceeded.

Cross-course references

Kubernetes: multi-cluster deployments face identical read-after-write constraints
Linux: asynchronous database replication has the same lag and conflict characteristics