RGW multi-site replication as a DR mechanism

Advanced⏱ ~18 minradosgw-admin

What you'll learn

Describe the multi-site model
Configure and verify replication
Monitor sync lag
Distinguish replication from backup

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Multi-site gives a live second copy with a low RPO, and it is frequently mistaken for a backup.

The model

Realm — the top-level namespace
  Zonegroup — a set of zones; one is the master
    Zone — an RGW deployment backed by one Ceph cluster

radosgw-admin realm list
radosgw-admin zonegroup list
radosgw-admin zone list
radosgw-admin period get | python3 -c '
import sys,json
d = json.load(sys.stdin)
print("realm:", d.get("realm_name"), " epoch:", d.get("epoch"))'

Concept	Role
Realm	the boundary of a replication configuration
Period	a versioned snapshot of the realm configuration
Zonegroup master	where bucket and user metadata changes are made
Zone	one cluster’s copy of the data
Sync	asynchronous, per bucket and per user

Metadata — users, buckets — flows from the zonegroup master.
Data flows between zones according to the sync policy.

Configuring

# on the primary
radosgw-admin realm create --rgw-realm=acme --default
radosgw-admin zonegroup create --rgw-zonegroup=global --master --default \
  --endpoints=https://rgw-a.example.net
radosgw-admin zone create --rgw-zonegroup=global --rgw-zone=site-a --master \
  --default --endpoints=https://rgw-a.example.net
radosgw-admin period update --commit

# on the secondary, pull the realm
KEY=report.pdf
SECRET=REDACTED
radosgw-admin realm pull --url=https://rgw-a.example.net \
  --access-key=${KEY} --secret=${SECRET}
radosgw-admin zone create --rgw-zonegroup=global --rgw-zone=site-b \
  --endpoints=https://rgw-b.example.net \
  --access-key=${KEY} --secret=${SECRET}
radosgw-admin period update --commit

ceph orch apply rgw acme --realm=acme --zone=site-b --placement=2

Verifying and monitoring

radosgw-admin sync status

          realm ... (acme)
      zonegroup ... (global)
           zone ... (site-b)
  metadata sync syncing
                full sync: 0/64 shards
                incremental sync: 64/64 shards
                metadata is caught up with master
      data sync source: ... (site-a)
                        syncing
                        full sync: 0/128 shards
                        incremental sync: 128/128 shards
                        data is caught up with source

radosgw-admin data sync status --source-zone=site-a
radosgw-admin bucket sync status --bucket=acme-data

Signal	Meaning
“caught up”	replication is current
Shards behind	lag; check the rate
Full sync in progress	initial or recovering
A bucket persistently behind	investigate that bucket specifically
Errors in the sync log	`radosgw-admin sync error list`

radosgw-admin sync error list | head -20
radosgw-admin sync error trim --end-date=$(date -d '7 days ago' +%Y-%m-%d)

Replication is not backup

Threat	Multi-site helps
Site loss	yes — this is its purpose
Cluster loss	yes
Hardware failure	yes
Accidental deletion	no — the delete replicates
Ransomware	no — the encryption replicates
Application corruption	no
Operator error	no

A delete at the primary propagates to the secondary within the sync
interval. Multi-site answers availability, not recoverability.

Quiz

Knowledge check · 4 questions

Q1. Why does RGW multi-site route all metadata changes through the zonegroup master?

To reduce network traffic, since metadata changes would otherwise have to be broadcast to every zone in the realm Bucket and user names are a shared namespace, and serialising changes through one master makes it consistent by construction The master holds the encryption keys for the realm, so any new user's credentials have to be minted there Secondary zones are read-only by design, so a write of any kind has to be directed at the master zone
Q2. RGW multi-site protects against accidental object deletion.

True False
Q3. Assess a multi-site deployment as data protection.

A team has RGW multi-site between two data centres and considers their object data protected.
Q4. What can a secondary zone not do while the zonegroup master is unreachable?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Treat multi-site as availability protection, not recoverability — a deletion at the primary is gone from every zone within the sync interval. Add versioning with object lock, or an independent backup outside the realm, for the deletion and corruption cases.

Cross-course references

Kubernetes: cross-cluster replication of manifests propagates mistakes identically
Linux: synchronous replication answers availability; it never answers recoverability