What RGW multi-site does that block and file mirroring cannot

Advanced⏱ ~18 minradosgw-admin

What you'll learn

Compare the replication model of each Ceph interface
Configure selective replication with sync policy
Build an asymmetric topology with an archive zone
Identify the operations that still require the metadata master

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Choosing a multi-site mechanism per interface is a design decision, and RGW is the only one of the three that accepts writes at both ends.

The three mechanisms side by side

Property	RBD mirroring	CephFS snapshot mirroring	RGW multi-site
Unit of replication	one image	one directory tree	one zone, filtered by policy
Write model	one primary, others read-only	source only	active-active
Direction	one-way per image	one-way	symmetrical or directional
Granularity control	per image	per configured path	per bucket, per prefix, per tag
Failover unit	the image	the mount	DNS plus a period commit
Namespace conflicts possible	no	no	yes — hence a metadata master

Block and file mirroring have exactly one writable copy by construction.
Object storage does not, and everything distinctive about RGW multi-site
follows from that.

Active-active, and what it costs

radosgw-admin sync status
radosgw-admin zonegroup get | python3 -c '
import sys,json
d = json.load(sys.stdin)
print("master:", d.get("master_zone"))
for z in d.get("zones", []):
    print("  %-12s %s" % (z.get("name"), z.get("endpoints")))'

Both zones serve PUT. Two clients writing the same key in different
zones produce a last-writer-wins resolution by object mtime, and the
loser is discarded silently — there is no conflict record.

Operation	Works at a non-master zone
GET, PUT, DELETE on an existing bucket	yes
Multipart upload	yes
Create a bucket	no — requires the metadata master
Create a user or key	no
Change a bucket policy or ACL	yes, replicated as metadata

Selective replication with sync policy

radosgw-admin sync group create --group-id=global --status=allowed
radosgw-admin sync group flow create --group-id=global \
  --flow-id=both-ways --flow-type=symmetrical --zones=site-a,site-b
radosgw-admin sync group pipe create --group-id=global --pipe-id=all \
  --source-zones='*' --dest-zones='*'
radosgw-admin period update --commit

# then enable only the buckets that need it
radosgw-admin sync group create --bucket=acme-data --group-id=acme \
  --status=enabled
radosgw-admin sync group pipe create --bucket=acme-data --group-id=acme \
  --pipe-id=acme-pipe --source-zones='*' --dest-zones='*'
radosgw-admin period update --commit

# what will actually flow for one bucket
radosgw-admin sync info --bucket=acme-data

Group status	Effect
`enabled`	the pipes in the group replicate
`allowed`	replication is permitted but only where a lower level enables it
`forbidden`	replication is blocked regardless of lower-level policy

Asymmetric topologies

# a receive-only zone: data arrives, nothing leaves
radosgw-admin sync group flow create --group-id=global \
  --flow-id=to-dr --flow-type=directional \
  --source-zone=site-a --dest-zone=site-dr

# an archive zone keeps every version of every object it receives
radosgw-admin zone modify --rgw-zone=site-archive --tier-type=archive
radosgw-admin period update --commit

An archive zone versions objects on arrival even when the source bucket
is unversioned, so a deletion at the source leaves the archived copy
intact. It is the one topology in Ceph where replication is not a
faithful copy of the source.

Quiz

Knowledge check · 4 questions

Q1. Why can RGW multi-site accept writes at both zones while RBD mirroring cannot?

RGW replicates faster than RBD mirroring, so the window in which two sites can write conflicting data is short enough to ignore RGW zones are fully independent clusters that share no state at all, so a write accepted at one zone never has to be reconciled against the other S3 objects are independent keys with no shared address space, so only the bucket and user namespace needs serialising through a master RBD lacks a journal in snapshot mirroring mode, and without one there is no record from which divergent writes could be merged
Q2. A sync group with status `allowed` causes the buckets it covers to replicate.

True False
Q3. Replicate only two of 140 buckets to a second site.

A zonegroup has two zones and 140 buckets. Only `acme-data` and `audit-logs` justify the cost of a second copy. The current configuration replicates everything.
Q4. What does an archive zone do that a normal secondary zone does not?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Set the zonegroup-level sync group to allowed and enable replication per bucket — replicating everything by default is how a second site ends up holding 140 buckets when two matter. Include bucket creation in the availability check for secondary zones; it fails independently of object access when the metadata master is down.

Cross-course references

Kubernetes: multi-primary writes need a conflict rule, and last-writer-wins discards silently
Linux: shared mutable state is what forces serialisation, not the volume of traffic