Skip to main content
RunBook Academy

CephXLVII · RGW High AvailabilityRGW High Availability

Setting up RGW multisite

Expert⏱ ~19 minradosgw-admin

What you'll learn

  • Configure a realm, zonegroup, and master zone
  • Add a secondary zone and start replication
  • Verify sync is working
  • Recognise the common setup errors

Prerequisites

  • T
  • w
  • o
  • i
  • n
  • d
  • e
  • p
  • e
  • n
  • d
  • e
  • n
  • t
  • C
  • e
  • p
  • h
  • c
  • l
  • u
  • s
  • t
  • e
  • r
  • s
  • ,
  • n
  • e
  • t
  • w
  • o
  • r
  • k
  • c
  • o
  • n
  • n
  • e
  • c
  • t
  • i
  • v
  • i
  • t
  • y
  • b
  • e
  • t
  • w
  • e
  • e
  • n
  • t
  • h
  • e
  • i
  • r
  • g
  • a
  • t
  • e
  • w
  • a
  • y
  • e
  • n
  • d
  • p
  • o
  • i
  • n
  • t
  • s
  • ,
  • a
  • n
  • d
  • a
  • m
  • a
  • i
  • n
  • t
  • e
  • n
  • a
  • n
  • c
  • e
  • w
  • i
  • n
  • d
  • o
  • w
  • o
  • n
  • t
  • h
  • e
  • p
  • r
  • i
  • m
  • a
  • r
  • y
  • .

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Not yet marked complete on this device.

Why this matters in production

Multisite setup is a sequence where each step depends on the previous one having been committed. Skipping a period commit or getting the endpoint wrong produces a configuration that looks correct and does not replicate, which is a frustrating place to end up.

On the primary cluster

# 1. realm
radosgw-admin realm create --rgw-realm=global --default

# 2. zonegroup, with the endpoint clients and peers will use
radosgw-admin zonegroup create --rgw-zonegroup=eu \
    --endpoints=https://rgw-site-a.example.com --rgw-realm=global --master --default

# 3. zone
radosgw-admin zone create --rgw-zonegroup=eu --rgw-zone=site-a \
    --endpoints=https://rgw-site-a.example.com --master --default

# 4. system user for replication
radosgw-admin user create --uid=sync --display-name="Sync User" \
    --system --access-key=$SYNC_KEY --secret=$SYNC_SECRET

radosgw-admin zone modify --rgw-zone=site-a \
    --access-key=$SYNC_KEY --secret=$SYNC_SECRET

# 5. commit
radosgw-admin period update --commit
ceph orch restart rgw.default

The --system user is what the secondary authenticates as to pull data. Its credentials must be identical on both sides.

On the secondary cluster

# 1. pull the realm configuration from the primary
radosgw-admin realm pull --rgw-realm=global \
    --url=https://rgw-site-a.example.com \
    --access-key=$SYNC_KEY --secret=$SYNC_SECRET

radosgw-admin period pull --url=https://rgw-site-a.example.com \
    --access-key=$SYNC_KEY --secret=$SYNC_SECRET

# 2. create the secondary zone
radosgw-admin zone create --rgw-zonegroup=eu --rgw-zone=site-b \
    --endpoints=https://rgw-site-b.example.com \
    --access-key=$SYNC_KEY --secret=$SYNC_SECRET

# 3. commit and start
radosgw-admin period update --commit
ceph orch apply rgw default --realm=global --zone=site-b

Verifying

radosgw-admin sync status
          realm 8f3a... (global)
      zonegroup 2b1c... (eu)
           zone 9d4e... (site-b)
  metadata sync syncing
                full sync: 0/64 shards
                incremental sync: 64/64 shards
      data sync source: 7c2a... (site-a)
                        syncing
                        full sync: 0/128 shards
                        incremental sync: 128/128 shards

incremental sync: N/N shards on both metadata and data means replication is established and caught up.

# functional test
aws --endpoint-url https://rgw-site-a.example.com s3 mb s3://synctest
aws --endpoint-url https://rgw-site-a.example.com s3 cp file s3://synctest/file
sleep 30
aws --endpoint-url https://rgw-site-b.example.com s3 ls s3://synctest/

Common setup errors

ErrorCause
realm pull failsendpoint unreachable, or wrong system user credentials
Metadata syncs, data does notzone endpoints not set correctly
Nothing syncsperiod not committed, or gateways not restarted
Sync starts then stallsnetwork between endpoints, or clock skew
Buckets appear but are emptydata sync not started; check data sync status

The endpoint values are the most common mistake: they must be reachable from the other cluster, not just from clients.

Quiz

Knowledge check · 4 questions

  1. Q1. Metadata syncs correctly between zones but no object data replicates. What should you check first?

  2. Q2. The system user's access and secret keys must be identical in both zones' configurations.

  3. Q3. Verify a newly-configured multisite deployment.

    A multisite deployment has been configured following the documentation. `radosgw-admin sync status` on the secondary reports metadata sync as syncing with full sync 0/64 shards, but data sync reports "init".

  4. Q4. Why does a credential mismatch between zones surface as a stalled sync rather than an authentication error?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Verify zone endpoints from the peer cluster rather than from a client before declaring a multisite setup complete; the partial success it produces is the most common and most confusing failure. Run a functional write-and-read test across sites as the acceptance criterion, not sync status alone.

Cross-course references

  • Kubernetes: cross-cluster federation has the same endpoint-reachability-from-peer requirement
  • Linux: any replication setup depends on the peer being able to reach the advertised address