Diagnosing multisite sync problems

Expert⏱ ~19 minradosgw-admin

What you'll learn

Interpret sync status output in detail
Distinguish lag from a stall from an error
Diagnose per-bucket sync failures
Recover a stalled or diverged sync

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Sync problems are silent — the primary serves correctly and only the secondary is affected, so nothing alerts unless you are watching sync status. By the time someone notices, the backlog can be large.

Reading sync status

radosgw-admin sync status

      data sync source: 7c2a... (site-a)
                        syncing
                        full sync: 0/128 shards
                        incremental sync: 128/128 shards
                        data is behind on 14 shards
                        behind shards: [3,7,12,...]
                        oldest incremental change not applied: 2026-08-18T04:12:33

Field	Meaning
`full sync: 0/N`	initial sync complete
`incremental sync: N/N`	all shards in incremental mode
`data is behind on N shards`	current backlog
`oldest incremental change not applied`	the actual lag, as a timestamp

The timestamp is the number that matters — it is the lag in wall-clock terms.

Lag, stall, or error

radosgw-admin sync status > /tmp/a; sleep 300
radosgw-admin sync status > /tmp/b; diff /tmp/a /tmp/b

Observation	Diagnosis
Behind-shard count falling	lag, catching up
Count stable, timestamp advancing	keeping pace at a steady offset
Count and timestamp unchanged	stalled
Error entries in the status	a specific failure

Working a stall

# is the source reachable from here?
curl -s -o /dev/null -w '%{http_code}\n' https://rgw-site-a.example.com/

# are the credentials still matching?
radosgw-admin zone get --rgw-zone=site-a | jq '.system_key'
radosgw-admin zone get --rgw-zone=site-b | jq '.system_key'

# per-shard detail
radosgw-admin data sync status --source-zone=site-a --shard-id=3

# gateway logs on the syncing side
journalctl -u ceph-radosgw@* --since '1 hour ago' | grep -iE 'sync|error'

Cause	Signature
Endpoint unreachable	connection errors in the log
Credential mismatch	signature errors, sync stalled at start
Destination pool full	write errors on the secondary
Source object missing	a specific shard erroring repeatedly
Clock skew	intermittent signature errors

Per-bucket problems

radosgw-admin bucket sync status --bucket=data
radosgw-admin bucket sync run --bucket=data

A single bucket failing to sync while others succeed usually means an object in it cannot be fetched — check the source for that object and the sync error log:

radosgw-admin sync error list
radosgw-admin sync error trim --start-date=2026-08-01

The error list accumulates and is worth reading before assuming a stall is mysterious.

Forcing a resync

# for one bucket
radosgw-admin bucket sync init --bucket=data --source-zone=site-a
radosgw-admin bucket sync run --bucket=data

# for the whole data sync — expensive
radosgw-admin data sync init --source-zone=site-a

sync init restarts from a full sync, re-fetching everything. On a large bucket or a whole zone that is a significant operation, so it is a last resort after the specific cause has been investigated.

Quiz

Knowledge check · 4 questions

Q1. Which field in sync status expresses replication lag in wall-clock terms?

The behind-shard count, which counts one shard per second of lag The full sync shard count, which stalls at the lagging shard The incremental shard count, which falls as the backlog grows `oldest incremental change not applied` — a timestamp
Q2. A healthy sync status guarantees the secondary zone holds every object from the primary.

True False
Q3. Investigate a secondary zone missing objects despite healthy sync status.

An audit finds that the secondary zone is missing about 200 objects that exist at the primary. `radosgw-admin sync status` reports data sync as syncing with no shards behind.
Q4. Why does a sync worker skip a failing object rather than blocking?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Alert on the oldest incremental change not applied timestamp against your RPO, since sync problems are otherwise silent — the primary serves correctly throughout. Include sync error list in routine health checks, because a healthy sync status does not mean a complete secondary.

Cross-course references

Kubernetes: a controller that logs and skips failing objects leaves the same silent gap
Linux: replication tools that continue past individual failures need the same error-log discipline