RGW recovery targets and failover

Why this matters in production

Object storage failover is a DNS and endpoint change plus a promotion, and the promotion has to be done correctly or the namespace splits.

Computing achievable targets

radosgw-admin sync status

RPO delivered = time since the secondary was last caught up
RTO           = promotion time + endpoint change + client convergence

# how far behind the secondary is
radosgw-admin data sync status --source-zone=site-a | head -20
radosgw-admin bucket sync status --bucket=acme-data

Measuring delivered RPO

# per-bucket sync state, which is where lag actually hides
radosgw-admin bucket list --format json | python3 -c '
import sys,json,subprocess
for b in json.load(sys.stdin)[:20]:
    try:
        out = subprocess.check_output(
            ["radosgw-admin","bucket","sync","status","--bucket",b],
            stderr=subprocess.DEVNULL).decode()
        state = "caught up" if "caught up" in out else "BEHIND"
    except Exception:
        state = "?"
    print("%-32s %s" % (b, state))'

Whole-zone status can report healthy while one high-churn bucket lags
badly. The per-bucket view is the one that matches what a user would
lose.

radosgw-admin sync error list | head -10

Failing over

1. stop writes at the primary, if it is reachable
2. confirm the secondary's sync state
3. promote the secondary zone to master
4. commit the period
5. update DNS or the load balancer to the secondary endpoints
6. confirm clients reach it and can write

# on the secondary
radosgw-admin sync status
radosgw-admin zone modify --rgw-zone=site-b --master --default
radosgw-admin period update --commit

radosgw-admin period get | python3 -c '
import sys,json
d = json.load(sys.stdin)
print("epoch:", d.get("epoch"), " master zone:",
      d.get("period_map",{}).get("zonegroups",[{}])[0].get("master_zone"))'

ceph orch ps --daemon-type rgw
curl -sf https://rgw-b.example.net/ >/dev/null && echo "endpoint OK"

Promoting while the original master is still running and reachable by
clients produces two masters and a split namespace. Confirming the
original is down is part of the procedure, not a preliminary.

Failing back

# the original site returns; it must sync from the current master first
KEY=report.pdf
SECRET=REDACTED
radosgw-admin realm pull --url=https://rgw-b.example.net \
  --access-key=${KEY} --secret=${SECRET}
radosgw-admin period pull --url=https://rgw-b.example.net \
  --access-key=${KEY} --secret=${SECRET}
radosgw-admin sync status

# only once caught up, promote it back
radosgw-admin zone modify --rgw-zone=site-a --master --default
radosgw-admin period update --commit

Failing back before the returned site has caught up discards whatever
was written at the secondary during the outage.

Quiz

Production discipline

Check per-bucket sync status for the buckets that matter — zone-level status can report caught up while one high-churn bucket lags by hours, and that bucket’s lag is its users’ actual RPO. Confirm the old master is genuinely down before promoting; two masters split the namespace.

Cross-course references

• Kubernetes: split-brain on promotion is prevented by fencing, not by assumption
• Linux: aggregate health metrics hide the specific lag that users experience