Delivered RPO across three replication mechanisms

Advanced⏱ ~18 minrbdcephradosgw-admin

What you'll learn

Measure delivered lag for RBD, CephFS, and RGW replication
Distinguish configured cadence from delivered RPO
Compute a service-level RPO across interfaces
Alert on the number that matches user-visible loss

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Each mechanism reports progress in a different unit, and none of them reports the number a user would recognise as data loss.

What each mechanism exposes

Mechanism	Progress signal	Unit
RBD journal-based	`entries_behind_primary`	journal entries
RBD snapshot-based	remote snapshot timestamp	wall-clock time
CephFS snapshot mirroring	last synced snapshot	snapshot name and time
RGW multi-site	shards caught up or behind	shard count

Journal entries and shard counts are not time. Converting them to a
recovery point requires knowing the rate, and the rate changes.

Measuring RBD lag

rbd mirror image status rbd/vm-101-disk-0

vm-101-disk-0:
  global_id:   4b1e0f5a-3c2d-4f1a-9e77-0a1b2c3d4e5f
  state:       up+replaying
  description: replaying, {"bytes_per_second":18400.0,
               "entries_behind_primary":214,"non_primary_position":...}
  service:     rbd-mirror.site-b on ceph-b-01
  last_update: 2026-08-18 09:41:02

# snapshot mode reports timestamps, which convert directly to an RPO
rbd mirror image status rbd/vm-204-disk-0 --format json | python3 -c '
import sys,json,datetime
d = json.load(sys.stdin)
for p in d.get("peer_sites", []):
    print(p.get("site_name"), p.get("description"))'

rbd mirror pool status rbd --verbose | grep -E 'image|state|description' | head -30

Measuring CephFS and RGW lag

ceph fs snapshot mirror daemon status
ceph fs snapshot mirror peer_list cephfs

# the age of the newest snapshot present at the peer is the delivered RPO
ls -1 /mnt/peer-cephfs/volumes/tenants/acme/.snap | tail -1

radosgw-admin bucket sync status --bucket=acme-data
radosgw-admin sync error list | head -10

Symptom	Delivered RPO
`entries_behind_primary` growing steadily	unbounded; the link is undersized
Remote snapshot timestamp 4h old, schedule 1h	4 hours, not 1
Peer snapshot list not advancing	since the last one that landed
Sync errors accumulating for one bucket	that bucket is not protected at all

Service-level RPO

A service that stores VM disks in RBD, shared state in CephFS, and
uploads in RGW has one recovery point: the oldest of the three. Averaging
them is meaningless.

# collect all three into one line for the dashboard
{
  rbd mirror pool status rbd --verbose | grep -c 'state: *up+replaying'
  ceph fs snapshot mirror daemon status | head -5
  radosgw-admin sync status | grep -c 'caught up'
} 2>/dev/null

Component	Configured	Delivered	Contribution
RBD, snapshot mode	15 min	22 min	not binding
CephFS mirroring	1 hour	68 min	not binding
RGW multi-site	continuous	4 min	not binding
RBD, one lagging image	15 min	6 hours	binding — service RPO is 6 hours

Quiz

Knowledge check · 4 questions

Q1. A pool has a 15-minute mirror snapshot schedule and the peer's newest snapshot is 6 hours old, with no errors reported. What is the delivered RPO?

15 minutes, since the schedule is configured and the mirror daemon reports no errors against it 6 hours — the schedule governs when snapshots are taken, not when they finish transferring Undefined, because a snapshot that has not finished transferring has no recovery point at all 15 minutes plus the transfer time of the one snapshot currently in flight to the peer
Q2. A service storing data in RBD, CephFS, and RGW has an RPO equal to the average of the three mechanisms.

True False
Q3. Establish the real recovery point for a replicated service.

A service uses 40 mirrored RBD images, one mirrored CephFS tree, and two replicated buckets. The team reports an RPO of 15 minutes based on the RBD schedule.
Q4. Why can a cross-interface reference break after a site failover even when every mechanism is healthy?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Alert on the age of the newest replicated snapshot at the peer, not on the presence of a schedule — a schedule that is running correctly can still deliver a six-hour recovery point. State a service RPO as the worst of its components, and order cross-interface writes so that a dangling reference is harmless rather than corrupting.

Cross-course references

Kubernetes: a CronJob that runs on time still tells you nothing about whether it finished
Linux: replication lag measured in internal units needs a rate before it means anything