Meeting RBD recovery targets at scale

Why this matters in production

Per-image targets are easy. A fleet of hundreds of images sharing one cluster, one network, and one backup target is where they stop being met.

Computing achievable targets

# Substitute your own values before running:
POOL=rbd-vms

# the fleet's actual footprint
rbd ls "$POOL" --format json | python3 -c '
import sys,json,subprocess
pool = sys.argv[1]
tot = 0; n = 0
for img in json.load(sys.stdin):
    d = json.loads(subprocess.check_output(
        ["rbd","du",pool+"/"+img,"--format","json"]))
    u = sum(i.get("used_size",0) for i in d.get("images", []))
    tot += u; n += 1
print("images:", n, " used: %.1f TiB" % (tot/1024**4))' "$POOL"

Achievable RPO = time to complete one full backup cycle.
Achievable RTO = time to restore the images an incident affects,
                 not the time to restore one.

# daily change rate, measured from consecutive diffs
ls -l /backups/*.diff | awk '{s+=$5} END {printf "daily diff total: %.1f GiB\n", s/1024^3}'

The binding constraint

# Substitute your own values before running:
POOL=rbd-vms
IMAGE=vm-101-disk-0
SNAP=nightly-2026-08-18
BACKUP_HOST=backup.example.com

# time a representative export to find the real rate
time rbd export "$POOL/$IMAGE@$SNAP" - | ssh "$BACKUP_HOST" 'cat > /dev/null'

# during a backup run
POOL=rbd-vms
ceph osd pool stats ${POOL}
iostat -x 5 3

Choosing mechanisms

# Substitute your own values before running:
POOL=rbd-vms
IMAGE=vm-101-disk-0

rbd mirror pool enable "$POOL" image
rbd mirror image enable "$POOL/$IMAGE" snapshot
rbd mirror snapshot schedule add --pool "$POOL" 1h
rbd mirror pool status "$POOL" --verbose | head -20

Mirroring moves the RPO problem from the backup window to the network:
the change rate must fit the link continuously rather than fitting a
nightly window.

Verifying

# actual RPO across the fleet: age of the newest backup per image
POOL=rbd-vms
rbd ls ${POOL} | while read img; do
  newest=$(rbd snap ls "${POOL}/$img" --format json 2>/dev/null | python3 -c '
import sys,json
s = json.load(sys.stdin)
print(s[-1]["name"] if s else "NONE")')
  printf '%-28s %s\n' "$img" "$newest"
done | grep -c NONE

# Substitute your own values before running:
POOL=rbd-vms

# actual RTO: restore a representative image and time the whole sequence
time ( rbd import /backups/rep.raw "$POOL/rto-test" && \
       rbd map "$POOL/rto-test" && \
       mount "/dev/rbd/$POOL/rto-test" /mnt/rto && \
       ls /mnt/rto >/dev/null )
umount /mnt/rto; rbd unmap "/dev/rbd/$POOL/rto-test"; rbd rm "$POOL/rto-test"

Quiz

Production discipline

State fleet RTO against a realistic incident scope and measure the parallel restore rate, not the single-image one. Put mirror lag on a dashboard; when mirroring falls behind, RPO degrades silently to whatever the link delivers.

Cross-course references

• Kubernetes: recovery targets stated per-pod mislead when a node fails
• Linux: continuous replication is sized by peak rate, batch backup by window