Costing a second site, and what to replicate to it

Why this matters in production

The storage at the second site is the part of the cost that is easy to predict, and usually not the part that overruns.

The components

A DR cluster is a whole cluster. The recurring cost is that it must be
operated, not that it holds a copy.

Measuring change rate

# bytes changed between two mirror snapshots on one image
# the older snapshot name, from rbd snap ls rbd/vm-101-disk-0
OLDER_SNAP=mirror-20260817-0000

rbd diff --from-snap "$OLDER_SNAP" rbd/vm-101-disk-0 --format json | python3 -c '
import sys,json
print(sum(e["length"] for e in json.load(sys.stdin))/1024**2, "MiB changed")'

ceph osd pool stats vms
ceph df detail

# object storage change rate, from usage rather than from capacity
radosgw-admin usage show --show-log-entries=false \
  --start-date=$(date -d 'yesterday' +%Y-%m-%d) | python3 -c '
import sys,json
d = json.load(sys.stdin)
for e in d.get("summary", []):
    t = e.get("total", {})
    print("%-24s sent %8.1f MiB" % (e.get("user"), t.get("bytes_received",0)/1024**2))'

Sizing the link

Required = peak change rate / target utilisation
Headroom = accumulated backlog / acceptable catch-up time
Size for the larger of the two.

A link sized exactly at the change rate never recovers from an outage —
it can carry the new writes or the backlog, not both.

Replicating less

rbd mirror pool enable rbd image        # not pool mode; opt in per image
rbd mirror image enable rbd/vm-101-disk-0 snapshot
rbd mirror pool info rbd

# a cheaper peer: erasure coding where the primary is replicated
ceph osd erasure-code-profile set dr-4-2 k=4 m=2 \
  crush-failure-domain=host crush-device-class=hdd
ceph osd pool create vms-dr erasure dr-4-2

Quiz

Production discipline

Size the inter-site link for peak change rate plus a catch-up term — a link matched to the change rate reports healthy while never converging after an outage. Enable mirroring per image, per directory, and per bucket rather than pool-wide, and keep the peer cluster version on the same dashboard as its sync lag.

Cross-course references

• Kubernetes: a standby cluster that misses upgrade windows fails over into an incompatible control plane
• Linux: a queue drained at exactly its arrival rate never clears a backlog