CephFS recovery targets and mirroring

Why this matters in production

CephFS RTO is dominated by file count as well as size, which makes the usual bandwidth arithmetic optimistic.

Computing achievable targets

ceph fs status cephfs
ceph df detail | grep -A3 cephfs

# file count, which drives metadata restore time
ceph tell mds.0 perf dump 2>/dev/null | python3 -c '
import sys,json
d = json.load(sys.stdin).get("mds_mem", {})
print("inodes tracked:", d.get("ino"))'

Restore time ≈ max( bytes / bandwidth,
                    files / achievable create rate )

# measure the create rate against this cluster
time ( mkdir -p /mnt/cephfs/rate-test && \
       for i in $(seq 1 10000); do : > /mnt/cephfs/rate-test/f$i; done )
rm -rf /mnt/cephfs/rate-test

Snapshot mirroring

ceph mgr module enable mirroring
ceph fs snapshot mirror enable cephfs

# add the peer, using a bootstrap token from the remote cluster
TOKEN=REDACTED
ceph fs snapshot mirror peer_bootstrap create cephfs-remote client.mirror site-b
ceph fs snapshot mirror peer_bootstrap import cephfs ${TOKEN}

ceph fs snapshot mirror add cephfs /volumes/tenants/acme
ceph fs snapshot mirror ls cephfs

ceph orch apply cephfs-mirror --placement=2
ceph orch ps --daemon-type cephfs-mirror

Mirroring replicates snapshots of the configured directories to the peer
filesystem. The RPO is the snapshot cadence plus the transfer time.

Monitoring progress

ceph fs snapshot mirror daemon status

ceph fs snapshot mirror peer_list cephfs

# FSCID is the filesystem cluster id — the number `ceph fs dump` prints after
# the filesystem name, as in `Filesystem 'cephfs' (1)`:
FSCID=1

# per-directory sync state
ceph --admin-daemon /var/run/ceph/ceph-client.cephfs-mirror.*.asok \
  fs mirror status "cephfs@$FSCID" 2>/dev/null | head -20

As with RBD mirroring, falling behind degrades RPO silently. The age of
the peer's newest snapshot is the number to alert on.

Failing over

Failover to a mirrored filesystem:
  stop writes to the source, if it is reachable
  confirm the peer has the newest snapshot you expect
  point clients at the peer filesystem
  accept the data written after the last synced snapshot is lost

# on the peer
ceph fs ls
ceph fs subvolume ls cephfs --group_name tenants

# what the peer actually has
ls /mnt/cephfs-peer/volumes/tenants/acme/.snap/

Failover is a client repointing operation, and CephFS clients hold
mount state — so it is a remount on every client, not a transparent
switch.

Quiz

Production discipline

Estimate CephFS RTO as max(bytes/bandwidth, files/create-rate) and measure the create rate — file count binds more often than bandwidth on large trees. Alert on the age of the mirror peer’s newest snapshot; lag degrades RPO silently.

Cross-course references

• Kubernetes: restoring many small files into a PV hits the same metadata ceiling
• Linux: filesystem restore is bounded by inode operations as much as by throughput