Off-cluster file backup with deduplicating tools

Advanced⏱ ~18 minresticrsyncceph

What you'll learn

Run a deduplicating backup against CephFS
Tune it for a large tree
Handle the metadata that matters
Verify and restore

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

A deduplicating backup tool gives long retention at bounded cost, which is what the corruption detection delay actually requires.

Running it

# snapshot first, back up from the snapshot
mkdir /mnt/cephfs/projects/.snap/backup-$(date +%Y%m%d)
SNAP=/mnt/cephfs/projects/.snap/backup-$(date +%Y%m%d)

export RESTIC_REPOSITORY=s3:https://backup.example.net/ceph-projects
export RESTIC_PASSWORD_FILE=/run/restic-pw
restic backup --tag cephfs --tag projects "$SNAP"

restic snapshots --tag projects | tail -5

rmdir "$SNAP"

The snapshot is the point-in-time source; the repository is the
off-cluster copy with its own retention.

Tuning for a large tree

Problem	Tuning
Metadata walk dominates	more read concurrency; parallelise by subtree
MDS cache thrashing	raise `mds_cache_memory_limit`, or walk less
Repository operations slow	local cache on fast storage
Backup exceeds the window	split by subtree, run concurrently
Change detection expensive	rely on the tool’s cache, keep it warm

ceph config get mds mds_cache_memory_limit
ceph tell mds.0 perf dump 2>/dev/null | python3 -c '
import sys,json
d = json.load(sys.stdin).get("mds", {})
print("cache hit rate inputs — inodes:", d.get("inodes"),
      " inodes_top:", d.get("inodes_top"))'

# parallelise by subtree
for d in /mnt/cephfs/projects/*/; do
  echo "$d"
done | xargs -P 4 -I{} restic backup --tag cephfs {}

# keep the cache on local fast storage
export RESTIC_CACHE_DIR=/var/cache/restic

Metadata that matters

# hard links, xattrs, ACLs, sparseness
rsync -aHAX --numeric-ids --sparse "$SNAP/" backup-host:/backups/projects/

Metadata	Preserved by
Ownership and mode	both `rsync -a` and restic
Hard links	`rsync -H`; restic preserves them
Extended attributes	`rsync -X`; restic preserves them
POSIX ACLs	`rsync -A`; restic preserves them
Sparseness	`rsync --sparse`
CephFS layouts	neither — recorded separately

# capture layouts, which no file backup preserves
getfattr -n ceph.dir.layout /mnt/cephfs/projects 2>/dev/null
getfattr -n ceph.dir.layout.pool /mnt/cephfs/projects 2>/dev/null

File layouts — which pool a directory's data goes to, the stripe unit —
are CephFS-specific and must be recorded and reapplied at restore.

Verify and restore

restic check --read-data-subset=5%
restic snapshots --tag projects

# restore a single path
# Snapshot id from the `restic snapshots` output above:
SNAPSHOT_ID=4a1b2c3d

restic restore "$SNAPSHOT_ID" --target /mnt/cephfs/projects \
  --include /projects/needed-dir

# restore everything
restic restore latest --target /restore-target

# reapply the layout after restoring a directory tree
# POOL is the data pool recorded by the `getfattr` step above:
POOL=cephfs.cephfs.data

setfattr -n ceph.dir.layout.pool -v "$POOL" /mnt/cephfs/projects/restored

Quiz

Knowledge check · 4 questions

Q1. What do file backup tools not preserve about a CephFS tree?

Extended attributes, which are silently dropped unless the restore itself runs as root on the target filesystem POSIX ACLs, since a portable backup archive format carries no representation for them Hard links, which are restored as independent copies and silently double the space used CephFS file layouts — the target pool, stripe unit, and object size stored in the `ceph.` xattr namespace
Q2. Ninety-day retention on a multi-terabyte tree is a question of change rate rather than of total size.

True False
Q3. Set up long-retention CephFS backup.

A 15 TiB CephFS tree needs 90 days of retention to cover the application corruption detection delay. Nightly full copies are not affordable.
Q4. Why back up from a `.snap` path rather than the live tree?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Record CephFS directory layouts with getfattr at backup time — no file backup tool preserves them, and a restored tree otherwise lands in the default data pool. Use content-addressed deduplication so retention cost tracks change rate rather than total size.

Cross-course references

Kubernetes: StorageClass parameters are not carried by a file-level restore either
Linux: filesystem-specific attributes need explicit capture alongside file backup