Restoring CephFS data

Advanced⏱ ~18 mincephrsync

What you'll learn

Choose the appropriate recovery path
Restore from a snapshot
Restore from an off-cluster backup
Handle a whole-filesystem recovery

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

CephFS has no rollback, so every recovery is a copy — and the right copy mechanism differs by an order of magnitude in cost.

Choosing the path

Scope	Path	Cost
One file	copy from `.snap`	seconds
A directory	copy from `.snap`	proportional to that directory
A whole subvolume, keeping the live one	subvolume clone	full copy of the subvolume
A whole subvolume, replacing it	clone, then repoint the consumer	full copy
Data older than snapshot retention	off-cluster backup	transfer time
The filesystem is gone	off-cluster backup, into a rebuilt filesystem	full restore

The first two cover the overwhelming majority of real recoveries, and
they are nearly free.

From a snapshot

# find the snapshot holding the wanted state
ls /mnt/cephfs/projects/.snap/

# restore one file
cp -a /mnt/cephfs/projects/.snap/backup-20260817/report.xlsx \
      /mnt/cephfs/projects/report.xlsx.restored

# restore a directory, without clobbering current content
rsync -aHAX \
  /mnt/cephfs/projects/.snap/backup-20260817/design/ \
  /mnt/cephfs/projects/design.restored/

Restore to a new path and let the owner confirm before replacing the
live copy. Copying over the live path destroys whatever was there.

# subvolume equivalent
ceph fs subvolume snapshot clone cephfs acme snap-20260817 acme-restored \
  --group_name tenants --target_group_name tenants
ceph fs clone status cephfs acme-restored --group_name tenants

From off-cluster backup

# SNAPSHOT_ID: the short id from the restic snapshots listing below
SNAPSHOT_ID=7f3c1a9e

restic snapshots --tag projects
restic restore "$SNAPSHOT_ID" --target /mnt/cephfs/projects-restored \
  --include /projects/design

# reapply layouts, which the backup did not carry
setfattr -n ceph.dir.layout.pool -v cephfs.cephfs.data \
  /mnt/cephfs/projects-restored/design

ls -la /mnt/cephfs/projects-restored/design | head
getfattr -n ceph.dir.layout /mnt/cephfs/projects-restored/design 2>/dev/null

Whole-filesystem recovery

If the filesystem itself is lost, recovery is:
  create a new filesystem with the intended layout
  restore the tree from off-cluster backup
  reapply layouts
  recreate subvolumes and their quotas
  reissue or restore client capabilities
  repoint clients

ceph fs volume create cephfs-new
ceph fs subvolumegroup create cephfs-new tenants
ceph fs subvolume create cephfs-new acme --size 5497558138880 --group_name tenants

ceph fs ls
ceph fs status cephfs-new

Note that recreating the shape is fast and restoring the data is not.
The RTO is dominated by the transfer, which is why it must be measured.

Quiz

Knowledge check · 4 questions

Q1. What dominates CephFS whole-filesystem recovery time?

Recreating the volume, the subvolume groups, and the subvolumes together with their quotas and file layouts on the rebuilt cluster Reissuing the client capabilities, since every entity has to be recreated before any client can mount Restoring the content — bounded by bandwidth times volume and by the metadata operation rate for creating millions of files Reapplying the file layouts, since each directory's target pool and stripe unit must be set before content lands
Q2. Restoring a directory from `.snap` should be copied directly over the live path.

True False
Q3. Choose a CephFS recovery path.

A tenant reports a directory was deleted this morning. Nightly subvolume snapshots exist and the subvolume holds 4 TiB.
Q4. What must be reapplied after restoring a CephFS tree from off-cluster backup?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Restore alongside the live tree and let the owner confirm before replacing — copying over the live path destroys work done since the snapshot. For CephFS, include file count as well as byte count in the RTO estimate; metadata operations dominate large-tree restores.

Cross-course references

Kubernetes: restoring into a new PVC before switching keeps the original available
Linux: file-count dominates restore time for large trees more often than byte-count