CephFS snapshots and the .snap directory

Why this matters in production

CephFS snapshots are per-directory and are accessed as ordinary directories, which makes self-service restore genuinely easy for users. They also have costs that scale with the tree being snapshotted, and those costs are not obvious from the interface.

Creating

Snapshots are created by making a directory inside .snap:

mkdir /mnt/cephfs/projects/alpha/.snap/daily-2026-08-18
ls /mnt/cephfs/projects/alpha/.snap/
rmdir /mnt/cephfs/projects/alpha/.snap/daily-2026-08-11

The .snap directory is virtual — it does not appear in ls but exists in every directory. That is what makes user self-service restore work:

cp /mnt/cephfs/projects/alpha/.snap/daily-2026-08-18/report.txt \
   /mnt/cephfs/projects/alpha/report.txt

A user can recover a file without any operator involvement, which removes a whole class of support request.

Through the subvolume interface

ceph fs subvolume snapshot create cephfs project-alpha snap-2026-08-18 --group_name research
ceph fs subvolume snapshot ls cephfs project-alpha --group_name research
ceph fs subvolume snapshot rm cephfs project-alpha snap-2026-08-18 --group_name research

Enabling

ceph fs set cephfs allow_new_snaps true
ceph fs get cephfs | grep allow_new_snaps

The costs

Metadata. A snapshot of a directory tree creates metadata state proportional to the tree — the MDS tracks which inodes belong to which snapshot. Snapshotting a directory containing tens of millions of files is substantially more expensive than snapshotting one with thousands.

Copy-on-write. As with RBD, the first write to any object after a snapshot clones it, so capacity grows with post-snapshot change.

Deletion. Removing a snapshot means releasing the preserved versions, which is background work proportional to what it held.

Nesting. Snapshots at multiple levels of one tree multiply the tracking the MDS must do. Take them at one consistent level.

Retention

# create with a sortable name
mkdir "/mnt/cephfs/projects/alpha/.snap/auto-$(date -u +%Y%m%dT%H%M%SZ)"

# remove beyond retention
ls /mnt/cephfs/projects/alpha/.snap/ | grep '^auto-' | sort | head -n -14 | \
  xargs -r -I{} rmdir "/mnt/cephfs/projects/alpha/.snap/{}"

Or use the built-in scheduler:

ceph mgr module enable snap_schedule
ceph fs snap-schedule add /projects/alpha 1d
ceph fs snap-schedule retention add /projects/alpha 7d4w
ceph fs snap-schedule status /projects/alpha

The scheduler is the better option: it handles creation and retention together, which is the pairing that hand-rolled automation gets wrong.

Quiz

Production discipline

Use the snap_schedule module rather than hand-rolled automation; it pairs creation with retention, which is the part scripts reliably forget. Take snapshots at one consistent level of the tree and document it, since nesting is easy for users to introduce and its cost falls on the MDS.

Cross-course references

• Kubernetes: VolumeSnapshot lifecycles need the same paired creation and retention
• Linux: the ZFS .zfs/snapshot directory is the direct equivalent and works the same way