CephFS snapshots: the .snap directory and its behaviour

Why this matters in production

CephFS snapshots are created by making a directory, which is unlike every other snapshot mechanism and catches people out.

Creating and removing

# from a mounted client, snapshot any directory
mkdir /mnt/cephfs/projects/.snap/before-migration
ls /mnt/cephfs/projects/.snap/

# removing is rmdir
rmdir /mnt/cephfs/projects/.snap/before-migration

There is no `ceph fs snap create`. Snapshots are directory operations
performed by a client with the right capability, on any directory.

# the subvolume interface, which wraps the same mechanism
ceph fs subvolume snapshot create cephfs acme snap-20260818 --group_name tenants
ceph fs subvolume snapshot ls cephfs acme --group_name tenants
ceph fs subvolume snapshot rm cephfs acme snap-20260818 --group_name tenants

# snapshots must be permitted on the filesystem
ceph fs get cephfs | grep -i snap
ceph fs set cephfs allow_new_snaps true

Where the contents appear

ls /mnt/cephfs/projects/.snap/before-migration/

The .snap directory is virtual — it does not appear in `ls` without
being named explicitly, and it exists under every directory in the
filesystem.

# a snapshot taken at a parent is visible from children
ls /mnt/cephfs/projects/subdir/.snap/

Cost

A snapshot costs nothing at creation. It costs as the live tree
diverges: overwritten and deleted files retain their snapshotted
content in the data pool.

ceph df detail
ceph fs status cephfs

# metadata cost, which is the one people miss
ceph tell mds.0 perf dump 2>/dev/null | python3 -c '
import sys,json
d = json.load(sys.stdin).get("mds_mem", {})
print("inode count:", d.get("ino"), " dentries:", d.get("dn"))'

Operational limits

Snapshots interact with the MDS in ways that matter at scale:
  many snapshots on a heavily-modified tree increase MDS metadata work
  snapshots on directories that are moved have historically been
    a source of complexity
  a snapshot of a very large subtree makes every subsequent metadata
    operation in that subtree slightly more expensive

ceph fs status cephfs
ceph health detail | grep -i -E 'mds|snap'

The practical guidance: snapshot at the subvolume or project level, not
at the filesystem root, and keep the count bounded.

Quiz

Production discipline

Snapshot at the subvolume or project level rather than the filesystem root — the MDS tracks snapshot state for every inode in scope, so a root snapshot makes every metadata operation in the filesystem slightly more expensive. There is no rollback; plan recovery as a copy.

Cross-course references

• Kubernetes: CSI snapshots of CephFS subvolumes wrap this same mechanism
• Linux: snapshot cost scales with the scope it covers, not with its count alone