CephFS subvolumes and multi-tenant file storage

Why this matters in production

Subvolumes are the supported abstraction for handing a piece of a CephFS filesystem to a tenant. They bundle the directory, the quota, the data placement, and the credential generation into one managed object — which matters because doing those four things by hand is where multi-tenant CephFS deployments go wrong.

Volumes, groups, and subvolumes

volume (a CephFS filesystem)
└── subvolume group (a container, e.g. per department)
    └── subvolume (a tenant's directory, with a quota)
        └── subvolume snapshot

ceph fs volume create shared-fs
ceph fs subvolumegroup create shared-fs research
ceph fs subvolume create shared-fs project-alpha --group_name research \
    --size 5497558138880          # 5 TiB
ceph fs subvolume ls shared-fs --group_name research
ceph fs subvolume info shared-fs project-alpha --group_name research

The path is managed for you:

ceph fs subvolume getpath shared-fs project-alpha --group_name research
# /volumes/research/project-alpha/8f3a...

That trailing UUID matters: it means a deleted and recreated subvolume of the same name gets a different path, so a stale mount cannot silently attach to a new tenant’s data.

Quotas

ceph fs subvolume resize shared-fs project-alpha 10995116277760 --group_name research

Quotas are enforced by the client through the ceph.quota.max_bytes extended attribute on the directory. Enforcement is therefore client-cooperative for the kernel client and has a propagation delay — a tenant can briefly exceed its quota before writes are refused. It is a capacity-management control, not a hard security boundary.

Tenant credentials

ceph fs subvolume authorize shared-fs project-alpha client.alpha \
    --group_name research --access_level=rw
ceph auth get client.alpha

This generates a path-restricted capability pointing at the subvolume’s managed path — the correct MDS clause and the tag-based OSD grant, without hand-writing either.

ceph fs subvolume deauthorize shared-fs project-alpha client.alpha --group_name research
ceph fs subvolume authorized_list shared-fs project-alpha --group_name research

Subvolumes versus separate filesystems

Separate filesystems mean separate MDS daemons and separate metadata pools — real resource isolation, at real cost. Reserve them for tenants whose metadata load would otherwise disturb others.

Quiz

Production discipline

Use the subvolume commands rather than creating directories and capabilities manually; the managed path and generated capability prevent two distinct classes of error. Be explicit with tenants that quotas allow a small overshoot, and never present a quota as a security boundary — the path restriction is what provides that.

Cross-course references

• Kubernetes: a namespace with a ResourceQuota is the same bundling of isolation and limit
• Linux: per-project XFS quotas have the same advisory-with-delay character