CephFS subvolumes as Kubernetes volumes

Intermediate⏱ ~17 minkubectlceph

What you'll learn

Explain what a CephFS subvolume provides
Trace a PVC to its subvolume in the filesystem
Manage subvolume groups
Diagnose subvolume-related problems

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

A CephFS PVC is a subvolume — a directory with quota, a data pool assignment, and its own identity. Understanding the mapping is what makes the filesystem navigable.

What a subvolume provides

ceph fs subvolume create cephfs mysubvol --size 10737418240
ceph fs subvolume ls cephfs
ceph fs subvolume info cephfs mysubvol

Property	Provided by the subvolume
A directory root	its own path
A quota	`ceph.quota.max_bytes` on that directory
A data pool	`ceph.dir.layout.pool`
Snapshots	per-subvolume
Isolation	separate paths, separately mountable

A subvolume is not a separate filesystem — it is a managed directory within one, with attributes set consistently.

The mapping from a PVC

# the PV bound to your PVC, from kubectl get pvc
PV=pvc-8f1d3c9a-4b2e-11f0-9c7a-0242ac110002

kubectl get pv "$PV" -o jsonpath='{.spec.csi.volumeAttributes}' | python3 -m json.tool

{
  "clusterID": "b3d5f2a1-...",
  "fsName": "cephfs",
  "pool": "cephfs_data",
  "subvolumeName": "csi-vol-8f3c...",
  "subvolumePath": "/volumes/csi/csi-vol-8f3c.../a1b2c3d4-..."
}

ceph fs subvolume ls cephfs csi
ceph fs subvolume info cephfs csi-vol-8f3c... csi

The path shows the layout: /volumes/<group>/<subvolume>/<uuid>, where the trailing UUID directory is what is actually mounted.

Subvolume groups

ceph fs subvolumegroup create cephfs csi
ceph fs subvolumegroup ls cephfs

# in the StorageClass
parameters:
  volumeNamePrefix: "pvc-"
  fsName: cephfs
  pool: cephfs_data

Ceph-CSI uses a group named csi by default. Groups allow separating volumes by consumer or by policy, with quotas applied at the group level:

ceph fs subvolumegroup create cephfs team-a --size 10995116277760

A group quota bounds every subvolume within it, which is how a whole consumer’s usage is limited.

Diagnosing subvolume problems

# does the subvolume exist?
NAME=acme
ceph fs subvolume ls cephfs csi | grep ${NAME}

# what is its quota and usage?
ceph fs subvolume info cephfs ${NAME} csi | python3 -c '
import sys,json; d=json.load(sys.stdin)
print("bytes_quota:", d.get("bytes_quota"))
print("bytes_used: ", d.get("bytes_used"))
print("path:       ", d.get("path"))'

Symptom	Cause
PVC Pending	subvolume group missing, or capability insufficient
Writes fail with ENOSPC	subvolume or group quota reached
Mount fails	MDS unavailable, or path incorrect
Volume appears empty	mounted the group rather than the subvolume path

Quiz

Knowledge check · 4 questions

Q1. What does a CephFS subvolume provide beyond an ordinary directory?

A managed directory with a quota, a data pool assignment, its own snapshots, and a stable identity A separate filesystem with its own MDS ranks, metadata pool, and mount root Block-level access to its contents, so a pod can map the directory as a device Independent replication settings, so that each subvolume carries size and min_size values of its own
Q2. Per-subvolume quotas are sufficient to bound a consumer's total CephFS usage.

True False
Q3. Bound a consumer's CephFS usage.

A team has been allocated 10 TB of CephFS capacity. They have created 40 PVCs each with a 500 GB quota, totalling 20 TB of potential usage.
Q4. Why does a subvolume path end in a UUID directory rather than the subvolume name?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Apply a quota to the subvolume group, not just to each subvolume — the group quota is what corresponds to a consumer’s allocation and per-volume limits never sum to a total. Read the PV’s subvolumePath to locate a volume in the filesystem; the trailing UUID directory is the mounted one.

Cross-course references

Kubernetes: namespace quotas bound aggregate usage where per-object limits cannot
Linux: directory quotas and per-user quotas serve different scopes