Permissions and identity on CephFS volumes

Advanced⏱ ~18 minkubectlceph

What you'll learn

Explain how identity maps between pod and CephFS
Configure fsGroup and security context correctly
Diagnose permission errors
Handle multi-tenant permission requirements

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

CephFS stores POSIX ownership, containers run as arbitrary UIDs, and nothing translates between them. The result is permission errors that appear inconsistent.

How identity maps

A pod runs as UID 1000
  → writes a file to the CephFS mount
  → the file is owned by UID 1000 in CephFS
  → another pod running as UID 2000 cannot write it

There is no UID mapping layer: the numeric UID from the container is the numeric UID stored in the filesystem.

# Substitute your own value before running:
POD=web-5f9c7d8b6c-2xk9p

kubectl exec "$POD" -- id
kubectl exec "$POD" -- ls -ln /data

# from a CephFS mount on a host
SUBVOL=subvol
UUID=uuid
ls -ln /mnt/cephfs/volumes/csi/${SUBVOL}/${UUID}

fsGroup

spec:
  securityContext:
    fsGroup: 2000
    runAsUser: 1000
    fsGroupChangePolicy: OnRootMismatch

fsGroup causes Kubernetes to change group ownership of the volume’s contents and set the setgid bit, so files created in it inherit the group.

`fsGroupChangePolicy`	Behaviour
`Always` (default)	recursively chown on every mount
`OnRootMismatch`	only if the root directory’s ownership differs

A volume with a million files and fsGroupChangePolicy: Always
  → every pod start walks and chowns a million files
  → pod startup takes minutes

OnRootMismatch is almost always the right setting on a shared volume.

Diagnosing permission errors

# Substitute your own value before running:
POD=web-5f9c7d8b6c-2xk9p

# what the pod runs as
kubectl get pod "$POD" -o jsonpath='{.spec.securityContext}'
kubectl exec "$POD" -- id

# what the files are owned by
kubectl exec "$POD" -- ls -ln /data

# what the directory permits
kubectl exec "$POD" -- stat -c '%a %u %g' /data

Symptom	Cause
Permission denied writing to an existing directory	UID mismatch with the owner
Can create files but not modify others’	different pods running as different UIDs
Works on one pod, fails on another	different `runAsUser`
Slow pod startup on a large volume	`fsGroupChangePolicy: Always`
Root can write, non-root cannot	directory mode too restrictive

Multi-tenant requirements

# a per-tenant Ceph user restricted to a path
ceph fs authorize cephfs client.team-a /volumes/csi/team-a rw
ceph auth get client.team-a

# a StorageClass using that user, scoped to that path
parameters:
  fsName: cephfs
  subvolumeGroup: team-a
  csi.storage.k8s.io/provisioner-secret-name: csi-cephfs-team-a

Path-scoped Ceph users mean a tenant’s credentials cannot read another tenant’s subvolumes even if the UIDs happen to match.

Quiz

Knowledge check · 4 questions

Q1. Why does `fsGroupChangePolicy: Always` cause slow pod startup on large volumes?

It remounts the volume from scratch, re-reading the whole directory tree It revalidates the Ceph credentials against the MDS once for every file and directory in the volume It creates a snapshot of the volume first so that the ownership change is reversible It walks the volume recursively and changes group ownership on every file at every pod start
Q2. Kubernetes translates container UIDs to filesystem UIDs when writing to CephFS.

True False
Q3. Resolve inconsistent permission errors on a shared volume.

Several applications share a CephFS volume. Some pods can write files they created but not files created by other pods. Pod startup has also become slow.
Q4. What provides tenant isolation on CephFS where UID coordination cannot?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Set fsGroupChangePolicy: OnRootMismatch on any shared CephFS volume — the default recursively chowns every file at every pod start. Use path-scoped Ceph users via ceph fs authorize for genuine tenant isolation; UID coordination alone does not provide it.

Cross-course references

Kubernetes: fsGroup behaviour on large volumes is a common startup latency cause
Linux: NFS without idmapping presents the identical UID passthrough behaviour