When a client sees the wrong thing

Advanced⏱ ~17 mincephmount

What you'll learn

Diagnose files that appear missing on one client
Resolve permission errors from capability or UID mismatches
Identify genuine stale-view conditions
Verify with a controlled test

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

“The client is seeing stale data” is the usual first hypothesis and is almost never right — CephFS coherence is maintained by the MDS. The actual causes are more mundane and each is confirmed in a couple of commands.

Files missing on one client

Check the mount, not the consistency:

mount | grep ceph
findmnt -t ceph

# what subtree is this client actually mounted on?
cat /proc/mounts | grep ceph

Cause	Check
Different subtree mounted	compare the mount source path
Different filesystem	`fs=` mount option, if several exist
Path restriction on the capability	`ceph auth get client.<id>`
Different Ceph cluster entirely	compare monitor addresses

The first two account for the large majority.

Permission denied

ls -l /mnt/cephfs/projects/alpha
id
ceph auth get client.myapp

Cause	Symptom
POSIX permissions	ordinary `ls -l` shows it
UID mismatch between hosts	file owned by an unexpected numeric uid
`root_squash` on the capability	root cannot write where it expects to
Path restriction	the directory is not visible at all
Read-only capability	reads work, writes fail

UID mismatch is the one that surprises people: CephFS stores numeric UIDs, so a user with different UIDs on two hosts sees different ownership. There is no name mapping.

Genuinely stale views

Rare, and worth confirming rather than assuming:

# Substitute your own values before running:
MONS=10.20.0.10:3300,10.20.0.11:3300,10.20.0.12:3300
SECRETFILE=/etc/ceph/admin.secret

# from a third, freshly-mounted client
mount -t ceph "$MONS:/" /mnt/verify -o name=admin,secretfile="$SECRETFILE"
ls -la /mnt/verify/projects/alpha

A fresh mount shows the authoritative state. If it matches one client and not the other, the disagreeing client has a real problem:

# the client whose view disagrees:
CLIENT_HOST=app-07

ceph tell mds.a client ls | jq -r --arg h "$CLIENT_HOST" '.[] | select(.client_metadata.hostname==$h)'
ceph health detail | grep -i client
dmesg | grep -i ceph        # on the affected client

Stale file handle

ls: cannot access '/mnt/cephfs/data': Stale file handle

The session was evicted or timed out. The client must remount:

ADDR=10.20.0.11
ceph osd blocklist ls           # is it blocklisted?
ceph osd blocklist rm ${ADDR}
umount -l /mnt/cephfs && mount -a

Quiz

Knowledge check · 4 questions

Q1. A user has different numeric UIDs on two CephFS clients. What do they see?

Consistent ownership, because the MDS resolves the numeric ID to a name for each client Different ownership on each host, because CephFS stores numeric UIDs with no name translation Permission denied on both hosts, because the MDS rejects a UID it cannot reconcile Files owned by nobody, in the way NFSv4 falls back when its ID mapping fails
Q2. Comparing against a freshly-mounted client is a quick way to establish the authoritative filesystem state.

True False
Q3. Investigate files that one client cannot see.

A team reports that files created by their CI runners are not visible on their development workstations. Both mount CephFS. They suspect a consistency problem and have opened a high-priority ticket.
Q4. Why does CephFS not perform UID name mapping between clients?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Check the mount paths before investigating consistency; two clients looking at different subtrees explains this symptom far more often than anything else. Make consistent UID assignment an explicit part of CephFS rollout planning, since there is no mapping layer to paper over inconsistency.

Cross-course references

Kubernetes: fsGroup and runAsUser mismatches produce the same ownership confusion
Linux: NFS without idmapd shows exactly this numeric-UID behaviour