CephFS client behaviour and consistency

Why this matters in production

CephFS provides stronger consistency than NFS and people frequently assume otherwise, applying workarounds that are unnecessary and occasionally harmful. Knowing what is actually guaranteed lets applications rely on it.

The guarantee

CephFS provides close-to-open consistency and better: a file closed by one client is fully visible to another client opening it afterwards. In practice the capability mechanism provides more than that — when two clients access the same file concurrently, the MDS revokes the caching capabilities that would allow divergence, so both see coherent data.

This is materially stronger than NFS’s default behaviour, and it means applications generally do not need the flush-and-revalidate dances written for NFS.

Where caching happens

The MDS drives all of it: a client caches only what its capabilities permit, and the MDS revokes those capabilities when another client needs conflicting access.

When a client sees stale data

Genuinely stale views are rare and usually indicate a problem:

# what does the client hold?
ceph daemon mds.a session ls | jq -r '.[] | "\(.id) \(.num_caps) \(.client_metadata.hostname)"'

# is the MDS complaining about it?
ceph health detail | grep -i client

The last row is the most common by far: two clients mounting different subtrees and expecting to see each other’s files.

Tuning client cache

# Substitute your own monitor addresses before running:
MONS=192.0.2.11:6789,192.0.2.12:6789,192.0.2.13:6789

# client-side cache limits
ceph config set client client_cache_size 32768
ceph config set client client_oc_size 209715200

# mount options
mount -t ceph "$MONS:/" /mnt/cephfs -o name=app,rasize=67108864

Larger client caches reduce MDS round trips and increase the memory a revocation has to work through. On a client with many files open, that trade favours a moderate cache.

Session timeouts

ceph config get mds mds_session_autoclose        # 300 s
ceph config get mds mds_session_blocklist_on_timeout

A client that stops responding has its session closed after the timeout and is blocklisted, which is what produces Stale file handle on that client and requires a remount.

Quiz

Production discipline

Verify what consistency your workload actually needs against what CephFS provides before carrying over NFS-era workarounds; the extra flushes cost measurable performance and address a gap that is not there. When files appear missing across clients, compare mount paths before investigating consistency — it is the far more common cause.

Cross-course references

• Kubernetes: ReadWriteMany volume semantics depend entirely on the underlying filesystem’s guarantees
• Linux: NFS close-to-open consistency is the weaker model these workarounds were written for