CephFS capabilities: the client caching contract

Why this matters in production

Capabilities are what make CephFS fast — a client holding the right caps performs local operations with no MDS round trip. They are also the source of the most confusing CephFS problems, where one client’s behaviour makes another client hang.

What a capability grants

Each inode has a capability set issued per client. The bits control what the client may cache and do locally:

A client with Fb can buffer writes locally and acknowledge them to the application before they reach the OSDs, which is a large performance win — and is only safe because the MDS guarantees no other client holds a conflicting capability.

ceph daemon mds.a session ls | jq -r '.[] | "\(.id) \(.num_caps) \(.client_metadata.hostname)"'

Issue and revocation

1. client opens a file → MDS issues caps
2. second client opens the same file for writing
3. MDS must revoke the first client's conflicting caps
4. MDS → client 1: release Fb
5. client 1 flushes buffered writes, then releases
6. MDS → client 2: issue write caps

Step 5 is where problems occur. If client 1 is slow to flush — a large buffer, a slow network, an unresponsive host — client 2 waits.

The characteristic symptom

[WRN] MDS_CLIENT_LATE_RELEASE: 1 clients failing to respond to capability release
[WRN] MDS_SLOW_REQUEST: 3 slow requests are blocked

ceph health detail
ceph daemon mds.a dump_blocked_ops
ceph tell mds.a client ls | jq -r '.[] | select(.num_caps > 100000)'

A client holding a very large number of capabilities is often the cause: it has walked a huge directory tree and cached everything, and now every revocation involves searching that set.

Responding

# ask a client to trim its cache
ID=12
SESSION_ID=12
ceph tell mds.a client config set ${ID} ...

# evict a client that will not respond
ceph tell mds.a client evict id=${SESSION_ID}

Eviction blocklists the client, so it cannot write anything further — protecting the filesystem from a client that may be in an unknown state. The client must remount to return.

ADDR=10.20.0.11
ceph osd blocklist ls
ceph osd blocklist rm ${ADDR}

Quiz

Production discipline

Set mds_max_caps_per_client and mds_cache_memory_limit explicitly rather than relying on client behaviour — a single scanning process can otherwise consume everything the MDS has. Treat eviction as a data-loss action in the runbook, because a client holding buffered writes loses them and its applications were told they succeeded.

Cross-course references

• Kubernetes: informer caches holding references to every object have the same unbounded-growth risk
• Linux: NFS delegations and their recall behave identically, including the slow-client problem