Session timeouts, eviction, and stale file handles

Why this matters in production

Eviction protects the filesystem from a client in an unknown state, and it costs that client its buffered writes. Understanding when it fires lets you tune it for an environment with unreliable clients rather than being surprised by it.

The sequence

1. Client stops responding to the MDS
2. MDS waits mds_session_autoclose (300 s default)
3. MDS closes the session and blocklists the client
4. Client's subsequent operations fail with ESTALE
5. Client must remount

ceph config get mds mds_session_autoclose
ceph config get mds mds_session_blocklist_on_timeout
ceph config get mds mds_session_blocklist_on_evict

Why blocklisting rather than a simple disconnect

A client that stops responding may be partitioned rather than dead, and a partitioned client holding write capabilities could still be writing directly to the OSDs. Blocklisting stops that at the OSDs, which is the only place it can be enforced.

The consequence is that the client cannot simply reconnect — its address is refused until the entry expires or is removed.

Diagnosing a stale file handle

# on the client
ls /mnt/cephfs
# ls: cannot access '/mnt/cephfs': Stale file handle
dmesg | grep -i ceph

# on the cluster
ceph osd blocklist ls
ceph tell mds.a session ls | jq -r '.[].client_metadata.hostname'
ceph health detail | grep -i client

If the client’s address appears in the blocklist, that is the answer.

Recovering

# remove the blocklist entry
ceph osd blocklist rm 10.20.0.55:0/1234567

# on the client
umount -l /mnt/cephfs
mount -a

The lazy unmount is often necessary because processes are still holding the stale mount.

Buffered writes that had not reached the OSDs are lost. Applications that believed those writes succeeded were misinformed, which is worth knowing before declaring the recovery complete.

Tuning

# more tolerant, for unreliable clients
ceph config set mds mds_session_autoclose 600

# faster reclamation, for a controlled environment
ceph config set mds mds_session_autoclose 120

An environment with intermittently-connected clients — laptops, VPN users — benefits from a longer timeout. A controlled datacentre environment benefits from a shorter one.

Quiz

Production discipline

Record evictions as data-loss events rather than as disconnections; the affected applications were told their writes succeeded and someone should know. Tune mds_session_autoclose for the least reliable client population you serve, and monitor MDS cache pressure after increasing it.

Cross-course references

• Kubernetes: node eviction with unflushed local state carries the same silent-loss property
• Linux: NFS server-side lock recovery and grace periods address the same partitioned-client problem