The MDS cache and its memory limit

Why this matters in production

Every metadata operation served from cache avoids a metadata pool round trip. Every one served from a miss costs that round trip and occupies the MDS while it waits. The cache size therefore determines MDS throughput more than any other setting, and running it too small is a common and easily corrected mistake.

Configuring

ceph config set mds mds_cache_memory_limit 17179869184     # 16 GiB
ceph config get mds mds_cache_memory_limit
ceph daemon mds.a cache status

{
  "pool": { "items": 8412034, "bytes": 14203847168 }
}

The limit is a target, not a hard cap — the MDS trims towards it and can exceed it transiently.

What lives in the cache

Capability state is why a client behaving badly affects MDS memory: caps issued to clients are tracked in the MDS, and a client holding millions of them consumes MDS memory it cannot reclaim until the client releases them.

Cache pressure

[WRN] MDS_CACHE_OVERSIZED: 1 MDSs report oversized cache
[WRN] MDS_CLIENT_RECALL: 2 clients failing to respond to cache pressure

The second warning is the important one: the MDS has asked clients to release capabilities so it can trim, and they have not. Common causes:

• A client running a filesystem-wide scan
• A client on an old kernel with recall bugs
• A genuinely busy client legitimately holding a large working set

ceph tell mds.a client ls | \
  jq -r '.[] | "\(.id) \(.num_caps) \(.client_metadata.hostname)"' | sort -k2 -rn | head

Bounding client behaviour

ceph config set mds mds_max_caps_per_client 1048576
ceph config set mds mds_recall_max_caps 30000
ceph config set mds mds_recall_max_decay_rate 1.5

The per-client cap limit is the important one. Without it, a single client can drive the MDS out of memory, and the MDS has no recourse but to ask politely.

Sizing

rough guide: 2–4 KB of MDS cache per cached inode
1 million active files ≈ 2–4 GB
plus capability state, plus overhead

Measure rather than estimate:

ceph daemon mds.a cache status
ceph daemon mds.a perf dump | jq '.mds_mem'
ceph daemon mds.a perf dump | jq '.mds.inodes, .mds.inodes_top, .mds.inodes_bottom'

A cache hit rate well below 90% on a workload with a stable working set means the cache is too small.

Quiz

Production discipline

Set mds_max_caps_per_client explicitly on every production filesystem; the recall mechanism is cooperative and a single scanning client can otherwise exhaust MDS memory. Monitor cache size against the limit and treat recall warnings as a precursor to an out-of-memory event rather than as informational.

Cross-course references

• Kubernetes: an unbounded informer cache exhibits the same unbounded-growth failure
• Linux: NFS delegation recall has the same cooperative, unenforceable character