CephXLIII · CephFS Failure ScenariosCephFS Failure Scenarios
Diagnosing a slow MDS
What you'll learn
- Identify the four common causes of MDS slowness
- Measure the indicators that distinguish them
- Apply the appropriate remedy for each
- Prevent recurrence
Prerequisites
None — start here.
Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18
Why this matters in production
An MDS can be slow for reasons that have nothing to do with the MDS, and the most common one is a slow metadata pool. Adding MDS capacity to a metadata-pool problem changes nothing, which makes the diagnosis worth doing properly.
The four causes
| Cause | Indicator | Fix |
|---|---|---|
| Slow metadata pool | high pool latency, MDS not CPU-bound | move the pool to flash |
| Cache too small | low hit rate, high pool read rate | raise mds_cache_memory_limit |
| Client capability pressure | MDS_CLIENT_RECALL, high per-client caps | bound caps, fix the client |
| Genuine CPU saturation | MDS at 100% of a core | add ranks |
Measuring each
Metadata pool latency:
ceph osd pool stats cephfs-meta
ceph osd perf | head -20
ceph daemon mds.a perf dump | jq '.objecter.op_latency'Cache effectiveness:
ceph daemon mds.a cache status
ceph daemon mds.a perf dump | jq '.mds.inodes, .mds.inodes_top, .mds.inodes_bottom'
ceph daemon mds.a perf dump | jq '.mds_cache'Client capability pressure:
ceph health detail | grep -i client
ceph tell mds.a client ls | jq -r '.[] | "\(.id) \(.num_caps)"' | sort -k2 -rn | headCPU:
top -H -p $(pgrep ceph-mds)The MDS is largely single-threaded for request handling, so the relevant figure is one thread near 100%, not the whole process across all cores.
The order to check
- Metadata pool latency — the most common cause and the one people skip, because it is not an MDS metric
- Client capability counts — one bad client explains a great deal
- Cache hit rate — a small cache means every miss is a pool round trip
- CPU — only after the first three are ruled out
Adding ranks is the last resort, not the first response.
Slow requests
ceph health detail
# [WRN] MDS_SLOW_REQUEST: 4 slow requests are blocked
ceph daemon mds.a dump_blocked_ops
ceph daemon mds.a dump_ops_in_flightThe blocked operations name what they are waiting for, which frequently identifies the cause directly — a capability revocation from a specific client, or an outstanding pool operation.
Quiz
Knowledge check · 4 questions
Q1. Which cause of MDS slowness should be checked first?
Q2. An MDS process showing 130% CPU across all threads is definitely not saturated.
Q3. Investigate MDS slowness before changing the configuration.
Users report CephFS metadata operations taking seconds. The team proposes increasing max_mds from 1 to 3. The MDS host shows the ceph-mds process at 45% CPU total, the metadata pool is on HDD, and no client capability warnings are present.
Q4. Why does scaling an MDS mean adding ranks rather than adding cores?
Passing score: 75%. Answers are checked in this browser.
Production discipline
Check metadata pool latency before any MDS-side change; it is the most common cause and the one whose symptoms are indistinguishable from MDS overload. Measure per-thread rather than per-process CPU, since the single-threaded request path makes the aggregate figure misleading.
Cross-course references
- Kubernetes: a controller bottlenecked on API server latency looks identical to one that is CPU-bound
- Linux: distinguishing I/O wait from CPU saturation is the same fundamental diagnostic