MDS ranks and how the namespace is divided

Why this matters in production

A single MDS handles a great deal, and multiple ranks are the answer when it cannot. They also introduce subtree partitioning, cross-rank operations, and a more complicated failure surface — so the decision deserves a measurement rather than a default.

What a rank is

A rank is a numbered share of the filesystem’s metadata. With max_mds 3, ranks 0, 1, and 2 each own part of the namespace tree and serve requests for their own subtrees.

ceph fs status cephfs

cephfs - 42 clients
RANK  STATE     MDS       ACTIVITY     DNS    INOS
 0    active  mds.a  Reqs: 412 /s  1240k  1180k
 1    active  mds.b  Reqs: 388 /s  1102k  1051k
 2    active  mds.c  Reqs: 201 /s   640k   612k
STANDBY MDS
 mds.d

Rank 0 is special: it holds the filesystem root and certain global structures, and it must be present for the filesystem to be available.

Configuring

ceph fs set cephfs max_mds 3
ceph fs status cephfs

Ceph promotes standby daemons to fill the new ranks. You need at least max_mds daemons active plus standbys for failover:

ceph fs set cephfs standby_count_wanted 1

Three active ranks with no standby means a single MDS failure leaves the filesystem degraded until a replacement daemon appears.

When multiple ranks help

They help when: the MDS is CPU-saturated, request latency is high with a healthy metadata pool, and the workload spans many directories.

They help less when: the workload concentrates on one directory, since a single directory’s metadata largely lives on one rank; when the MDS is memory-constrained rather than CPU-constrained; or when the metadata pool is the bottleneck.

# is the MDS actually CPU-bound?
top -p $(pgrep ceph-mds)
ceph daemon mds.a perf dump | jq '.mds.request'

Measure before adding ranks. An MDS waiting on a slow metadata pool gains nothing from having peers.

Reducing rank count

ceph fs set cephfs max_mds 1
ceph fs status cephfs

Ranks are stopped one at a time, each exporting its subtrees to a remaining rank before shutting down. This takes time proportional to the metadata being moved, and the filesystem stays available throughout.

Quiz

Production discipline

Add MDS ranks on measured evidence of CPU saturation rather than on deployment size, and always provision standbys in addition to the active count. Check whether the workload renames across subtree boundaries before committing — that is the case where more ranks makes things slower.

Cross-course references

• Kubernetes: sharding a controller across replicas has the same cross-shard coordination cost
• Linux: NUMA-partitioned workloads pay a similar penalty on cross-node operations