How ranks divide one namespace

Why this matters in production

The partitioning is automatic and continuous, which is convenient until it is not. A workload whose access pattern shifts causes subtrees to migrate between ranks, and migration is expensive. Understanding when to override the automation is the difference between multiple ranks helping and multiple ranks thrashing.

Dynamic subtree partitioning

Each rank owns a set of subtrees. The MDS cluster monitors load and migrates subtrees between ranks to balance it:

ceph fs status cephfs
ceph daemon mds.a get subtrees | jq -r '.[] | "\(.dir.path) rank=\(.auth_first)"'

/                    rank=0
/projects/alpha      rank=1
/projects/beta       rank=2
/scratch             rank=1

Clients see one namespace; the MDS cluster routes each request to the rank owning the relevant subtree.

The cost of migration

Exporting a subtree means transferring its metadata state between MDS daemons, coordinating with clients holding capabilities on it, and updating the authority records. During the export, operations on that subtree are delayed.

A workload with shifting hot spots can cause continuous migration — metadata churning between ranks without ever settling — which is worse than a single rank would have been.

Pinning

# pin a directory and its children to rank 1
setfattr -n ceph.dir.pin -v 1 /mnt/cephfs/projects/alpha

# remove the pin
setfattr -n ceph.dir.pin -v -1 /mnt/cephfs/projects/alpha

getfattr -n ceph.dir.pin /mnt/cephfs/projects/alpha

Pinning removes the balancer’s discretion for that subtree. It is the right tool when you know the workload’s structure better than the balancer can infer it — one team per rank, one project per rank.

Ephemeral pinning

# distribute a directory's children across ranks by hash
setfattr -n ceph.dir.pin.distributed -v 1 /mnt/cephfs/homes

# pin each subdirectory to a rank, but let the assignment be automatic
setfattr -n ceph.dir.pin.random -v 0.5 /mnt/cephfs/scratch

Distributed pinning is well suited to a directory of many equivalent subdirectories — home directories, per-tenant trees — where you want them spread but do not want to assign each one manually.

Observing the effect

ceph fs status cephfs           # per-rank request rates
ceph daemon mds.a perf dump | jq '.mds.exported, .mds.imported'

Rising export and import counters mean the balancer is actively moving subtrees. Persistently high values indicate thrashing and are the signal to pin.

Quiz

Production discipline

Pin subtrees where the filesystem has a natural partition; the balancer infers structure from load and does worse than explicit assignment when the structure is already known. Monitor export and import counters as the indicator of whether the balancer is settling or thrashing.

Cross-course references

• Kubernetes: pod topology constraints override the scheduler where you know better than it does
• Linux: CPU affinity pinning serves the same purpose against an automatic scheduler