Why CephFS does not have MDS split brain

Why this matters in production

“What if the old MDS comes back and both are active?” is the right question to ask of any failover system. For CephFS the answer is that it cannot serve, and the mechanism that guarantees this is worth understanding because it is the same mechanism you invoke manually when evicting a client.

Rank ownership is monitor state

The MDS map — which daemon holds which rank — is monitor state, maintained through Paxos exactly like the OSD map.

ceph fs dump
ceph mds stat

A daemon does not decide it holds a rank. The monitors assign it, by consensus, and publish the result. Two daemons cannot both believe they hold rank 0 in any map the cluster agrees on.

Fencing the replaced daemon

Assigning a rank to a new daemon is not sufficient on its own — the old daemon might still be running and might still write to the journal. So the monitors blocklist it:

ceph osd blocklist ls
# 10.20.0.31:6801/2947382 2026-08-19T09:14:22

A blocklisted address is rejected by every OSD. The old MDS cannot write to the journal, cannot write to the metadata pool, and cannot do anything that would corrupt the filesystem. Its I/O fails and it exits.

What a partitioned MDS experiences

1. MDS on rank 0 is partitioned from the monitors
2. Its beacons stop arriving; monitors declare the rank failed
3. Monitors assign the rank to a standby and blocklist the old daemon
4. The old daemon, still running, attempts to write to the journal
5. Every OSD rejects it — it is blocklisted
6. The daemon observes the failure and exits

At no point can it serve clients successfully, because clients also depend on the OSDs it can no longer reach.

Verifying

ceph osd blocklist ls
ceph fs dump | grep -A5 'mds_map'
ceph mds stat

Blocklist entries expire after a configurable period:

ceph config get mon mon_osd_blocklist_default_expire

The same mechanism protects against clients

Client eviction uses exactly this: the evicted client is blocklisted, so its writes are rejected even if the process keeps running.

SESSION_ID=12
ADDR=10.20.0.11
ceph tell mds.a client evict id=${SESSION_ID}
ceph osd blocklist ls
ceph osd blocklist rm ${ADDR}          # to allow it back

Quiz

Production discipline

Check ceph osd blocklist ls whenever a previously-fenced host or client will not reconnect; the symptom looks like a network or credential problem and the cause is visible in exactly one place. Understand blocklisting as the general fencing mechanism — it protects against replaced MDS daemons and evicted clients through the same path.

Cross-course references

• Kubernetes: fencing a partitioned node before rescheduling its pods is the identical requirement
• Linux: STONITH in cluster managers exists for exactly this reason and works the same way