CephFS through a manager outage

Why this matters in production

An MDS fails over correctly with every manager in the cluster dead, and at the same time no new CephFS volume can be provisioned — so the filesystem is simultaneously fine and unusable, depending on who is asking.

MDS failover does not involve the manager

The MDSMap lives in the monitors. The monitors track MDS beacons, declare a rank laggy when the grace period lapses, and promote a standby. No part of that path touches a manager.

ceph config get mds mds_beacon_grace
ceph config get mon mon_mgr_beacon_grace
ceph mds stat
ceph fs dump | head -25

cephfs:1 {0=ceph-b=up:active} 2 up:standby

What a manager outage removes from CephFS

Existing mounts read and write throughout. New PersistentVolumeClaims
stay Pending, and nothing on the Ceph side reports an error, because the
command was never routed anywhere.

Two standby models

ceph fs set cephfs allow_standby_replay true
ceph fs get cephfs | grep -E 'max_mds|standby'

Inspecting the filesystem without a manager

ceph fs dump --format json | python3 -c '
import sys,json
d = json.load(sys.stdin)
for fs in d["filesystems"]:
    m = fs["mdsmap"]
    print("fs %-12s max_mds=%s up=%d" % (m["fs_name"], m["max_mds"], len(m["up"])))
    for gid, i in m["info"].items():
        print("   rank %-3s %-24s %s" % (i["rank"], i["name"], i["state"]))
print("standbys:", [s["name"] for s in d.get("standbys", [])])'

ceph mds stat
ceph fs ls
ceph tell mds.0 status
ceph tell mds.0 session ls | python3 -c '
import sys,json; print("sessions:", len(json.load(sys.stdin)))'
ceph health detail

Quiz

Production discipline

When CephFS provisioning stalls but existing mounts are healthy, check manager state before filesystem state — every symptom points at CephFS and the cause is usually a manager. After any manager outage, audit whether CephFS snapshot schedules and RBD mirror schedules fired; nothing records the ones that did not.

Cross-course references

• Kubernetes: a CSI provisioner failure leaves running pods serving and new claims Pending
• Linux: a control-plane outage is invisible on the data path and total on the provisioning path