CephFS failure behaviour in Kubernetes

Why this matters in production

CephFS has an active component — the MDS — that RBD does not, and its failure modes are visible to pods in ways that OSD failures are not.

MDS failover

ceph fs status
ceph mds stat
ceph fs get cephfs | grep -E 'max_mds|standby'

Active MDS fails
  → monitors detect and promote a standby
  → the standby replays the journal
  → clients reconnect and re-establish capabilities
  → I/O resumes

ceph fs set cephfs allow_standby_replay true
ceph fs status

standby-replay reduces failover from potentially minutes to seconds by keeping a standby continuously replaying the active MDS’s journal.

What pods experience

Data reads and writes to already-open files with existing capabilities
continue during a brief failover. Anything requiring the MDS — opening a
file, listing a directory, stat — blocks.

This is why a pod may appear partly functional during a failover.

Client eviction

ceph tell mds.0 client ls
ceph osd blocklist ls

A client that does not respond within the timeout is evicted
  → its capabilities are revoked
  → its mount becomes invalid
  → operations return ESTALE
  → the pod's volume is effectively broken until remounted

ceph config get mds mds_session_timeout
ceph config get mds mds_session_autoclose

Unlike an MDS failover, eviction does not resolve itself: the client’s mount is void and only a remount restores it.

Recovering pods after an interruption

# identify pods with a broken mount
kubectl get pods --field-selector status.phase=Running -o json | python3 -c '
import sys, json
for p in json.load(sys.stdin)["items"]:
    for c in p.get("status", {}).get("containerStatuses", []):
        if not c.get("ready"):
            print(p["metadata"]["namespace"], p["metadata"]["name"])'

# the remedy is a restart, which remounts
NAME=acme
kubectl delete pod ${NAME}

# clear the blocklist entry if the client should be allowed back
ADDR=10.20.0.11
ceph osd blocklist rm ${ADDR}

Restarting the pod causes the node plugin to unmount and remount, which establishes a fresh session.

Configuring for faster recovery

ceph config set mds mds_cache_memory_limit 8589934592
ceph orch apply mds cephfs --placement="count:3"

Quiz

Production discipline

Enable allow_standby_replay — it reduces MDS failover from minutes to seconds and every CephFS pod blocks on metadata operations throughout that window. Recognise eviction as needing a pod restart; unlike a failover it does not resolve itself.

Cross-course references

• Kubernetes: control plane component failover affects new operations more than running ones
• Linux: NFS server failover has the same open-handle versus new-operation asymmetry