Responding to an MDS daemon crash

Why this matters in production

A single MDS crash with a standby present is a pause, not an incident. Knowing that lets you resist intervening in a recovery that is proceeding correctly, and reserve action for the cases that genuinely need it.

The automatic response

1. MDS process dies; beacons stop
2. Monitors wait mds_beacon_grace, then mark the rank failed
3. A standby is assigned the rank
4. It replays the journal, resolves, reconnects clients, rejoins
5. Rank active; clients resume

watch -n 2 'ceph fs status cephfs'
ceph -w | grep -i mds

No operator action is required for any of this.

What clients see

Applications doing bulk data transfer frequently do not notice an MDS failover at all, because their I/O never touches the MDS. Applications doing metadata work pause for the duration.

When action is required

No standby available.

ceph fs status cephfs
# RANK  STATE   MDS
#  0    failed

The rank has nowhere to go. Start an MDS:

ceph orch apply mds cephfs --placement="3 host1 host2 host3"

Repeated crashes. The standby takes over, crashes, the next takes over, crashes. This is a crash loop, and it usually means the MDS is hitting the same condition during recovery each time.

journalctl -u ceph-mds@a --since '1 hour ago' | grep -iE 'assert|abort|signal|FAILED'
coredumpctl list ceph-mds

Out-of-memory kills.

journalctl -k | grep -i 'killed process.*ceph-mds'
ceph config get mds mds_cache_memory_limit

An MDS killed by the OOM killer is a cache sizing or client capability problem, not an MDS bug.

Investigating a crash loop

# what state does it reach before dying?
ceph fs status cephfs

# increase logging for the next attempt
ceph config set mds debug_mds 10
ceph config set mds debug_ms 1
# ... reproduce ...
ceph config set mds debug_mds 1/5

An MDS crashing consistently in rejoin points at client capability state; one crashing in replay points at the journal.

Quiz

Production discipline

Resist intervening in an MDS failover that is progressing — the automatic sequence handles a single crash without help, and interference adds risk. Reserve action for the three cases that need it: no standby available, a crash loop, and out-of-memory kills, each of which has a distinct signature.

Cross-course references

• Kubernetes: a CrashLoopBackOff caused by one bad input has the identical investigation shape
• Linux: a service failing consistently at the same startup stage points at persistent state