What happens during MDS recovery

Why this matters in production

An MDS stuck in a recovery state means the filesystem is down, and the state name is the diagnosis. Knowing what each state is waiting for turns “the MDS is stuck in rejoin” from an alarming string into a specific investigation.

The states

ceph fs status cephfs
ceph mds stat

Slow versus stuck

# sample twice, several minutes apart
ceph fs status cephfs
sleep 180
ceph fs status cephfs

# is it making progress?
ceph daemon mds.b perf dump | jq '.mds_log.replayed, .mds.inodes'

Rising counters mean progress. Unchanged counters over several minutes on a large journal mean stuck.

Diagnosing each stuck state

Stuck in replay — the MDS cannot read the journal fast enough or at all.

ceph -s                                    # is the metadata pool healthy?
ceph health detail | grep -i pg
ceph daemon mds.b perf dump | jq '.objecter'

Inactive PGs in the metadata pool stop replay entirely.

Stuck in resolve — another rank is not responding.

ceph fs status cephfs                      # which ranks are up?

All ranks must be present for resolve to complete on a multi-rank filesystem.

Stuck in reconnect — waiting for clients.

ceph daemon mds.b session ls | jq 'length'
ceph config get mds mds_reconnect_timeout

It should end at the timeout at the latest. A reconnect stage lasting far longer indicates something else.

Stuck in rejoin — usually the most common stuck state, and usually a client holding an enormous number of capabilities that must be reconciled.

ceph tell mds.b client ls | \
  jq -r '.[] | "\(.id) \(.num_caps)"' | sort -k2 -rn | head

Intervening

# evict a client blocking rejoin, accepting its buffered-write loss
SESSION_ID=12
ceph tell mds.b client evict id=${SESSION_ID}

# restart the MDS to retry from the beginning
systemctl restart ceph-mds@b

# last resort: mark the rank failed and let a standby retry
ceph mds fail b

Restarting is often the right first move for a genuinely wedged MDS, since the journal is intact and the new attempt starts clean.

Quiz

Production discipline

Sample MDS counters twice before concluding a recovery is stuck; slow and stalled need different responses and look identical in a single snapshot. Set capability limits as an availability control — they bound how long a future recovery can take, which is a stronger argument than the memory one.

Cross-course references

• Kubernetes: a controller stuck in initial sync blocks reconciliation the same way
• Linux: a filesystem stuck in journal recovery presents the identical slow-or-stuck ambiguity