MDS failover: the sequence and its timings

Why this matters in production

CephFS availability during MDS failure is measured in the seconds these stages take, and each has a different tuning lever. Quoting an availability figure without knowing the stage timings is guessing.

The sequence

1. MDS stops sending beacons
2. Monitors wait mds_beacon_grace          ← detection
3. Monitors mark the rank failed and assign a standby
4. New MDS enters replay                   ← journal replay
5. resolve       (multi-rank only)
6. reconnect     ← clients re-establish sessions
7. rejoin        ← capability state rebuilt
8. active        ← serving again

watch -n 1 'ceph fs status cephfs'

Where the time goes

The two large terms are replay and reconnect.

Reducing replay

ceph fs set cephfs allow_standby_replay true

A standby-replay daemon has already consumed almost all of the journal, so its replay stage is short. This is the single most effective failover tuning available.

Alternatively, reduce the journal:

ceph config set mds mds_log_max_segments 64

Reducing reconnect

The reconnect stage waits for clients to re-establish their sessions. It completes when all known clients have reconnected — or when the timeout expires:

ceph config get mds mds_reconnect_timeout       # 45 s default

A client that is down or unreachable makes this stage run to the full timeout. Reducing the timeout speeds failover at the cost of evicting slow-to-reconnect clients.

Detection

ceph config get mds mds_beacon_grace            # 15 s default
ceph config get mds mds_beacon_interval         # 4 s default

Lowering the grace period detects failures faster and increases the chance of a false positive — an MDS marked failed because of a transient network problem, which triggers an unnecessary failover.

Quiz

Production discipline

Run a controlled failover and record the per-stage timings; the default figures vary enough between deployments that only the measured number belongs in an availability document. Address the largest stage first and re-measure after each change, since the tuning levers are independent and their effects are not additive in an obvious way.

Cross-course references

• Kubernetes: leader election timeout plus controller resync is the same staged recovery
• Linux: cluster fencing and service takeover follow an identical detect-then-recover sequence