Skip to main content
RunBook Academy

← All runbooks in Ceph

medium riskservice affecting~30 min

Recover a failed MDS daemon

1 · Prerequisites

Confirm every item is in place before any state change.

  • The active MDS daemon is lost
  • CephFS clients may see stale file handles

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

  • · ceph -s reports the MDS state
  • · ceph mds stat lists active and standbys
  • · CephFS mounts return errors
  • · The host is reachable

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

  1. 1Wait for the standby to take over (typically < 30s)
  2. 2If no standby: ceph orch daemon add mds <host>
  3. 3Verify the new MDS is active: ceph mds stat
  4. 4Clients remount to re-establish caps

4 · Verification

Confirm the procedure actually fixed the problem.

  • ✓ceph mds stat reports a healthy active MDS
  • ✓CephFS mounts respond to ls and stat
  • ✓Clients reconnect
  • ✓Cluster is HEALTH_OK

5 · Rollback

If verification fails, undo the procedure in reverse order.

  • ↶If the standby does not take over: check the host, time, network
  • ↶If MDS replay stalls: plan a journal reset
  • ↶If clients cannot reconnect: check the client kernel

6 · Escalation

When the runbook isn't enough, contact:

  • · Storage team
  • · Vendor support for the MDS issues

Recover a failed MDS daemon

This runbook covers Recover a failed MDS daemon on a Ceph Tentacle cluster.

When to use

  • Active production scenario matching the impact (service-affecting)
  • The prerequisites are met

Prerequisites

Before starting this runbook, verify each prerequisite has been met. If a prerequisite is not met, resolve it first or escalate.

  • The active MDS daemon is lost
  • CephFS clients may see stale file handles

Pre-checks

Collect evidence before taking action. Evidence is the proof the runbook worked.

# ceph -s reports the MDS state
# ceph mds stat lists active and standbys
# CephFS mounts return errors
# The host is reachable

Procedure

Step 1

Wait for the standby to take over (typically < 30s)

Step 2

If no standby: ceph orch daemon add mds <host>

Step 3

Verify the new MDS is active: ceph mds stat

Step 4

Clients remount to re-establish caps

Verification

After completing the procedure, verify the result:

# ceph mds stat reports a healthy active MDS
# CephFS mounts respond to ls and stat
# Clients reconnect
# Cluster is HEALTH_OK

Rollback

If the procedure does not produce the expected verification, roll back:

  • If the standby does not take over: check the host, time, network
  • If MDS replay stalls: plan a journal reset
  • If clients cannot reconnect: check the client kernel

Escalation

If you cannot roll back or the rollback fails, escalate:

  • Storage team
  • Vendor support for the MDS issues

References

  1. Ceph documentation