Recover a failed MGR daemon

This runbook covers Recover a failed MGR daemon on a Ceph Tentacle cluster.

When to use

• Active production scenario matching the impact (service-affecting)
• The prerequisites are met

Prerequisites

Before starting this runbook, verify each prerequisite has been met. If a prerequisite is not met, resolve it first or escalate.

• The active MGR daemon is lost
• A standby MGR should take over within seconds

Pre-checks

Collect evidence before taking action. Evidence is the proof the runbook worked.

# ceph -s reports the MGR state
# ceph mgr stat lists active and standbys
# The host is reachable
# Time and DNS are healthy

Procedure

Step 1

Wait for the standby to take over (typically < 30s)

Step 2

If no standby: ceph orch daemon add mgr <host>

Step 3

Verify the new MGR is active: ceph mgr stat

Step 4

Re-enable the modules the failed MGR had: ceph mgr module enable <module>

Step 5

Verify the modules are reporting: ceph mgr module ls

Verification

After completing the procedure, verify the result:

# ceph -s reports a healthy active MGR
# Modules are enabled and reporting
# Dashboard is reachable
# Cluster is HEALTH_OK

Rollback

If the procedure does not produce the expected verification, roll back:

• If a standby does not take over: check the host, time, network
• If modules fail to enable: verify the module name and the MGRs container

Escalation

If you cannot roll back or the rollback fails, escalate:

• Storage team
• Vendor support for module issues