Manager failover — what happens and what to verify

Why this matters in production

Manager failover is one of the few Ceph behaviours that is genuinely simple, and testing it takes one command. Clusters that have never tested it discover during an incident that the standby was never actually running.

The sequence

1. Active manager stops or becomes unresponsive
2. Monitors notice and select a standby
3. The chosen standby becomes active
4. It loads its modules: dashboard, prometheus, orchestrator, ...
5. Statistics begin collecting from scratch

Elapsed time is typically a few seconds to select, plus however long module loading takes — the dashboard and orchestrator take longest.

ceph mgr stat
ceph mgr dump | jq '{active: .active_name, standbys: [.standbys[].name]}'
ceph mgr fail                 # fail the active, promoting a standby

What is lost

• In-flight dashboard sessions. Users reconnect and log in again.
• Accumulated statistics. Rate calculations restart, so some Prometheus counters reset and rate graphs show a brief artefact.
• In-progress orchestrator work. Reconciliation resumes on the new active, so scheduled work is retried rather than lost.

What is preserved

• Everything durable. Module configuration lives in the monitor config store, not in the manager.
• The cluster map, obviously — managers hold no authority over it.
• Orchestrator service specifications.
• Dashboard users and roles.

Placement for failover

Two managers on the same host provide no failover for host failure. Two managers in the same rack provide none for rack failure.

ceph orch ps --daemon-type mgr
ceph orch apply mgr --placement="ceph-01 ceph-05"

Choose hosts in different failure domains, and verify placement after any cluster change — managers drift for the same reasons monitors do.

Verification checklist

• A standby exists and is running: ceph mgr dump.
• The two managers are on different hosts and different racks.
• ceph mgr fail promotes cleanly and ceph -s recovers within a minute.
• The dashboard URL still resolves after failover.
• Prometheus resumes scraping without manual intervention.

Quiz

Production discipline

Run ceph mgr fail as a scheduled test — quarterly is enough — and verify status, dashboard, and scraping recover afterwards, because the failure this catches is a standby that is declared but not running. Alert on daemons running below their declared count and specifically on an empty standby list. Place the two managers in different failure domains and re-verify after cluster changes. And write alerting rules with rate() rather than raw differences, so a failover does not fire them.

Cross-course references

• Ceph: Part X lesson on the manager role for what failover affects.
• Ceph: Part LXXVII (Alerting) for writing rules that tolerate resets.
• Ceph: Part LXXVIII (Monitoring the Monitoring Path) for the wider problem.