Placing managers

Why this matters in production

Manager outages are less severe than monitor outages and are frequently treated as equally urgent. Knowing what actually stops — and what does not — sets the right priority and prevents unnecessary intervention.

Active and standby

Exactly one manager is active; the others stand by and take over on failure.

ceph orch apply mgr --placement="2 label:mgr"
ceph mgr stat
ceph mgr dump | jq -r '.active_name, .standbys[].name'
ceph mgr fail

Two is the standard: one active, one standby. Three adds a second standby on large clusters where manager host failures are more likely.

What the active manager provides

ceph mgr module ls
ceph mgr module enable prometheus
ceph mgr services

What a manager outage costs

Stops: the dashboard, metrics collection, the orchestrator, the autoscaler, the balancer, and detailed cluster statistics.

Does not stop: client I/O, OSD operation, monitor consensus, authentication, or recovery.

A cluster with no manager keeps serving data. It cannot be managed through cephadm, produces no metrics, and reports reduced detail in ceph -s — but the storage keeps working.

That distinction sets the priority: a manager outage is urgent because it blinds you, not because it stops the service.

Placement

Manager hosts have the same character as monitor hosts — modest resources, latency more relevant than throughput — and are commonly the same hosts.

ceph orch host label add ceph-mon-01 mgr
ceph orch host label add ceph-mon-02 mgr
ceph orch apply mgr --placement="2 label:mgr"

Distribute them across failure domains for the same reason as monitors, though the consequence of losing all of them is less severe.

Failover

ceph mgr fail
ceph mgr stat

Failover is fast — the standby takes over in seconds. Module state is rebuilt rather than transferred, so a long-running module operation such as an autoscaler evaluation restarts rather than resuming.

Quiz

Production discipline

Distinguish manager outages from monitor outages in incident prioritisation: the first blinds you and the second stops the cluster. Deploy two managers as standard and distribute them across failure domains, treating them as the same class of host as monitors.

Cross-course references

• Kubernetes: losing the controller manager stops reconciliation while running workloads continue
• Linux: a monitoring agent outage is urgent for visibility rather than for service