Manager failover during an upgrade

Why this matters in production

Every upgrade includes at least one manager failover, and the gap it creates is routinely misread as an upgrade problem.

What a failover interrupts

The manager is not in the data path. Nothing a client does depends on
it being available.

ceph mgr stat
ceph -s | head -8

The expected gap

ceph mgr stat --format json | python3 -c '
import sys,json
d=json.load(sys.stdin)
print("active:", d.get("active_name"), "available:", d.get("available"))'

Typical failover: a few seconds to tens of seconds for the standby to
become active and populate its state.
Metrics and orchestration return once it has.

The last is the discriminator. Client impact during a manager failover
is not caused by the failover.

Verifying the new active

ceph mgr stat
ceph orch status
ceph mgr module ls --format json | python3 -c '
import sys,json; print(json.load(sys.stdin)["enabled_modules"])'
ceph -s

# and that metrics resumed
curl -sf "http://$(ceph mgr services --format json | python3 -c '
import sys,json; print(json.load(sys.stdin).get("prometheus","").split("//")[-1].rstrip("/"))')/metrics" \
  | head -3

A failover that does not complete

ceph mgr stat        # available: false, no active_name
ceph orch ps --daemon-type mgr --refresh

FSID=$(ceph fsid)
journalctl -u "ceph-$FSID@mgr.$(hostname -s)" --since '15 min ago' --no-pager | tail -40

# force a failover if the active is unresponsive but present
ceph mgr fail

Quiz

Production discipline

Treat client I/O errors during a manager failover as a separate investigation — the manager is not in the data path, and an OSD restart in the same window is the far more likely cause. Verify modules loaded on the new active; one may fail there even though it worked on the old.

Cross-course references

• Kubernetes: a controller-manager leader election gap does not affect running pods
• Linux: correlating an unrelated interruption with an outage sends investigations wrong