Verifying manager state

Why this matters in production

ceph mgr stat reporting available means a manager registered with the monitors — nothing more — and a manager whose dashboard, prometheus, and orchestrator modules all failed on load reports exactly the same thing.

The three views

ceph mgr stat

{"epoch":214,"available":true,"active_name":"ceph-a.qkjvzt","num_standby":2}

ceph mgr dump | python3 -c '
import sys,json
d = json.load(sys.stdin)
print("active:       ", d["active_name"])
print("active_change:", d["active_change"])
print("available:    ", d["available"])
print("standbys:     ", [s["name"] for s in d["standbys"]])
print("services:     ", d.get("services"))'

ceph mgr module ls --format json | python3 -c '
import sys,json
d = json.load(sys.stdin)
print("enabled:  ", ",".join(sorted(d["enabled_modules"])))
for m in d.get("disabled_modules", []):
    if not m.get("can_run") or m.get("error_string"):
        print("BLOCKED %-16s %s" % (m["name"], m.get("error_string") or "cannot run"))'

A healthy manager with a broken module

CRASH_ID=2026-08-18T02:11:04.128Z_9f2c
ceph health detail | grep -A4 -E 'MGR_MODULE_ERROR|MGR_MODULE_DEPENDENCY|RECENT_MGR_MODULE_CRASH'
ceph crash ls | tail -5
ceph crash info ${CRASH_ID}
ceph log last 100 debug cephadm | tail -20

Standby count is a capacity number

ceph orch ls mgr
ceph mgr stat --format json | python3 -c '
import sys,json
d = json.load(sys.stdin)
if not d["available"]:
    print("CRITICAL: no active manager")
elif d["num_standby"] == 0:
    print("WARNING: single manager, next failure is a full outage")
else:
    print("ok: active plus %d standby" % d["num_standby"])'

A cluster with one manager and no standby raises no health warning on its
own. If nobody watches num_standby, nobody finds out until the outage.

What to watch continuously

Quiz

Production discipline

Check ceph mgr module ls alongside ceph mgr stat; the first says whether anything is serving and the second only says whether a daemon registered. Alert explicitly on num_standby reaching zero, because Ceph raises no health warning for a single manager and the deficiency is discovered by the outage otherwise.

Cross-course references

• Kubernetes: a Running pod with a crashed sidecar reports Ready if the probe never covers the sidecar
• Linux: a supervisor that restarts threads silently converts failures into missing work