Module state across a manager failover

Why this matters in production

Module configuration survives a manager failover because it never lived in the manager, and the thing that is genuinely lost is every periodic task that had not yet fired.

Where module state actually lives

ceph config dump --format json | python3 -c '
import sys,json
for e in json.load(sys.stdin):
    if e["name"].startswith("mgr/"):
        print("%-44s %s" % (e["name"], e["value"]))'

ceph config-key ls --format json | python3 -c '
import sys,json
for k in json.load(sys.stdin):
    if k.startswith("mgr/"):
        print(k)' | head -20

What survives and what does not

ceph balancer status
ceph progress
ceph device ls | head -5

The observability discontinuity

The prometheus endpoint is served by the active manager, so a failover
produces a scrape gap that spans the promotion plus the module warm-up.

ceph mgr services
curl -sf http://$(hostname -f):9283/metrics | wc -l

Re-establishing state deliberately

ceph mgr module ls --format json | python3 -c '
import sys,json
d = json.load(sys.stdin)
print("enabled:", ",".join(sorted(d["enabled_modules"])))
for m in d.get("disabled_modules", []):
    if m.get("error_string"):
        print("ERROR", m["name"], m["error_string"])'
ceph balancer status
ceph orch status
ceph health detail

Quiz

Production discipline

After any manager failover, check ceph mgr module ls for a non-empty error string and ceph balancer status for a reset optimisation cycle — neither raises a health warning. Treat a moving active_change timestamp as an alert in its own right, because a flapping manager starves every periodic task while the cluster reports perfect health.

Cross-course references

• Kubernetes: a CronJob controller that restarts before the schedule fires also misses it silently
• Linux: work scheduled by an in-process timer disappears with the process, without an error