A manager module fails after upgrading

Why this matters in production

Manager modules provide the dashboard, metrics, and orchestration, and a module failure can look like a cluster failure while the data path is entirely fine.

Identifying a failed module

ceph -s
ceph health detail

HEALTH_ERR Module 'dashboard' has failed: ...
[ERR] MGR_MODULE_ERROR: Module 'dashboard' has failed

ceph mgr module ls --format json | python3 -c '
import sys,json
d = json.load(sys.stdin)
print("enabled: ", d.get("enabled_modules"))
print("always:  ", [m["name"] for m in d.get("always_on_modules", [])] if isinstance(d.get("always_on_modules"), list) else d.get("always_on_modules"))'

ceph crash ls | tail
ceph log last 200 cephadm | grep -i module

Reading the error

FSID=$(ceph fsid)
journalctl -u "ceph-$FSID@mgr.$(hostname -s)" --since '30 min ago' \
  --no-pager | grep -A20 -i 'traceback\|module.*fail' | tail -60

The traceback names the module and usually the import or attribute that
failed, which is enough to establish whether it is a missing dependency,
a removed internal API, or a configuration incompatibility.

Disabling safely

ceph mgr module disable dashboard
ceph -s

# and confirm the manager is healthy again
ceph mgr stat
ceph orch ps --daemon-type mgr

A failed module can destabilise the manager, so disabling it is the
correct immediate action even if the module is wanted — a stable manager
without the dashboard is better than an unstable one with it.

# some modules cannot be disabled
ceph mgr module ls --format json | python3 -c '
import sys,json
d=json.load(sys.stdin)
aon = d.get("always_on_modules", [])
print("always-on:", aon)'

What each absence affects

Only the orchestrator's absence blocks an upgrade. The rest are
operational conveniences whose loss should be noticed and scheduled for
repair, not treated as an emergency.

# after fixing, re-enable and verify
ceph mgr module enable dashboard
ceph mgr module ls --format json | python3 -c '
import sys,json; print("dashboard" in json.load(sys.stdin)["enabled_modules"])'

Quiz

Production discipline

Disable a failed manager module immediately, even one you need — a module failing on load can destabilise the manager itself, and everything except the orchestrator has a command-line equivalent. Treat a failed prometheus module as urgent: the cluster then runs unmonitored while appearing quiet.

Cross-course references

• Kubernetes: a failed controller in the control plane is disabled before being debugged
• Linux: absence of monitoring data is not itself an alert without a dead man’s switch