Replacing a manager daemon

Why this matters in production

ceph orch is a module running inside the active manager, so asking it to replace that manager is asking a process to remove itself — and the command output goes with it.

Fail it over before you touch it

ceph orch ps --daemon-type mgr

NAME              HOST    PORTS  STATUS         REFRESHED  MEM USE
mgr.ceph-a.qkjvzt ceph-a  *:9283 running (3d)   2m ago     612M
mgr.ceph-b.hzlwmt ceph-b  *:9283 running (3d)   2m ago     201M

# make the target a standby first
ceph mgr stat
ceph mgr fail ceph-a.qkjvzt
ceph mgr stat

The cephadm path

# rebuild the container in place, same name and keyring
ceph orch daemon redeploy mgr.ceph-a.qkjvzt

# remove it entirely — but read the spec first
ceph orch ls mgr --export
ceph orch daemon rm mgr.ceph-a.qkjvzt --force

If the manager service spec still places a manager on that host,
cephadm recreates it within a reconcile interval. Removing a daemon is
not the same as changing where managers run.

# to actually move a manager, change the placement
ceph orch apply mgr --placement="ceph-b,ceph-c,ceph-d"
ceph orch ps --daemon-type mgr --refresh

Without an orchestrator

sudo -u ceph mkdir -p /var/lib/ceph/mgr/ceph-ceph-d
ceph auth get-or-create mgr.ceph-d \
  mon 'allow profile mgr' osd 'allow *' mds 'allow *' \
  -o /var/lib/ceph/mgr/ceph-ceph-d/keyring
sudo systemctl enable --now ceph-mgr@ceph-d

sudo systemctl status ceph-mgr@ceph-d --no-pager | head -5
ceph mgr dump | python3 -c '
import sys,json
d = json.load(sys.stdin)
print("standbys:", [s["name"] for s in d["standbys"]])'

Prove the replacement works

ceph mgr fail                     # promote the new one deliberately
sleep 20
ceph mgr stat
ceph mgr module ls --format json | python3 -c '
import sys,json
d = json.load(sys.stdin)
bad = [m["name"] for m in d.get("disabled_modules", []) if m.get("error_string")]
print("enabled:", len(d["enabled_modules"]), " with errors:", bad or "none")'
ceph orch status
ceph mgr services

Quiz

Production discipline

Run ceph mgr fail before any orchestrator operation that touches the active manager — otherwise the daemon executing the change is the one being changed, and the result is ambiguous rather than wrong. Promote each standby deliberately at least once so you learn it can serve before an incident does.

Cross-course references

• Kubernetes: an operator that reconciles its own Deployment has the same self-reference problem
• Linux: a hot spare that has never carried load is an assumption, not a spare