Operating a cluster with no manager

Why this matters in production

With no manager the cluster serves every read and write exactly as before, and the first job in the incident is to say so before anyone starts an emergency that is not needed.

What keeps working

Clients get the osdmap from the monitors and talk to OSDs directly. RBD,
CephFS, and RGW data paths never involve a manager.

ceph osd tree
ceph osd dump | head -5
ceph mon dump
ceph auth ls | head -5
ceph osd set noout
ceph fs dump | head -20
ceph mds stat
ceph health detail

What stops

ceph -s

  cluster:
    health: HEALTH_WARN
            no active mgr
  services:
    mon: 3 daemons, quorum ceph-a,ceph-b,ceph-c (age 2h)
    mgr: no daemons active (since 4m)

The scheduled work is the part that matters after the incident. Nothing
records that a snapshot schedule did not fire.

Recovering without the orchestrator

# on any host that had a manager
sudo cephadm ls | python3 -c '
import sys,json
for d in json.load(sys.stdin):
    if d["name"].startswith("mgr."):
        print(d["name"], d["state"], d.get("systemd_unit"))'

sudo systemctl start ceph-$(ceph fsid)@mgr.ceph-a.qkjvzt
sudo journalctl -u ceph-$(ceph fsid)@mgr.ceph-a.qkjvzt -n 50 --no-pager

# no manager container survives anywhere — mint a key and deploy by hand
ceph auth get-or-create mgr.ceph-d \
  mon 'allow profile mgr' osd 'allow *' mds 'allow *' \
  -o /tmp/mgr.ceph-d.keyring
sudo cephadm deploy --fsid $(ceph fsid) --name mgr.ceph-d \
  --config /etc/ceph/ceph.conf --keyring /tmp/mgr.ceph-d.keyring

`ceph auth get-or-create` is a monitor command and works throughout. That
is what lets you mint a manager keyring during a manager outage.

Verifying

ceph mgr stat
ceph orch status
ceph df
ceph balancer status
ceph mgr module ls --format json | python3 -c '
import sys,json
d = json.load(sys.stdin)
print("enabled:", len(d["enabled_modules"]))
for m in d.get("disabled_modules", []):
    if m.get("error_string"):
        print("ERROR", m["name"], m["error_string"])'

Quiz

Production discipline

Alert on manager availability from the monitors, never from the manager metrics endpoint — an endpoint cannot report its own absence. Keep the fsid, the manager host list, and a working cephadm binary recorded somewhere that does not require the orchestrator, because during a total manager outage they are the only inputs the recovery has.

Cross-course references

• Kubernetes: a control plane outage leaves running workloads serving while nothing can be changed
• Linux: a monitoring agent cannot alert on its own failure; the check must live elsewhere