← All break/fix scenarios in Proxmox VE
Ceph MONs cannot form quorum after a network partition
Reported symptoms
- ●ceph -s hangs or returns "mon discovery failed"
- ●Multiple mons report different views of cluster membership
- ●PVE GUI shows "Ceph: connection refused" on all nodes
- ●Cluster is not processing new I/O
Evidence
- · ceph mon dump shows stale or missing entries
- · journalctl -u ceph-mon@<id> shows election timeouts
- · iptables -L -n shows no ceph rules (default 6789/3300/6800-7300)
- · Network captures show mons timing out on heartbeat packets
Diagnosis and resolutionclick to reveal
Root cause
A network partition (split-brain) has separated the monitors. With even a single remaining mon, ceph refuses to operate to avoid split-brain data corruption. Common causes: - Switch or VLAN misconfiguration - Firewall rules blocking mon traffic (port 6789) - Corrupted mon store (rare; usually requires recovery procedure)
Remediation
1. Restore network connectivity first. Check: - `ping <other-mon-host>` from each mon host - Switch logs / port state - Firewall rules: `iptables -L -n | grep 6789` 2. If connectivity is restored but quorum is still broken: `systemctl restart ceph-mon@<id>` on all mons 3. If a single mon has a corrupted store: - Stop the mon on the affected host - Move `/var/lib/ceph/mon/ceph-<id>/store.db` aside - Recover from another mon: `ceph-monstore-tool /var/lib/ceph/mon/ceph-<other-id>/store.db rebuild -- --mon-ids <id> --mon-ips <ip>` - Restart the mon 4. Verify: `ceph -s` should show all mons and HEALTH_OK
Verification
- All monitors are in the monmap - `ceph -s` returns HEALTH_OK - OSDs report active+clean - VMs on the Ceph pool resume normal I/O
Prevention
- Use a dedicated network for Ceph cluster traffic - Configure 5 monitors for a 5-node cluster (or 3 mons + qdevice) - Monitor mon health with a separate check - Document the mon recovery procedure and test it