Ceph quorum loss with VMs running

Advanced⏱ ~17 mincephqm

What you'll learn

Predict guest behaviour during quorum loss
Estimate how long guests survive
Prioritise the response correctly
Reduce the exposure

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Quorum loss stops the Ceph control plane. Running guests do not fail immediately, and understanding why determines how much time the response has.

What guests experience

Quorum lost
  → no new OSD map updates
  → existing client connections continue with their current map
  → reads and writes to reachable OSDs continue
  → any operation needing a map change blocks
  → new client connections cannot authenticate

Guest activity	Behaviour
Reads from cached data	unaffected
Reads from reachable OSDs	continue
Writes to reachable OSDs	continue for a period
Operations needing a map update	block
A VM starting	fails; cannot authenticate
A VM migrating	fails
An OSD failing during the outage	its PGs block; no remapping possible

The critical point: guests do not stop the moment quorum is lost. They degrade as operations requiring the monitors accumulate.

How long they survive

ceph config get osd mon_client_hunt_interval
ceph config get global rados_mon_op_timeout

There is no single number — it depends on what the guests do. A guest performing only reads from OSDs it already knows about can continue indefinitely. A guest whose writes trigger any map-dependent operation blocks quickly.

Practically:
  minutes for a busy write workload
  longer for a read-heavy one
  immediately for anything starting, migrating, or reconnecting

Prioritising the response

1. Restore quorum — everything else is secondary
2. Do not start or migrate VMs during the outage
3. Do not restart guests hoping to clear errors
4. Do not restart OSDs, which would need to re-authenticate
5. Once quorum returns, verify cluster health before resuming operations

# from each monitor host, since the CLI will hang
ceph daemon mon.$(hostname -s) mon_status | python3 -c '
import sys,json; d=json.load(sys.stdin)
print("state:", d["state"], "quorum:", d.get("quorum"))'

Restarting OSDs during quorum loss is the mistake that turns a recoverable situation into a worse one: an OSD that restarts cannot obtain a map or authenticate, so it stays down and its PGs become unavailable even after quorum returns.

Reducing the exposure

Measure	Effect
Monitors across failure domains	a single domain failure does not lose quorum
Five monitors rather than three	tolerates two failures
Monitor store on reliable, monitored storage	avoids the full-filesystem cause
Clock synchronisation monitored	avoids the skew cause
Alert on quorum margin, not loss	intervention before the outage
Rehearsed recovery	shorter outage when it happens

Quiz

Knowledge check · 4 questions

Q1. Why do running VMs survive Ceph quorum loss while new ones cannot start?

Running VMs keep their whole working set in the host page cache, so guest reads never reach the cluster at all and writes are absorbed there until that cache is flushed by the guest filesystem Proxmox buffers writes on the node while the cluster is unreachable and replays them once the monitors return, which a starting VM has no buffer for The monitors delegate map distribution to the managers when quorum is lost, and a manager can serve established clients but cannot authenticate new ones A connected client holds a current OSD map and authenticated sessions, and the data path does not involve monitors; a new client must authenticate and fetch maps, which requires a monitor
Q2. An OSD restarted during a quorum outage stays down until quorum returns, taking its PGs with it.

True False
Q3. Respond to Ceph quorum loss with VMs running.

Two of three Ceph monitors are down. VMs are still running but some report I/O errors. The team is considering restarting services to clear the errors.
Q4. Which guest activities block immediately during quorum loss?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Change nothing but the monitors during a quorum outage — a restarted OSD cannot authenticate or fetch a map, so it stays down and takes its PGs with it even after quorum returns. Alert on quorum margin rather than quorum loss; the margin alert arrives while the cluster is still fully working.

Cross-course references

Kubernetes: etcd loss leaves running pods working while nothing new can be scheduled
Linux: control plane and data plane separation produces this asymmetry everywhere