Total storage loss with VMs running

Advanced⏱ ~18 mincephqmha-manager

What you'll learn

Predict guest behaviour during total storage loss
Sequence the response
Recover guests after storage returns
Prepare so the scenario is survivable

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

If Ceph is unavailable, every VM with an RBD disk loses its storage. The guests do not crash immediately, which creates both an opportunity and a trap.

What guests do

Storage unavailable
  → in-flight I/O never completes
  → the guest's block layer retries until its timeout
  → the timeout expires; I/O errors are returned
  → the filesystem remounts read-only, or the guest hangs
  → processes in uninterruptible sleep accumulate
  → the guest becomes unresponsive but does not power off

Guest state	Recoverable by
Hung in I/O wait, filesystem intact	storage returns before the timeout; nothing needed
Filesystem remounted read-only	remount after storage returns
Filesystem corrupted by a partial write	fsck, possibly restore
Application state lost	application-level recovery

Guests that are entirely idle may survive with no visible effect.

The trap

Proxmox HA sees a node whose VMs are unresponsive
  → it may attempt to fence and restart them
  → the restarts fail, because storage is unavailable
  → HA retries, accumulating failed starts
  → when storage returns, HA may start VMs in an unexpected order

# stop HA from fighting the outage
ha-manager set vm:100 --state disabled
# or, cluster-wide
for v in $(ha-manager status | grep -oE 'vm:[0-9]+'); do
  ha-manager set "$v" --state disabled
done

Disabling HA during the outage prevents it consuming the response with failed restart attempts.

Sequencing the response

1. Confirm the scope — is Ceph reachable at all?
2. Disable Proxmox HA so it stops attempting restarts
3. Do not shut down or reset guests — they may recover intact
4. Restore Ceph; this is the entire task
5. Once storage returns, assess each guest before acting
6. Re-enable HA after the fleet is stable

ceph -s --connect-timeout 5
ceph daemon mon.$(hostname -s) mon_status 2>/dev/null

Recovering guests after storage returns

# per guest, in order of preference
qm guest exec 100 -- mount -o remount,rw /
qm guest exec 100 -- dmesg | tail -20

Guest state	Action
Responsive, filesystem read-write	nothing
Responsive, filesystem read-only	remount read-write
Unresponsive but running	reset; the filesystem journal replays
Filesystem errors on boot	fsck
Will not boot	restore from backup

qm reset 100

Working through the fleet in this order means the least invasive action is tried first for each guest.

Preparing

Preparation	Effect
Guest disk timeouts raised	short outages become invisible
Backups tested by restoring	the last resort actually works
Critical VMs with local storage	a subset survives independently
Monitors across failure domains	the scenario is far less likely
Rehearsed monitor recovery	the outage is shorter
HA behaviour understood in advance	it is disabled promptly

Quiz

Knowledge check · 4 questions

Q1. Why do guests hang rather than crash when Ceph becomes unavailable?

QEMU pauses the guest deliberately when its disk backend stops responding, which is why the console freezes rather than erroring The block layer retries until the device timeout expires, leaving processes in uninterruptible sleep with memory state intact The guest kernel panics silently, and without a serial console or crash dump configured the VM simply stops responding to input Proxmox suspends the VM to protect its filesystem and resumes it automatically once the storage backend answers again
Q2. Resetting unresponsive guests early speeds up the recovery.

True False
Q3. Respond to total storage unavailability.

Ceph has become entirely unreachable. VMs are unresponsive. Proxmox HA is attempting to restart them and failing repeatedly.
Q4. In what order should guests be assessed after storage returns?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Disable Proxmox HA early during a storage outage — its failed restart attempts consume the response and it may start VMs in an unchosen order when storage returns. Do not reset guests: a guest hung in I/O wait has intact memory state and resumes on its own if storage returns before its timeout.

Cross-course references

Kubernetes: pods stuck on unavailable volumes should not be force-deleted prematurely
Linux: uninterruptible sleep on a stalled device is recoverable if the device returns