Correlated failure in a hyper-converged cluster

Why this matters in production

A node failure in a dedicated deployment is either a compute event or a storage event. In a hyper-converged one it is both, simultaneously, on reduced capacity.

The sequence

flowchart TD
  A[Node fails] --> B[VMs on that node stop]
  A --> C[OSDs on that node go down]
  B --> D[Proxmox HA restarts VMs elsewhere]
  C --> E[Ceph waits mon_osd_down_out_interval]
  D --> F[Surviving nodes take more compute load]
  E --> G[Ceph begins recovery]
  G --> H[Surviving nodes take more storage load]
  F --> I[Both loads land on the same reduced capacity]
  H --> I

The restarted VMs and the recovery traffic arrive on the same surviving nodes at roughly the same time.

Quantifying the impact

4-node cluster, one node lost:
  compute: 25% of VMs restart on 3 nodes → each takes +33% load
  storage: 25% of OSDs gone → recovery moves that data onto 3 nodes
  network: recovery traffic plus additional VM traffic
  result:  surviving nodes at ~133% of prior compute plus recovery I/O

# what a node holds
ceph osd df tree | awk '/host/ {h=$NF} /osd\./ {s[h]+=$5} END {for (k in s) print k, s[k]}'
qm list | wc -l
ha-manager status

Planning capacity for the combined recovery

The sizing rule is stricter than for either layer alone:

compute: (N-1) nodes must run all VMs
storage: (N-1) nodes must hold all data below backfillfull
network: (N-1) nodes must carry recovery plus the redistributed VM traffic

# compute headroom
for n in $(pvecm nodes | awk 'NR>1 {print $3}'); do
  echo -n "$n "
  ssh "$n" "grep -c ^processor /proc/cpuinfo; free -g | awk '/Mem:/ {print \$2}'"
done

A cluster running at 80% CPU across four nodes cannot absorb a node failure: three nodes at 107% of capacity means VMs will not start or will run degraded.

Reducing the correlation

# stagger the restarts
ha-manager set vm:100 --max_restart 1 --max_relocate 1

Quiz

Production discipline

Size compute, storage, and network each for N-1 nodes; a node failure requires all three simultaneously and each layer sized separately misses that. Stagger Proxmox HA restarts so booting guests do not compete with recovery on the same devices.

Cross-course references

• Kubernetes: node failure triggers rescheduling and volume reattachment together
• Linux: correlated failures always cost more than the sum of their parts