HA failover lab

This lab turns on HA for a real VM and exercises the recovery path by fencing a node. The output is a measured RTO and a validated VM.

Steps

1. Create an HA group with node preferences

In the GUI: Datacenter → HA → Groups → Create.

ID: web-tier
Nodes: pve-01:2, pve-02:1, pve-03:1
nofailback: 0

The priorities mean: prefer pve-01, then pve-02, then pve-03.

2. Add the VM to HA

ha-manager add vm:100 --group web-tier \
  --max-restart 3 --max-relocate 2

Verify:

ha-manager status
# Expected: vm:100 listed, state "started", group "web-tier"

3. Confirm the VM is running

qm status 100
# Expected: running

4. Fence a node (simulate failure)

On the node that is not running VM 100:

# Pick a node to fence (the worst-case scenario: the node hosting VM 100)
ssh root@pve-01 'echo c > /proc/sysrq-trigger'
# This crashes the kernel immediately. Use ONLY in a lab.

Alternatively, in a nested lab:

qm stop 9001 --forceStop  # the nested PVE VM

5. Watch HA react

From pve-02 or pve-03:

ha-manager status
# Expected: vm:100 state changes through "started -> pending -> started" on the recovery node

Time the recovery:

date_start=$(date +%s)
# Wait for the VM to come back up on another node
while ! ha-manager status | grep -q "vm:100.*started"; do
  sleep 1
done
date_end=$(date +%s)
echo "RTO: $((date_end - date_start)) seconds"

6. Verify the VM

qm status 100
# Expected: running on a different node
# Inside the VM: application is up, data is intact

7. Bring the failed node back

# Boot pve-01 back up

Watch the cluster rebalance:

pvecm status
# Expected: 3 nodes, all online
ha-manager status
# Expected: vm:100 still on its current node (nofailback=0 means
# it can move back; check the migration if you want)

Verification

• VM came back online within your RTO budget (typically 30-120s)
• No data loss; application resumed cleanly
• Cluster recovered quorum and returned to 3 nodes
• HA group priorities were honoured (VM went to pve-02 or pve-03 first)

Cleanup

ha-manager remove vm:100
# On the failed node (now back up):
# Nothing to clean; HA state updates automatically

Notes

This lab intentionally uses echo c > /proc/sysrq-trigger for a hard crash. In production, prefer testing via fencing agent (IPMI, iLO, iDRAC) so the failed node can be cleanly powered off rather than crashed.