Skip to main content
RunBook Academy

← All labs in Proxmox VE

Lab · intermediate · ~60 min

Configure HA for a VM and verify failover

A · Physical hardwareB · Nested virtualisation

Objectives

  • Add a VM to HA management with a preference and priority
  • Trigger a node failure and watch the CRM restart the VM on a survivor
  • Verify the watchdog self-fences a stuck node
  • Confirm the failed VM comes back into service automatically

Prerequisites

  • A 3-node PVE cluster
  • Shared storage accessible from all nodes (Ceph, NFS, or shared iSCSI)
  • At least one VM running on shared storage

Configure HA and verify failover

This lab proves that HA actually works in your environment, not just that the toggle is set. Until you have done a real failover drill, you don’t have HA — you have HA configured.

Steps

1. Configure fencing

Each node needs a hardware watchdog. Most server hardware supports this out of the box; nested virtualisation needs it emulated.

# Check the watchdog device exists
ls -la /dev/watchdog
systemctl status systemd-modules-load

If the watchdog is not present, enable softdog as a fallback (not for production but fine for this lab):

echo "softdog" >> /etc/modules-load.d/watchdog.conf
modprobe softdog

2. Add the VM to HA

In the GUI: Datacenter → HA → Add → HA-managed VM.

  • VM: 100
  • Group: ha-group-1 (default)
  • Request state: started
  • Max restart: 1
  • Max relocate: 1

Or via the CLI:

ha-manager add vm:100 --state started --group ha-group-1
ha-manager status

3. Verify HA state

ha-manager status
# Look for vm:100 with status=started, group=ha-group-1

4. Trigger a failure (controlled)

Run on the node hosting vm:100:

# Option A: graceful migration to a different node first
ha-manager migrate vm:100 pve-02

# Option B: simulate a crash (more aggressive)
echo c > /proc/sysrq-trigger    # WARNING: hard reboot

For the lab, prefer Option A first. Then test Option B once you are confident the failover works.

5. Watch the recovery

Within 60 seconds:

  • The original node becomes unreachable
  • The CRM on the surviving master restarts vm:100 on a survivor
  • ha-manager status shows vm:100 as started on the new host
  • The VM’s IP becomes reachable again

6. Verify fencing

If you used Option B, after the failed node boots:

# On the survivor that hosted vm:100 after failover:
journalctl -u corosync -n 200
grep -i fence /var/log/syslog

You should see fencing actions issued by the CRM.

Verification

  • vm:100 is started after a simulated node failure
  • The failover completes within 60 seconds
  • ha-manager status reports no errors
  • Watchdog self-fence log entry exists when Option B is used

Cleanup

Remove the HA-managed VM from HA before tearing down the lab:

ha-manager remove vm:100

Deliverables

  • · ha-manager status output listing the VM as managed
  • · A simulated node failure that triggers HA recovery in under 60 seconds

Verification status

Executed end to end
not yet run on hardware

The commands and configuration here have been reviewed against the verified software versions, but nobody has run this lab start to finish on a system meeting its prerequisites. Treat the Expected Outcome as the intended result rather than an observed one, and keep the Cleanup section to hand.