HA in depth: groups, constraints, affinity, and the failure matrix

Basic HA — adding a VM to HA management — is straightforward. The hard parts are:

• Choosing which nodes a VM can run on (groups, restrictions)
• Keeping related VMs together (positive affinity) or apart (negative affinity)
• Understanding exactly what triggers HA recovery and what doesn’t
• Testing that failover actually works

This lesson covers the operational depth that the basic HA lessons don’t.

HA groups

An HA group is a set of nodes with priorities. VMs in the group run on the highest-priority node that’s healthy. If that node fails, the VM migrates to the next-highest-priority node.

Create a group:

ha-manager groupadd production-grp \
  --nodes "pve-01:50,pve-02:100,pve-03:50" \
  --restricted 1 \
  --nofailback 0

This means:

• pve-02 is preferred (score 100)
• pve-01 and pve-03 are fallbacks (score 50 each)
• restricted means VMs in this group can ONLY run on these nodes
• nofailback 0 means if a failed node comes back, VMs move back to it

The scores don’t have to be 50/100; any positive integer works. Picking the right pattern:

• Single primary, multiple fallbacks — pve-01:100,pve-02:50,pve-03:50 for VMs that should normally run on the fastest node.
• Any-of-many — pve-01:50,pve-02:50,pve-03:50 for stateless workloads that can run anywhere.
• Restricted vs unrestricted — restricted means “this VM can never run outside these nodes”. Use restricted for compliance (PCI, GDPR); unrestricted gives HA more freedom.

Apply a VM to a group:

ha-manager add vm:100 \
  --group production-grp \
  --state started \
  --max_relocate 1 \
  --max_restart 1

max_relocate and max_restart are the recovery policy:

• max_restart N — try restarting the service N times on the same node before giving up
• max_relocate N — try relocating (live migration) to another node N times before giving up

For a typical VM: max_restart 1, max_relocate 3 — try once on the current node, then try up to 3 other nodes.

Affinity rules

Affinity rules tell HA which VMs should run together or apart.

Positive affinity (keep together)

# Application cluster: web-01, web-02, web-03 should be on different nodes
ha-manager add vm:web-01 --group web-tier --affinity 1
ha-manager add vm:web-02 --group web-tier --affinity 1
ha-manager add vm:web-03 --group web-tier --affinity 1
# Wait — that's the wrong syntax. Use ha-manager group config:

Affinity is configured via ha-manager group parameters, not per-VM. The correct pattern:

# All web-tier VMs in the same group will try to spread across nodes
ha-manager groupadd web-tier --nodes "pve-01:50,pve-02:50,pve-03:50"
ha-manager add vm:web-01 --group web-tier
ha-manager add vm:web-02 --group web-tier
ha-manager add vm:web-03 --group web-tier

# HA will try to place them on different nodes if possible
# (this is automatic when all nodes have equal score)

Negative affinity (keep apart)

ha-manager add vm:db-master --group db-tier
ha-manager add vm:db-replica --group db-tier
# db-master and db-replica will be on different nodes by default

For more complex affinity (e.g., “web and db must be on the same node” or “two specific VMs must NOT be on the same node”), use PVE rules instead:

# In the VM config or via ha-manager
ha-manager add vm:web-01 --group web-tier --constraint "db-master:pve-01"

This is rarely needed and gets complex fast. For most clusters, the group-based default is sufficient.

The HA failure matrix

Not every failure triggers HA recovery. Understanding what does and doesn’t is essential:

The gap between “what HA does” and “what production expects” is the main source of HA-related outages. To close it:

• Service health checks: write your own monitoring that detects hangs and triggers HA action manually.
• Watchdog: PVE’s software watchdog (softdog) or hardware watchdog will force-reboot a hung node, which DOES trigger HA recovery.

Configure the watchdog:

# /etc/pve/priv/host.fw
# Enables watchdog on all cluster nodes
# Or per-node:
echo 'softdog' > /etc/modules-load.d/watchdog.conf
modprobe softdog

A real hardware watchdog (IPMI, iLO, etc.) is preferred because it works even when the kernel is hung.

Manual failover

For planned maintenance or testing:

# Migrate HA-managed VM to a specific node
ha-manager migrate vm:100 pve-02

# Relocate to a different group (e.g., maintenance group)
ha-manager relocate vm:100 maintenance-grp

# Take VM out of HA temporarily (but keep running)
ha-manager set vm:100 --state stopped
# Or fully remove from HA management
ha-manager remove vm:100

What HA doesn’t do

• HA does not protect against silent data corruption. Use ZFS scrub
  • replication for that.
• HA does not provide cross-site failover. For multi-site, use Ceph replication or PBS for off-site backup.
• HA does not guarantee zero downtime. Recovery takes 30–90 seconds for the CRM to detect the failure and start the VM elsewhere.
• HA does not replace backups. A VM with corrupted data will be failed-over with the same corrupted data.
• HA does not handle live migration gracefully during maintenance unless you do ha-manager migrate first.

Testing HA failover

The single most important operational habit: test failover quarterly.

# 1. Verify all VMs are healthy and HA-managed
ha-manager status

# 2. Pick a non-critical VM or use a test VM
qm create 900 --memory 1024 --net0 name=eth0,bridge=vmbr0,ip=dhcp \
  --scsi0 local-zfs:8 --ostype l26
qm start 900
ha-manager add vm:900 --group test-grp

# 3. Verify HA state
ha-manager status | grep 900
# Expected: state=started, group=test-grp, node=pve-XX

# 4. Simulate failure
# Option A: stop corosync on the host (clean fence)
ssh $(ha-manager status | grep $(ha-manager config | grep 900) | awk '{print $NF}') systemctl stop corosync
# The node loses cluster connectivity and self-fences after timeout

# Option B: hard reboot (simulates crash)
ssh pve-XX 'echo b > /proc/sysrq-trigger'
# Or via IPMI
ipmitool -H pve-XX -U admin chassis power cycle

# 5. Watch the recovery (on a surviving node)
watch -n 1 'ha-manager status'
# Expected: VM 900 transitions from started (pve-XX) to started (pve-YY) within 60s

# 6. Verify the VM is healthy on the new node
ssh pve-YY qm status 900
ssh pve-YY pct exec 900 -- systemctl status

# 7. Restore the failed node
ssh pve-XX systemctl start corosync
# VM stays where HA put it; you can migrate back manually

For a complete matrix test, do all four:

• Stop corosync on each node, one at a time
• Reboot each node via IPMI, one at a time
• Pull the network cable on each node, wait for self-fence, restore
• Kill the qemu process of an HA-managed VM (kill -9 $(pidof qemu))

Document the result of each test in your runbook.

Production considerations

• HA is not magic. It speeds up recovery; it doesn’t prevent failure. A 60-second failover is better than a 4-hour outage, but it’s still 60 seconds of downtime.
• Watch your resource headroom. HA requires capacity to migrate to. A cluster running at 80% capacity has no room to recover from a full-node failure. Plan for N+1 at minimum, N+2 for safety.
• Test failover on every configuration change. Adding storage, changing networks, upgrading kernels — each can affect HA.
• HA groups should match your application architecture. Web VMs in one group, databases in another (with anti-affinity to web if the database must be on a different node). Don’t put everything in one group unless you really want it all on one node.

Common mistakes

• Putting everything in one HA group. All VMs in one group try to land on one node, defeating the point of HA.
• No capacity headroom. HA can’t migrate VMs if the target node has no CPU/RAM/disk headroom.
• HA without watchdog. HA detects node crashes but not hangs. Configure a watchdog.
• No tested failover. HA configuration that has never been tested is hope, not engineering.

Key takeaways

• HA groups set per-VM preferences and restrictions.
• HA fails over on crash and service failure, not on hang.
• Test failover quarterly with all four failure modes.
• Plan capacity for N+1 minimum, N+2 for safety.

Knowledge check