← All break/fix scenarios in Proxmox VE
HA service restarts repeatedly with "service timeout" errors
Reported symptoms
- ●ha-manager status shows a VM with state "fence", "started", "started", "fence" in quick succession
- ●Log shows "service vm:100:timeout" entries
- ●The VM is alive and reachable from outside the cluster
- ●The fence agent (if used) is invoked repeatedly
Evidence
- · grep "service.*timeout" /var/log/pve-ha-manager.log
- · ha-manager config shows the watchdog type (self-fence or external)
- · ps aux | grep pve-ha-crm and pve-ha-lrm on the affected node
- · The VM responds to ping but the LRM reports it down
Diagnosis and resolutionclick to reveal
Root cause
The HA Local Resource Manager (LRM) cannot confirm the VM is running, even though it is. This usually means: - The HA watchdog is misconfigured (self-fence expects a hardware watchdog; without one, the LRM times out) - The VM is not responding to QEMU guest agent heartbeats - Network issue between the LRM and the VM's heartbeat target
Remediation
1. Check the HA configuration: `ha-manager config` Look for the "watchdog" mode (soft vs hardware) 2. Verify the VM has QEMU guest agent: `qm config <vmid> | grep agent` `qm guest exec <vmid> ping -c1 127.0.0.1` If this fails, install/enable the guest agent inside the VM 3. If using software watchdog: The HA manager must see the lrm heartbeat within 30 seconds. Check: `systemctl status pve-ha-lrm` `journalctl -u pve-ha-lrm -e` 4. If the VM genuinely is healthy but LRM reports down: - Try `ha-manager set vm:<vmid> --state started` to manually reset state - Investigate why the heartbeat is failing (network, agent) 5. If self-fencing is happening too aggressively, switch to non-fencing mode (crm-only) and investigate separately.
Verification
- HA status is stable for at least 30 minutes - The VM stays in "started" state on its current node - No new "service timeout" entries in the log - The guest agent responds (if configured)
Prevention
- Enable QEMU guest agent on every HA-managed VM - Use hardware watchdog (IPMI / iLO / iDRAC) for self-fencing - Test HA failover in a controlled environment first - Document the HA behavior expectations