← All break/fix scenarios in Proxmox VE
Corosync link between two nodes is flapping every few minutes
Reported symptoms
- ●pvecm status shows node going online and offline repeatedly
- ●HA VMs bounce between nodes
- ●corosync.log shows token timeout and member leaving messages
- ●Cluster briefly loses quorum during each flap
Evidence
- · grep -i "token\|left\|joined" /var/log/corosync/corosync.log
- · mtr -n <peer-node> shows packet loss or high jitter on the corosync link
- · switchport counters show CRC errors or input drops
- · ethtool -S <nic> shows crc, rx_missed_errors, or fifo errors
Diagnosis and resolutionclick to reveal
Root cause
The corosync link is experiencing transient failures, causing corosync to declare the peer lost and rejoin. Each flap causes a brief quorum loss and HA recovery activity. Common causes: - Faulty cable or SFP module - Switch port configured at wrong speed/duplex - MTU mismatch (large frames dropped silently) - NIC buffer overruns under load
Remediation
1. Check the physical layer: - Swap the cable - Try a different switch port - Replace the SFP if in use 2. Check the link layer: `ethtool <nic>` (look for speed/duplex) `ethtool -S <nic> | grep -E 'err|drop|crc'` 3. Check MTU consistency across the path: `ip link show <nic>` (each hop must match) 4. If the issue is buffer-related, tune the NIC: `ethtool -G <nic> rx 4096 tx 4096` 5. If the link is unavoidable flaky, add a second corosync link for redundancy: `pvecm updatecerts --link1 <peer-ip>` and edit `/etc/pve/corosync.conf` to add the second totem link. 6. Restart corosync on one node at a time: `systemctl restart corosync`
Verification
- `pvecm status` shows stable membership for at least 30 minutes - `grep -i "token\|left\|joined" /var/log/corosync/corosync.log` is empty - `mtr -n <peer-node>` shows 0% loss - HA VMs stay on their original nodes - `ethtool -S <nic>` shows no error counter increases
Prevention
- Use two physical corosync links (separate NICs and cables) - Set MTU consistently across all corosync hops - Configure SNMP monitoring on the switch ports for CRC errors - Document the corosync recovery procedure and test it