Network failure scenarios and debugging

Why this matters in production

Network incidents in production clusters are nearly always “the network is fine for small traffic but breaks under load” or “a configuration change broke something nobody noticed.” A systematic approach lets you triage in minutes rather than hours.

The debugging checklist

When the network misbehaves, walk this checklist in order:

flowchart LR
  A[Layer 1: physical link] --> B[Layer 2: switch / VLAN]
  B --> C[Layer 3: IP / route]
  C --> D[Layer 4: firewall]
  D --> E[Application: DNS / NTP]

1. Physical

for nic in $(ls /sys/class/net/ | grep -v lo); do echo "=== $nic ==="; ethtool $nic 2>/dev/null | grep -E 'Speed|Link detected|Auto|Negotiation'; done

2. Switch and VLAN

cat /proc/net/bonding/bond0

bridge vlan show

3. IP and route

ip -br addr && ip route

TARGET_IP=192.0.2.30
mtr -rwc 10 "$TARGET_IP"

4. Firewall

iptables -L -n -v && iptables -t nat -L -n -v

5. Application

HOSTNAME=pve-02.example.com
dig +short "$HOSTNAME"
chronyc tracking

Common failure modes

Bond in active-backup, one NIC down

cat /proc/net/bonding/bond0
...
Slave Interface: ens4
MII Status: up
...
Slave Interface: ens5
MII Status: down
...

The bond has failed over to the surviving slave. The cluster continues to work, but with half the available bandwidth. Action: replace the failed NIC or its cable. Confirm MII Status returns to “up” after the fix.

LACP not negotiated

cat /proc/net/bonding/bond0
...
802.3ad info
LACP rate: fast
...
Slave Interface: ens4
...
Actor Churn State: churned
Partner Churn State: churned

The bond is “up” but flapping. The switch side is not configured for LACP (or has a different aggregation group). Action: coordinate with the network team; verify the switch port is part of the correct LACP bundle.

MTU mismatch

Symptoms:

• ping -M do -s 8972 fails with “message too long” or hangs.
• TCP connections to certain ports succeed; others hang.
• Live migration appears to start, then stalls.

Action: set the MTU to a known value (1500 or 9000) on every hop: host NIC, bond, bridge, VM guest, switch port, storage target. Verify with ping -M do.

Firewall broke Corosync

Symptom: cluster loses quorum unexpectedly; nodes cannot ping each other on UDP 5404–5405 but TCP/22 works.

Action: check /etc/pve/firewall/cluster.fw and the node firewall. The fix is to allow the required ports (5404–5405 UDP, 2224 TCP) between cluster nodes. If you cannot reach the GUI to fix it, use OOB management.

corosync-cfgtool -s
pvecm status
journalctl -u corosync --since '15 minutes ago' | tail -30

# is /etc/pve writable right now?
touch /etc/pve/.write-test 2>&1 && rm -f /etc/pve/.write-test

SDN VNet misconfigured

Symptoms: VMs on a VNet can talk to each other but not to the network; or cannot talk to each other at all.

Action: check pvesh get /cluster/sdn/vnets --output-format yaml. Verify the zone, tag, and subnet configuration. Re-apply the SDN configuration with pvesh set /cluster/sdn after reading pvesh get /cluster/sdn/dry-run to see what the apply would change. If a VNet’s underlying bridge is missing on a node, the node may not be in the zone’s node list.

A break/fix exercise

Production considerations

Common mistakes

• Skipping the layer-by-layer checklist. Symptoms can mislead; layers are objective.
• Trusting “it worked yesterday” — most incidents follow a change, often a network change the operator did not connect to the symptom.
• Resetting the firewall to ACCEPT all during a panic. Document the original rules first.
• Forgetting that bond changes sometimes require a brief interface reset.

Key takeaways

• Walk the OSI layers from physical to application.
• MTU mismatches are silent; verify with ping -M do.
• Quorum loss makes /etc/pve read-only, so a firewall rule that broke corosync cannot be reverted through cluster.fw. Stop pve-firewall to restore connectivity, then fix the file once quorum returns. pvecm expected 1 is not the answer.
• Always have OOB access when debugging network incidents.

Knowledge check