Network troubleshooting: bridging, VLANs, MTU, and routing issues

Network problems are the most common PVE issue and the most misdiagnosed. “It’s the network” is a joke because it’s so often true. This lesson gives you a systematic method for finding the real cause instead of randomly restarting services.

The triage framework

When a network problem is reported, follow this order:

Physical — Is the cable plugged in? Is the link light on?
L2 — Is the bridge up? Is the VLAN correct? Is STP blocking?
L3 — Is the IP configured? Can it ping its gateway?
MTU — Can it transfer large packets?
DNS — Can it resolve names?
Application — Is the service listening on the right port?

Skipping ahead wastes hours. A “VM can’t reach the internet” problem that is actually a missing default route gets fixed by restarting nothing.

L1 — Physical layer

# Check link state
ip link show
ethtool ens4f0 | grep -E 'Link detected|Speed|Duplex'
# Link detected: yes
# Speed: 10000Mb/s
# Duplex: Full

# Check SFP / cable info (if fibre)
ethtool -m ens4f0 | head -20
# Look for temperature, RX power, TX power
# High temperature or low RX power = failing SFP or cable

If Link detected: no, the cable or SFP is bad. Swap and retest.

L2 — Bridge and VLAN

# Check bridge state
ip link show vmbr0
# vmbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000
# If state is not UP, the bridge is down

# List bridge members
bridge link show vmbr0
# Shows which interfaces are in the bridge and their state

# List VLANs
bridge vlan show vmbr0
# Shows which VLANs are configured on which ports

# Check STP state
bridge link show vmbr0 | grep -E 'state|forward'
# Look for "forwarding" state — "blocking" or "discarding" means STP is blocking

# Check for STP topology changes
brctl showstp vmbr0 | grep -E 'state|topology'

Common L2 issues:

Bridge without ports: the bridge has no physical interfaces. Check bridge link show vmbr0 — if no ports listed, traffic only flows between VMs on the same bridge, not to the outside.
Bridge port in blocking state: STP is preventing forwarding. Either disable STP on the port (bridge-stp off in /etc/network/interfaces) or check the STP topology.
VLAN not in bridge-vids: a VM on VLAN 100 won’t reach the gateway if VLAN 100 isn’t in the bridge’s VLAN list.
MTU mismatch: bridge at 1500 but VMs at 9000 — large packets silently dropped.

L3 — IP and routing

# IP addresses on every interface
ip -br addr show
# lo               UNKNOWN        127.0.0.1/8 ::1/128
# ens4f0           UP             192.168.1.10/24
# vmbr0            UP             192.168.1.10/24 fe80::.../64

# Routing table
ip route show
# default via 192.168.1.1 dev vmbr0
# 192.168.1.0/24 dev vmbr0 proto kernel scope link src 192.168.1.10

# Test the gateway
ping -c 3 192.168.1.1
# If this fails, the gateway is unreachable
# If the gateway doesn\'t respond to ping but the host can reach other hosts,
#   the gateway is configured to not respond to ICMP (check the firewall)

# Test a remote host
ping -c 3 8.8.8.8
# If gateway works but 8.8.8.8 doesn't, NAT or upstream routing is broken

Common L3 issues:

Missing default route: the host can reach its subnet but nothing else. Check ip route show for a default via line.
Wrong gateway IP: a typo in /etc/network/interfaces means the host sends return traffic to the wrong place.
Asymmetric routing: packets take one path, return packets another. Connection breaks even though each path individually works.

MTU — the silent killer

# Check MTU on each interface
ip -d link show | grep -E 'mtu|ens|vmbr'

# Test with a large packet (don\'t fragment)
ping -c 3 -M do -s 8972 &lt;destination&gt;
# If this fails with "message too long" or similar, MTU is broken

# Find the smallest MTU in the path
tracepath &lt;destination&gt;
# Shows MTU at each hop

MTU mismatch symptoms:

TCP connections work for small data but hang on large transfers
“No route to host” errors that don’t make sense
TCP works over IPv4 but not IPv6 (different MTU paths)

The fix is always to find the smallest MTU in the path and set every device to match.

DNS

# /etc/resolv.conf
cat /etc/resolv.conf
# nameserver 192.168.1.1
# search example.com

# Test DNS resolution
dig pve-01.cluster.example.com
nslookup pve-01.cluster.example.com 192.168.1.1

# Test against a specific server
dig @8.8.8.8 pve-01.cluster.example.com

# Capture DNS traffic
tcpdump -i any -n port 53

DNS failures:

connection timed out: the resolver IP is unreachable
server can't find <name>: the resolver doesn’t have the record
communications error: the resolver is responding but the response is malformed

Application

# Listening sockets
ss -tlnp
# LISTEN 0 4096 0.0.0.0:8006 0.0.0.0:* users:(("pveproxy",pid=1234,fd=6))

# Test a specific service
curl -v https://localhost:8006/api2/json --insecure

# Capture application traffic
tcpdump -i any -n port 8006 -X

Service failures:

Not listening: ss -tlnp doesn’t show the port. Start the service or check why it crashed.
Listening but unreachable from outside: firewall. Check iptables -L -n or the PVE firewall rules.
Listening and reachable but not responding: the service is hung. Restart it.

Common failure modes

“VM can’t reach the internet”

The most common complaint. Walk through:

# 1. Can the VM reach its gateway?
ssh vm100
ip route show
ping 10.0.0.1   # The gateway

# 2. If gateway unreachable, is the bridge correct?
qm config 100 | grep net
# Look at bridge and VLAN

# 3. Check the host\'s view of the VM
ip neigh show vmbr0 | grep &lt;vm-mac&gt;
# If no entry, the VM isn\'t on the bridge correctly

# 4. Capture traffic on the bridge
tcpdump -i vmbr0 -n host &lt;vm-ip&gt;
# If you see traffic from the VM but no return, the gateway
# (or upstream) is the issue

“Cluster communication broken”

Symptoms: GUI shows offline nodes, VMs can’t migrate, HA doesn’t fire.

# On the affected node
systemctl status corosync
journalctl -u corosync --since '10 minutes ago'

# Check corosync rings
corosync-cfgtool -n
# Look for "ACTIVE" on each ring

# Test connectivity to other nodes
ping pve-02
ping pve-03

# Test the cluster network specifically
iperf3 -c pve-02 -p 5405

“Live migration hangs”

Migration stalls at some percentage. Common causes:

Network bandwidth insufficient
Storage not shared
Memory copy in progress

# Monitor migration progress
watch -n 1 'qm status 100'

# Check the cluster network
# Migration uses corosync network for control + storage network for data

# Check the storage backend
pvesm status
# Verify the storage is shared (accessible from both source and target)

# If migration stalls at 99%, it\'s often a final sync taking too long
# This is usually not a bug — it's a slow final pass

Production considerations

Have tcpdump in muscle memory. When a network problem is mysterious, tcpdump -i any -n shows everything.
Bridge naming consistency. Use predictable names (vmbr0 for management, vmbr1 for storage, vmbr2 for VMs). Avoid “vmbr0 + vmbr10 + vmbr99”.
MTU defaults. Most installs default to MTU 1500. Document your MTU strategy so future admins know what to expect.
Spare NIC cable. Always have a known-good Cat6/Cat6A cable in the rack.

Common mistakes

Restarting services without diagnosis. “Restart pveproxy, see if it helps” wastes time and hides the real problem.
Trusting “it works for me”. The problem might be specific to one VM’s MAC address or one VLAN.
Assuming the firewall. Most network problems are not firewall problems; they’re MTU, VLAN, or routing issues. Check those first.
Looking in the wrong place. A “VM can’t reach the internet” problem might actually be a DNS issue (name doesn’t resolve) or an application issue (port not listening).

Key takeaways

Triage in order: physical, L2, L3, MTU, DNS, application.
Use tcpdump when the answer isn’t obvious.
Document the network topology so future admins can find issues.
MTU mismatch is the silent killer.