Network performance: MTU, offload, and tuning for VM workloads

Default Linux network settings are tuned for compatibility, not throughput. For VM workloads on a 10 GbE or faster network, several settings need to change. This lesson covers the practical tuning.

MTU 9000 (jumbo frames)

The default MTU of 1500 is conservative. Every network protocol adds overhead (TCP/IP is ~40 bytes for headers), so the effective payload is ~1460 bytes per packet. At 10 Gbps, that’s 812,000 packets per second per direction — each consuming CPU cycles for interrupt and buffer management.

MTU 9000 reduces this by 6x: 1.35 million packets per second at 10 Gbps becomes 225,000. Lower interrupt rate, fewer buffer copies, better throughput for large transfers.

Configuring end-to-end

Every hop in the path must support MTU 9000:

# 1. Physical NIC on PVE node
ip link set ens4f0 mtu 9000
# Verify
ip link show ens4f0 | grep mtu

# 2. Bridge (does NOT need MTU 9000 — bridges forward frames; the
#    MTU is set on the bridge ports)
ip link set vmbr0 mtu 9000
ip link show vmbr0 | grep mtu

# 3. Inside the VM (Linux guest)
ip link set eth0 mtu 9000
ip route show | grep default
# Verify MTU on the route

And the switch between the PVE node and the storage backend:

! Cisco IOS
interface GigabitEthernet1/0/1
 mtu 9000

! Arista EOS
interface Ethernet1
 mtu 9000

! Linux bridge (used as a managed switch)
ip link set br0 mtu 9000

If any hop doesn’t support MTU 9000, large packets are silently dropped (the IP stack doesn’t fragment), causing connectivity failures with no obvious error message. Always test end-to-end:

# Send a large ping with DF (don't fragment)
ping -c 5 -M do -s 8972 pve-02
# If MTU mismatch, this hangs or fails

# Or use iperf3 (more realistic)
iperf3 -c pve-02 -t 30 -P 4
# Throughput should be near line rate (9.4 Gbps on 10 GbE)

The MTU on each layer

Confusion point: MTU is set at multiple layers and they must agree.

Physical NIC ─── MTU 9000 ─── must match
Bond          ─── MTU 9000 ───
Bridge        ─── MTU 9000 ───
VNet          ─── MTU 9000 ─── inside SDN, used by VM
VM NIC (virtio) ─ MTU 9000 ─── inside the guest
Guest kernel  ─── MTU 9000 ───

If the bridge is MTU 1500 but the VM NIC is MTU 9000, the VM silently drops packets larger than 1500. If the physical NIC is MTU 9000 but the switch is MTU 1500, frames larger than 1500 are dropped at the switch.

When to skip jumbo frames

For networks with mixed devices (some supporting 9000, some not), jumbo frames cause more problems than they solve. The standard solution is to enable jumbo frames only on the storage network (one dedicated VLAN with all-MTU-9000 devices) and keep the management network at MTU 1500.

TCP offload features

Modern NICs offload TCP processing to hardware. Each offload has a trade-off:

TSO (TCP Segmentation Offload)

The NIC splits large TCP segments into MTU-sized packets. Without TSO, the CPU does this in software.

Default: enabled
For VM workloads: keep enabled. TSO saves significant CPU.
Tuning: nothing to do; ethtool reports it as tcp-segmentation-offload.

LRO (Large Receive Offload)

The NIC coalesces multiple TCP segments into one large buffer before delivering to the kernel.

Default: enabled
For VM workloads: keep enabled, but inside VMs, disable it. LRO merges packets in a way that confuses guest TCP RTT measurement. Symptoms: latency spikes, throughput anomalies.

# Inside the VM
ethtool -K eth0 lro off
# Or via sysctl
sysctl net.ipv4.tcp_low_latency=1

For storage workloads (NFS, iSCSI, Ceph), disable LRO on the host’s storage NIC too — it causes similar issues.

GRO (Generic Receive Offload)

The kernel’s modern equivalent of LRO. Should stay enabled.

# Verify
ethtool -k eth0 | grep -E 'gro|lro|tso'

Checksum offload

Hardware computes TCP/UDP checksums. Almost always beneficial; keep enabled. Disable only for packet capture / analysis.

RSS (Receive Side Scaling)

Distributes incoming packets across multiple CPU queues. Critical for multi-core systems. Most modern NICs enable this automatically.

# Verify
ethtool -x eth0 | grep -E 'queue|cpu'
# Should show 4+ queues on a multi-core system

Tuning virtio-net

The default virtio-net driver in PVE 9.x is already well-tuned. For specific workloads:

# Throughput-optimised (default)
qm set 100 --net0 virtio,bridge=vmbr0,mtu=9000
# The "mtu=9000" in the VM config sets the guest\'s MTU automatically

# Latency-optimised
qm set 100 --net0 virtio=...,bridge=vmbr0,mtu=1500
# Smaller MTU = more predictable latency (less time waiting for
# large packets to assemble)

PVE 9.x supports the packed virtio-net mode for lower overhead:

qm set 100 --net0 virtio=...,bridge=vmbr0
# "packed" is enabled by default on supported kernels

For very high throughput (10 GbE+ per VM), consider mtu=9000 on the guest NIC and verify with iperf3.

TCP buffer autotuning

Linux’s TCP buffer autotuning is conservative by default. For high-bandwidth networks, increase the maximums:

# /etc/sysctl.d/10-network-performance.conf
# Increase TCP buffer maxes
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216

# Increase TCP congestion window
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_sack = 1

# Netdev backlog
net.core.netdev_max_backlog = 30000
net.core.netdev_budget = 600

# TCP Fast Open (saves one round-trip on repeated connections)
net.ipv4.tcp_fastopen = 3

# Increase connection tracking table (if using conntrack heavily)
net.netfilter.nf_conntrack_max = 262144

sysctl --system

These settings are safe on any modern Linux. They don’t break anything; they just allow larger buffers when the kernel decides to use them.

Diagnosing network performance

# Live socket stats
ss -tin
# Look at:
#   cwnd (congestion window) — should grow to ~10 packets
#   rtt / rttvar (round-trip time) — should be sub-ms on LAN
#   bytes_sent / bytes_received — throughput
#   retrans — should be near-zero on healthy network

# Per-CPU interrupts (find imbalance)
cat /proc/interrupts | grep eth
# A single CPU handling all interrupts is a bottleneck

# CPU affinity for NIC interrupts
echo &lt;cpu-mask&gt; > /proc/irq/&lt;n&gt;/smp_affinity
# Or use irqbalance (default in PVE)

# iperf3 between two hosts
iperf3 -s    # Server
iperf3 -c &lt;host&gt; -t 30 -P 4
# -P 4: 4 parallel streams (test aggregate throughput)

# Per-VM bandwidth inside the VM (guest-side)
iperf3 -s
# From another VM on the same bridge
iperf3 -c &lt;other-vm&gt;

The diagnostic shape for healthy 10 GbE:

Single-stream iperf3: 9.4+ Gbps
4-stream iperf3: 9.4+ Gbps (saturates the link)
Latency: <100 µs on the same bridge
CPU usage on the receiving host: <50% of one core at 10 Gbps
Retransmits: <1 per million packets

Production considerations

MTU and storage. NFS, iSCSI, NVMe-oF all benefit hugely from MTU 9000. The biggest performance win is on the storage network.
Offload and capture. If you run tcpdump on a host with TSO/LRO enabled, you see fewer packets than the wire actually carried. Disable offloads temporarily for capture, or use a SPAN/mirror port.
Bridge offload. Bridge interfaces inherit the MTU of their ports. If a port is 1500 and the bridge is 9000, frames larger than 1500 are dropped.
Bond MTU. LACP bonds report their MTU as the minimum of their slaves. If one slave is 9000 and another is 1500, the bond is 1500.

Common mistakes

MTU 9000 on storage but MTU 1500 on management. Works, but jumbo frames on the storage side need jumbo frames everywhere in that path — not just on the host’s NIC.
Disabling all offloads. Each offload has a reason. Disabling them all puts the network stack in software mode, which is slow. Disable only specific offloads (LRO for VMs).
iperf3 with default settings. Default iperf3 uses 1 stream and 1 thread. Use -P 4 to test parallel streams, which matches real workloads.
Testing with TCP only. Some workloads use UDP heavily (Ceph replication, monitoring). Test UDP too.

Key takeaways

MTU 9000 end-to-end on the storage network for the biggest performance win.
Disable LRO inside VMs and on storage NICs.
Tune TCP buffers with sysctl.
Diagnose with iperf3 (multi-stream) and ss.