MTU validation — end-to-end verification, jumbo on all path, sub-interface alignment

End-to-end MTU validation is the operator’s discipline of verifying that every path in the network delivers the MTU that the applications expect. A path that works for small packets but fails for large packets is a silent failure — the network “looks fine” until the application sends a 100 KB upload, a database backup, or a video stream. The operator’s discipline is to validate the path MTU before deploying the application, not after users complain.

This lesson is the production reference for MTU validation on VyOS 1.5 LTS: the validation script, the jumbo-frame verification, the 802.1Q sub-interface alignment, the tunnel inner MTU validation, and the audit discipline.

The MTU validation script

The operator maintains a script that probes the path MTU to every critical destination:

#!/bin/bash
# validate-mtu.sh — probe the path MTU to every critical destination.
# Usage: validate-mtu.sh <destination-file>

DESTINATIONS="$1"
EXPECTED_MTU=1500

while IFS= read -r dest; do
    [[ -z "$dest" || "$dest" == \#* ]] && continue
    echo "=== Testing $dest ==="
    # Send a 1472-byte payload (1500-byte IP packet) with DF=1
    result=$(ping -M do -s 1472 -c 3 -W 2 "$dest" 2>&1)
    if echo "$result" | grep -q "Frag needed"; then
        # PMTUD worked; extract the bottleneck MTU
        bottleneck=$(echo "$result" | grep -oP 'mtu = \K\d+' | head -1)
        echo "  PMTUD: path MTU = $bottleneck"
        if [[ "$bottleneck" -lt "$EXPECTED_MTU" ]]; then
            echo "  WARNING: path MTU below expected ($EXPECTED_MTU)"
        fi
    elif echo "$result" | grep -q "0 received"; then
        echo "  PMTUD: BLACK HOLE (no ICMP reply)"
        echo "  WARNING: ICMP Fragment Needed may be filtered"
    else
        echo "  PMTUD: path MTU >= $EXPECTED_MTU"
    fi
done < "$DESTINATIONS"

The script reads a list of destinations from a file and probes each one. The output identifies:

• PMTUD works, path MTU known. The path MTU is reported by the bottleneck router.
• PMTUD black hole. No ICMP reply; the path may filter ICMP.
• Path MTU >= expected. No issue.

The operator runs the script weekly or after every topology change:

$ ./validate-mtu.sh destinations.txt
=== Testing 198.51.100.1 ===
  PMTUD: path MTU >= 1500
=== Testing 198.51.100.2 ===
  PMTUD: path MTU = 1450
  WARNING: path MTU below expected (1500)
=== Testing 198.51.100.3 ===
  PMTUD: BLACK HOLE (no ICMP reply)
  WARNING: ICMP Fragment Needed may be filtered

The script flags two issues: 198.51.100.2 has a path MTU of 1450 (a tunnel or partner network); 198.51.100.3 has a PMTUD black hole (ICMP filtered).

Jumbo-frame validation

For data-centre backbones running jumbo frames (MTU 9000), the operator must verify that every interface in the path supports jumbo. The validation:

#!/bin/bash
# validate-jumbo.sh — verify jumbo support on every interface in the path.

# 1. Check the local interface MTU
echo "=== Local interface MTU ==="
for intf in $(ip -o link show | awk -F': ' '{print $2}'); do
    mtu=$(ip link show "$intf" | awk '/mtu/ {print $5}')
    if [[ -n "$mtu" && "$mtu" -gt 1500 ]]; then
        echo "  $intf: MTU $mtu (JUMBO)"
    elif [[ -n "$mtu" ]]; then
        echo "  $intf: MTU $mtu"
    fi
done

# 2. Probe the path to the peer with a 8972-byte packet (9000-byte IP packet)
echo
echo "=== Path MTU probe ==="
result=$(ping -M do -s 8972 -c 3 -W 2 "$1" 2>&1)
if echo "$result" | grep -q "Frag needed"; then
    bottleneck=$(echo "$result" | grep -oP 'mtu = \K\d+' | head -1)
    echo "  Path MTU = $bottleneck"
    if [[ "$bottleneck" -ge 9000 ]]; then
        echo "  JUMBO: path supports MTU >= 9000"
    else
        echo "  WARNING: path MTU below 9000; jumbo not end-to-end"
    fi
elif echo "$result" | grep -q "0 received"; then
    echo "  PMTUD BLACK HOLE: cannot determine path MTU"
else
    echo "  Path MTU >= 9000"
fi

The operator runs the script before deploying a jumbo-enabled workload:

$ ./validate-jumbo.sh 198.51.100.1
=== Local interface MTU ===
  eth0: MTU 9000 (JUMBO)
  eth1: MTU 9000 (JUMBO)
  eth2: MTU 1500
  lo: MTU 65536
=== Path MTU probe ===
  Path MTU = 9000
  JUMBO: path supports MTU >= 9000

The output confirms that the local interfaces (eth0, eth1) are jumbo, and the path to 198.51.100.1 supports 9000-byte packets. eth2 is standard MTU; the operator should not send jumbo traffic on eth2.

802.1Q sub-interface alignment

For VLAN sub-interfaces, the operator must verify that the sub-interface MTU is aligned with the parent interface and with the path MTU.

#!/bin/bash
# validate-vlan-mtu.sh — verify 802.1Q sub-interface MTU alignment.

echo "=== Parent interface MTU ==="
ip link show eth0 | awk '/mtu/ {print "  eth0: MTU", $5}'

echo
echo "=== Sub-interface MTU ==="
for sub in $(ip -o link show | awk -F': ' '{print $2}' | grep -E '^eth0\.'); do
    mtu=$(ip link show "$sub" | awk '/mtu/ {print $5}')
    echo "  $sub: MTU $mtu"
done

echo
echo "=== Probe path MTU on sub-interface ==="
# The operator must test the path with the sub-interface's VLAN tag in place
# Use ping with the -I flag to specify the source interface
result=$(ping -M do -s 1472 -c 3 -W 2 -I eth0.100 "$1" 2>&1)
if echo "$result" | grep -q "Frag needed"; then
    bottleneck=$(echo "$result" | grep -oP 'mtu = \K\d+' | head -1)
    echo "  Path MTU via eth0.100 = $bottleneck"
elif echo "$result" | grep -q "0 received"; then
    echo "  PMTUD BLACK HOLE"
else
    echo "  Path MTU >= 1500"
fi

The output verifies that the sub-interface (eth0.100) has the correct MTU and that the path through the sub-interface supports the expected packet size.

Tunnel inner MTU validation

For tunnels, the operator validates the inner MTU end-to-end:

#!/bin/bash
# validate-tunnel-mtu.sh — validate the tunnel inner MTU end-to-end.

TUNNEL="$1"
DEST="$2"
INNER_MTU="$3"  # e.g., 1468 for WireGuard IPv4, 1420 for IPv6

echo "=== Tunnel interface MTU ==="
mtu=$(ip link show "$TUNNEL" | awk '/mtu/ {print $5}')
echo "  $TUNNEL: MTU $mtu"
echo "  Expected inner MTU: $INNER_MTU"
if [[ "$mtu" != "$INNER_MTU" ]]; then
    echo "  WARNING: tunnel MTU does not match expected inner MTU"
fi

echo
echo "=== Probe path through tunnel ==="
# Use ping with the source interface as the tunnel
result=$(ping -M do -s $((INNER_MTU - 28)) -c 3 -W 2 -I "$TUNNEL" "$DEST" 2>&1)
if echo "$result" | grep -q "Frag needed"; then
    bottleneck=$(echo "$result" | grep -oP 'mtu = \K\d+' | head -1)
    echo "  Tunnel path MTU = $bottleneck"
    if [[ "$bottleneck" -lt "$INNER_MTU" ]]; then
        echo "  WARNING: tunnel path MTU below configured inner MTU"
    fi
elif echo "$result" | grep -q "0 received"; then
    echo "  PMTUD BLACK HOLE inside tunnel"
    echo "  Consider MSS clamping"
else
    echo "  Tunnel path MTU = $INNER_MTU"
fi

The operator runs the script to verify the tunnel:

$ ./validate-tunnel-mtu.sh wg0 10.0.0.1 1468
=== Tunnel interface MTU ===
  wg0: MTU 1468
  Expected inner MTU: 1468
=== Probe path through tunnel ===
  Tunnel path MTU = 1468

The output confirms that the tunnel MTU matches the expected inner MTU and the path through the tunnel supports 1468-byte packets.

The MTU inventory

The operator maintains an MTU inventory:

# MTU inventory — R1
# Last validated: 2026-08-15 13:42 UTC

Interface         MTU    Notes
eth0              9000   Data-centre backbone (jumbo)
eth1              1500   WAN uplink
eth2              1500   LAN access
eth0.100          1500   VLAN 100 (engineering)
eth0.200          1500   VLAN 200 (production)
wg0               1468   WireGuard tunnel to R2 (IPv4 outer)
vti0              1438   IPsec tunnel to R3 (AES-GCM-128, IPv4 outer)
tun0              1426   GRE over IPsec (double overhead)

Path MTU to R2 (via wg0):    1468
Path MTU to R3 (via vti0):   1438
Path MTU to public Internet: 1500 (most), 1450 (some)

The inventory is updated after every interface or tunnel change. The operator audits the inventory quarterly.

Applying the fix on VyOS 1.5

Validation is only half the job. When the probe says the path cannot carry what you are sending, there are exactly three levers on a VyOS 1.5 router, and each one is a different tree.

Interface MTU — change what this router puts on the wire:

configure
set interfaces ethernet eth0 mtu 9000
set interfaces ethernet eth0 vif 100 mtu 1500
commit
save

A vif carries its own mtu independently of its parent, which is why sub-interface drift is possible at all. The parent must be large enough for the sub-interface plus the 4-byte 802.1Q tag; a vif MTU above the parent’s is the misalignment this lesson’s third script exists to catch.

MSS clamping — leave the MTU alone and shrink what TCP asks for. In VyOS 1.4 and 1.5 this is a property of the interface, not a firewall rule:

configure
set interfaces ethernet eth3 ip adjust-mss 1360
set interfaces ethernet eth3 ipv6 adjust-mss 1340
commit
save

The documentation describes ip adjust-mss as configuring “the MSS advertised in outgoing TCP SYN packets on the specified interface”. If you have seen set firewall name ... tcp-mss or a per-interface firewall in name ... binding used for this, that is the pre-1.4 arrangement: the tcp-mss leaf and the interface firewall bindings are both gone, and neither will commit on 1.5.

Neither — accept the path MTU and let PMTUD do its job. This is the correct answer whenever ICMP is not filtered, and it is why the validation script distinguishes “PMTUD works” from “PMTUD black hole”: only the black-hole case forces you onto one of the first two levers.

Production failure modes

The MTU validation failure modes the operator encounters:

• Validation script does not cover all paths. A new path is added (a new partner, a new cloud provider) and the validation script is not updated. Result: an unvalidated path becomes a silent failure source. Fix: update the script after every topology change.
• Jumbo on source but not on path. The source’s interface is MTU 9000 but the path’s intermediate router is MTU 1500. Result: large packets are dropped at the bottleneck. Fix: validate the path end-to-end; either align the MTU or use PMTUD/MSS clamping.
• Sub-interface MTU mismatch. The parent interface is MTU 9000 but the sub-interface is MTU 1500. Result: traffic on the VLAN cannot exceed 1500 bytes. Fix: align the sub-interface MTU with the parent.
• Tunnel inner MTU not validated. The tunnel is configured but the inner MTU is the default; the operator never tested end-to-end. Result: traffic over the tunnel fails at the wire MTU. Fix: validate the tunnel inner MTU end-to-end.
• Validation script reports “all good” but users complain. The validation script uses ICMP, but the application uses TCP. The application may fail even when ICMP succeeds. Fix: validate with TCP-based tools (iperf3 with large segments, scp with large files).

Rollback

MTU validation changes are about audit, not rollback. The discipline:

• Save validation outputs to a file or ticket.
• Update the MTU inventory after every validated change.
• Apply fixes (MSS clamping, MTU change, firewall rule) and re-validate.

Production discipline

Cross-course references

• Part LI-01 (LI-VyOS-MTU / MTU basics) covers the canonical MTU values.
• Part LI-02 (LI-VyOS-MTU / tunnel overhead) covers the overhead calculations for tunnels.
• Part LI-03 (LI-VyOS-MTU / PMTUD) covers Path MTU Discovery.
• Part LI-04 (LI-VyOS-MTU / MSS clamping) covers the canonical fix when PMTUD fails.
• Part LI-05 (LI-VyOS-MTU / MTU and fragmentation troubleshoot) covers the operational diagnostic.
• Part XLIX (XLIX-VyOS-Monitoring) covers the monitoring integration that consumes the validation results.

Quiz