IPsec Tunnel Routes but MTU Drops Packets · Break/Fix · VyOS

Reported symptoms

br-lon is a VyOS router at a new London branch. It reaches head office over a route-based IPsec tunnel — a VTI called vti0, addressed out of a /30, with eBGP running across it. The branch WAN is a PPPoE service and the branch sits behind the carrier’s NAT.

The tunnel was commissioned three weeks ago. The commissioning runbook was followed and every check passed. Users were cut over last Monday.

Since Monday there are five tickets from three teams and none of them names the tunnel. The branch says the intranet is down, but the site loads its page shell and then spins forever on one large asset, and ping and DNS to head office are both fine. SSH from the branch connects and authenticates instantly and then freezes the moment a command produces more than a screenful of output. The nightly file sync to the branch NAS has not completed once, stalling at a different byte offset every night, which had the storage team examining the array for three days. Email works, but attachments over about a megabyte sit in the outbox until they time out.

The network team’s own checks all pass. show vpn ipsec sa shows the SA up. BGP over the tunnel is Established with every expected prefix. The helpdesk has pinged across the tunnel at several sizes with no loss. Nothing is logged anywhere: no firewall drops, no interface errors, no counter climbing on either router.

Evidence provided

Read-only / Safemtu 1500 — and every counter is clean

vyos@br-lon:~$ show interfaces vti vti0

vti0: <NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
  link/none
  inet 10.10.10.2/30 scope global vti0
  RX:  bytes    packets  errors  dropped  overrun  mcast
  418822901     3104882       0        0        0      0
  TX:  bytes    packets  errors  dropped  overrun  mcast
  502117440     3298104       0        0        0      0

Illustrative output

Read-only / Safethe underlay is 1492, not 1500

vyos@br-lon:~$ ip link show pppoe0

7: pppoe0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1492 qdisc fq_codel state UNKNOWN mode DEFAULT group default qlen 3
  link/ppp

Illustrative output

Read-only / SafeESP inside UDP 4500 — NAT-T is active

vyos@br-lon:~$ sudo tcpdump -ni pppoe0 udp port 4500 -c 4

13:41:02.118 IP 100.64.19.7.4500 > 203.0.113.10.4500: UDP-encap: ESP(spi=0xc4a19f3b,seq=0x91a2), length 1464
13:41:02.119 IP 203.0.113.10.4500 > 100.64.19.7.4500: UDP-encap: ESP(spi=0x7fd20c81,seq=0x8ce1), length 148
13:41:02.204 IP 100.64.19.7.4500 > 203.0.113.10.4500: UDP-encap: ESP(spi=0xc4a19f3b,seq=0x91a3), length 116
13:41:02.207 IP 203.0.113.10.4500 > 100.64.19.7.4500: UDP-encap: ESP(spi=0x7fd20c81,seq=0x8ce2), length 1464

Illustrative output

The probe ladder, run from a branch workstation to a head-office host across the tunnel. -M do sets the DF bit; -s is the ICMP payload, so the IP packet on the wire is 28 bytes larger.

Read-only / Safea 1430-byte packet gets through

user@lon-ws01:~$ ping -M do -s 1402 -c 3 10.20.0.50

PING 10.20.0.50 (10.20.0.50) 1402(1430) bytes of data.
1410 bytes from 10.20.0.50: icmp_seq=1 ttl=62 time=18.4 ms
1410 bytes from 10.20.0.50: icmp_seq=2 ttl=62 time=18.1 ms
1410 bytes from 10.20.0.50: icmp_seq=3 ttl=62 time=18.6 ms

--- 10.20.0.50 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss

Illustrative output

Read-only / Safeone byte more and it vanishes — with no error

user@lon-ws01:~$ ping -M do -s 1403 -c 3 10.20.0.50

PING 10.20.0.50 (10.20.0.50) 1403(1431) bytes of data.

--- 10.20.0.50 ping statistics ---
3 packets transmitted, 0 received, 100% packet loss

Illustrative output

Read-only / Saferequests enter the tunnel; nothing comes back, and no fragmentation-needed either

vyos@br-lon:~$ sudo tcpdump -ni vti0 icmp -c 10

13:44:51.902 IP 10.0.10.31 > 10.20.0.50: ICMP echo request, id 4711, seq 1, length 1411
13:44:52.906 IP 10.0.10.31 > 10.20.0.50: ICMP echo request, id 4711, seq 2, length 1411
13:44:53.910 IP 10.0.10.31 > 10.20.0.50: ICMP echo request, id 4711, seq 3, length 1411

Illustrative output

Read-only / Safe1460 in both directions — nothing is clamping

vyos@br-lon:~$ sudo tcpdump -ni vti0 -S 'tcp[tcpflags] & tcp-syn != 0' -c 2

13:46:11.004 IP 10.0.10.31.51022 > 10.20.0.50.443: Flags [S], seq 2914418822, win 64240, options [mss 1460,sackOK,TS val 88214 ecr 0,nop,wscale 7], length 0
13:46:11.023 IP 10.20.0.50.443 > 10.0.10.31.51022: Flags [S.], seq 118440290, ack 2914418823, win 65160, options [mss 1460,sackOK,TS val 3391 ecr 88214,nop,wscale 7], length 0

Illustrative output

Work the evidence before reading on

Five tickets, three teams. Start by looking for what the working cases have in common rather than what the broken ones do.

List everything that works: ICMP echo at the helpdesk’s default size, DNS, the TCP handshake, HTTP request headers, IKE, BGP keepalives, email under a megabyte. What single property do all of them share?
The ladder passes at -s 1402 and fails at -s 1403. -s is the ICMP payload. What is the size of the IP packet at each of those two points, and what does the transition tell you that a loss percentage alone would not?
The failing probe produces no local error and no ICMP reply of any kind. What would you have expected to come back, and what does its absence rule out about how the sender can recover?
vti0 reports MTU 1500, pppoe0 reports 1492, and the capture shows ESP inside UDP 4500. Add those three facts up.

Then the question the incident turns on: every check on the commissioning runbook passed three weeks ago and the tunnel has been broken since the day it was built. What do ping, show vpn ipsec sa, show ip route and an Established BGP session have in common that makes them unable to detect this?

Root cause

1. Everything that works is small

The working list is not a set of unrelated successes. A default ping sends a 56-byte payload. A DNS query and response fit in a few hundred bytes. A TCP handshake carries no data at all. An HTTP request header is under a kilobyte. IKE messages and BGP keepalives are small by design.

The failing list is the complement: a large JS bundle, a terminal flooding output, a file sync, a mail attachment. Every one of them is an application that fills a TCP segment as soon as it has data to move.

Five tickets, one behaviour: this path carries small packets perfectly and drops full-size ones without a trace.

2. The arithmetic, and the measurement that confirms it

The branch WAN is PPPoE, which costs 8 bytes and leaves an underlay of 1492 rather than 1500. The tunnel is route-based IPsec in tunnel mode with an AES-GCM-128 proposal, and every one of those bytes is countable rather than approximate. NAT-T is active — the capture shows ESP inside UDP 4500 — which adds the 8-byte UDP header on top.

underlay path MTU (PPPoE)                       1492
  less outer IPv4 header                         -20
  less ESP header (SPI + sequence number)         -8
  less explicit IV / nonce for GCM                -8
  less ESP trailer (pad length + next header)     -2
  less ICV                                       -16
  less NAT-T UDP header                           -8
                                                -----
maximum inner packet                            1430

The ESP trailer also pads the ciphertext up to a 4-byte boundary, which costs 0 to 3 bytes more depending on the length of the inner packet. 1430 happens to need none of it, which is why the edge lands exactly there.

The ladder found the same edge independently: -s 1402 is a 1430-byte IP packet and gets through; -s 1403 is 1431 and does not. The calculation and the measurement agree, which is the point at which you can say you understand the path rather than having found a number that happens to work.

vti0 is configured to accept 1500.

3. A tunnel that accepts what it cannot deliver

Because the interface claims 1500, the router takes a full-size packet into the tunnel and only discovers the problem after encapsulation, when the resulting outer packet is too large for the 1492-byte underlay. With DF set — which TCP sets by default — it is dropped rather than fragmented.

Nothing counts that drop in a place anyone looked. vti0 shows no errors and no drops, which is why the network team’s checks stayed green while the branch was unusable.

4. Path MTU Discovery is not rescuing this path

PMTUD exists precisely for this: the sender should receive an ICMP fragmentation-needed carrying the correct MTU, cache it, and re-send smaller. The capture on vti0 during the failing probe shows the requests going out and nothing at all coming back.

Whether the message was never generated or was dropped somewhere on the way is a second question and not one you need to answer to fix this. The operational conclusion is the one the course states directly: configure the tunnel MTU explicitly and never build a design that depends on PMTUD working across an administrative boundary. A mechanism you cannot observe is not a mechanism you can rely on.

5. Why the acceptance test passed

ping at its default size, show vpn ipsec sa, show ip route and a BGP session are all small-packet checks. Every one of them passes on a tunnel that cannot carry a single full-size packet.

The tunnel did not break at cut-over. It has never worked, and it was signed off by a runbook that was structurally incapable of noticing.

Under the hood

The reason the MTU alone does not settle this, and the reason MSS clamping is a separate control, comes down to who knows what.

A TCP endpoint advertises its MSS in the SYN, and it computes that value from its own interface: 1500 minus 20 bytes of IP header and 20 of TCP header gives the 1460 in the capture. That host has no knowledge of a tunnel two hops away and no mechanism for acquiring it other than PMTUD. Both endpoints therefore agree on 1460, both send 1500-byte packets, and both are wrong.

MSS clamping rewrites the option as the SYN passes through the router, so both ends settle on a value that fits the tunnel. It is a control-plane fix applied once per connection, and it costs nothing per packet afterwards.

The two controls cover different ground and neither substitutes for the other. The interface MTU protects everything, including UDP bulk transfer and any protocol that never negotiates anything — and it restores the router’s ability to reject an oversized packet at the entrance rather than losing it after encapsulation. The clamp covers TCP specifically and removes TCP’s dependence on PMTUD signalling that this particular path has already been observed not to deliver.

One more detail worth carrying, because it turned up in this incident as a false negative. ping -s sets the ICMP payload, not the packet size: the IP packet is 28 bytes larger, being 20 of IP header and 8 of ICMP header. The helpdesk reported “ping works at 1400”, which was a 1428-byte packet — two bytes under the real ceiling of 1430, and therefore evidence of nothing. It is the closest any check came to finding this, and it still passed.

A commissioning runbook that cannot fail

Three weeks of a branch on a tunnel that has never carried a full-size packet, signed off by a checklist where every item passed.

The failure is not that the checks were wrong. show vpn ipsec sa correctly reported an established SA and BGP was correctly Established. The failure is that the checklist contained no item that a broken MTU could fail, so its passing carried no information about the property that mattered.

The general form is worth recognising because it is common: a test suite that only exercises the control plane will certify a data plane that does not work. For a tunnel, the missing item is one line — a DF-bit probe at the intended MTU, run in both directions — and it is the only item on the list that this fault could not have survived.

Resolution

Measure the ceiling before changing anything. Run the DF probe ladder from a host on each side and record the largest payload that returns and the first that does not. Two numbers one byte apart are worth far more than a loss percentage, and the whole procedure is read-only.
Check the arithmetic against the measurement. Underlay MTU, less the ESP overhead for the negotiated proposal, less the NAT-T UDP header if the capture shows UDP 4500. Agreement between calculation and probe is the evidence that you have found the mechanism rather than a coincidence.
Set the VTI MTU below the measured edge, not at it: set interfaces vti vti0 mtu 1400. The ESP trailer pads the ciphertext to a 4-byte boundary, so the true edge moves by up to three bytes with the length of the inner packet. An MTU set exactly at the measured maximum produces intermittent failure, which is materially harder to diagnose than consistent failure.
Clamp TCP on the tunnel interface: set interfaces vti vti0 ip adjust-mss 1360, which is the 1400-byte MTU less 20 bytes of IP and 20 of TCP header. Add ipv6 adjust-mss 1340 if the tunnel carries IPv6, where the header is 40 bytes rather than 20.
Commit with commit-confirm and confirm only after the probe ladder passes. The change touches the interface that carries your own management path to the branch if you are working through the tunnel.
Apply the same pair at head office. An MTU set on one end fixes one direction; the result is a half-working tunnel that presents as a new and different fault and costs another investigation.
Do not ask the ISP to raise the WAN MTU. 1492 is what PPPoE gives you on this service and it is not a misconfiguration to be corrected.
Do not resolve this by disabling DF or forcing fragmentation. It converts a clean drop into a fragmented path that is slower, that reassembles under load at the far end, and that conceals the same misconfiguration until the next time somebody changes the underlay.
If no change window is available, hold deliberately: route the nightly file sync back over the path it used before cut-over, so the largest workload stops failing while the tunnel keeps carrying the small traffic it already carries correctly. Record the hold with a named owner and an end time.
Update the commissioning runbook before closing the ticket, while the reason is fresh. The tunnel passed the old one.

Verification

The ladder passes at the new MTU and fails one byte above it. Both results are required: a probe that only ever succeeds has not been shown to be measuring anything.
The ladder passes in both directions. Run it from head office toward the branch as well; the two directions have independent MTU settings, so a one-sided fix passes half the tests.
The clamp is visible on the wire. Capture SYNs on vti0 and require mss 1360 on both the SYN and the SYN-ACK, where the evidence previously showed 1460.
The applications recover. The intranet asset that used to hang loads, an SSH session survives a command that floods the terminal, and the nightly sync completes rather than stalling. These are the only checks that are independent of the router's own view, which is exactly why they belong here.
Throughput is sane. Run iperf3 across the tunnel and compare it against the WAN rate. A tunnel that passes ping at every size and still delivers a fraction of the line rate has a second problem in front of it.
The fix survives a rekey. Re-run the probe after the next IKE rekey so that the verification covers a fresh SA rather than the one that happened to be up when you made the change.
The commissioning runbook now contains a check this fault would have failed. Run the updated runbook against the repaired tunnel and confirm the new item passes for the right reason.

Prevention

Never accept a tunnel on a small-packet test. Ping, an established SA, a populated routing table and an Established BGP session all pass on a tunnel that cannot carry one full-size packet, and that combination signed this one off.
Treat an unset tunnel MTU as a build defect rather than a default. The kernel’s 1500 is a guess about an interface that is not Ethernet, and it is wrong for every encapsulation the course covers.
Record the underlay MTU as a per-site fact. This branch is 1492 because it is PPPoE; another site will be 1500; a site behind a second layer of encapsulation will be lower again. The 1400 that fits all three is a margin, not a calculation, and the margin needs to be a deliberate decision.
Configure MSS clamping on every tunnel carrying TCP, on both ends, and record the value next to the MTU so the two stay consistent through the next rebuild.
Do not depend on Path MTU Discovery across an administrative boundary. The ICMP it needs is filtered often enough that a design requiring it works at some sites and not others, which is the worst available outcome.
Add a synthetic full-size DF probe per tunnel to monitoring. One probe a minute would have raised this three weeks before the users did, and it is the same command the ladder already uses.

What if the branch had a plain 1500-byte underlay and no NAT?

The tunnel would still be broken, which is the part worth internalising.

Without PPPoE the underlay is 1500, and without NAT there is no UDP header, so the same ESP arithmetic less the 8 bytes of UDP leaves a ceiling of 1446. A VTI at 1500 is still 54 bytes too large, and the same five tickets arrive with the same shape. The probe ladder would have found its edge at a different byte, and everything else about the incident would be identical.

That matters because the natural reading of this evidence is “PPPoE and carrier NAT broke our tunnel”, and it is wrong in a way that produces the wrong prevention. PPPoE and NAT-T moved the number. The fault is an interface configured to accept more than it can deliver, and that fault exists on a 1500-byte underlay with no NAT anywhere in sight.

The version of this that is genuinely harder is a site where the underlay MTU changes without anybody being told — a carrier inserting an extra layer of encapsulation during a network migration, for instance. A tunnel MTU set with no margin works until that day; a tunnel MTU set at 1400 with a clamp to match absorbs it silently. That is the argument for the margin, and it is worth more than the handful of bytes of efficiency it costs.

Why this matters

A branch office spent three weeks on a link that carried every check and no work, while three teams investigated a web application, a storage array and an email server.

The transferable idea is that a path has a size, and almost nothing you routinely run tells you what it is. Reachability tests answer “can a small packet get there”, and operators reach for them because they are fast and because they are usually enough. The moment an encapsulation is involved they stop being enough, and they stop being enough silently — the check does not become unreliable, it becomes irrelevant while continuing to pass.

The second idea is that defaults have opinions. A VyOS VTI is a kernel XFRM interface, and the kernel initialises one at ETH_DATA_LEN — 1500 — because it has to initialise it at something, not because 1500 is right for a tunnel. It is in fact wrong for every tunnel type in the course. The number that a build inherits without a decision is the number that will be wrong, and the interfaces where nobody made a decision are where you should look first.

Cross-course references

The Linux course’s XXII-Linux-NetTroubleshoot covers the host side of this: what a DF probe measures, how the kernel caches a discovered path MTU, and why a stale cached value can outlive the fix.
The Observability course’s XCVIII-Observability-Troubleshooting covers turning the probe ladder into the synthetic check this tunnel needed from the day it was built.
The Ansible course’s XLII-Ansible-BeyondLinux covers the fleet question: auditing every tunnel you own for an unset MTU and a missing clamp, once you know they are the kind of thing a build silently omits.

Reported symptoms

Evidence

Root cause

Remediation

Verification

Prevention

Reported symptoms

Evidence provided

Work the evidence before reading on

Root cause

1. Everything that works is small

2. The arithmetic, and the measurement that confirms it

3. A tunnel that accepts what it cannot deliver

4. Path MTU Discovery is not rescuing this path

5. Why the acceptance test passed

Resolution

Verification

Prevention

Cross-course references