MSS clamping — ip adjust-mss, MSS = MTU - 40, clamp-mss-to-pmtu, and which interface to clamp on

MSS clamping is the operator’s fallback when Path MTU Discovery (PMTUD) cannot function. MSS is the TCP-layer concept of “the largest segment I can receive in a single TCP packet”; MSS is negotiated in the TCP SYN handshake via a TCP option. MSS clamping rewrites the MSS option to fit a known path MTU, so the sender never sends a TCP segment larger than the path allows — even if PMTUD is broken.

This lesson is the production reference for MSS clamping on VyOS 1.5 LTS: the MSS calculation, the configuration node that actually does it, which interface to attach it to, and how to prove it worked.

What MSS is

The TCP MSS (Maximum Segment Size) is the largest TCP payload (data) that the receiver will accept in a single TCP segment. It does NOT include the TCP header (20 bytes minimum) or the IP header (20 bytes IPv4, 40 bytes IPv6). The MSS is negotiated in the TCP three-way handshake:

sequenceDiagram
  participant Client
  participant Server
  participant Router as VyOS router<br/>(MSS clamp to 1380)

  Client->>Router: SYN, MSS=1460
  Note over Router: Clamp MSS to 1380
  Router->>Server: SYN, MSS=1380
  Server->>Router: SYN-ACK, MSS=1380
  Router->>Client: SYN-ACK, MSS=1380
  Note over Client,Server: Both sides use MSS=1380<br/>(1380 + 40 TCP/IP = 1420 fits path)

Both sides advertise their MSS in the SYN; both sides use the smaller of the two. The resulting MSS governs all subsequent TCP segments on the connection.

The relationship between MSS and MTU:

• IPv4 TCP: MSS = MTU - 40 (20 IP + 20 TCP)
• IPv6 TCP: MSS = MTU - 60 (40 IPv6 + 20 TCP)

For a 1500-byte Ethernet MTU:

• IPv4 MSS = 1460 (1500 - 40)
• IPv6 MSS = 1440 (1500 - 60)

For a tunnel with inner MTU 1420 (WireGuard IPv4 outer):

• IPv4 MSS = 1380 (1420 - 40)
• IPv6 MSS = 1360 (1420 - 60)

The discipline: the operator calculates MSS from the inner MTU, not the wire MTU.

Why MSS clamping is necessary

PMTUD relies on ICMP Fragment Needed messages to inform the sender of the path MTU. When ICMP is filtered (by an ISP, by a firewall, by a tunnel), PMTUD fails. The sender sends large TCP segments; a router in the path drops them silently; the receiver never gets the data; the TCP connection stalls.

flowchart LR
  S["Source<br/>(sends 1460-byte TCP segments)"] --> R1["R1<br/>forwards"]
  R1 --> R2["R2<br/>(tunnel entry, inner MTU 1420)<br/>DROPS 1460-byte segments"]
  R2 --> D["Destination<br/>(never receives)"]

The TCP sender waits for an ACK that never arrives; the TCP retransmission timer fires; the sender retransmits the same 1460-byte segment; the segment is dropped again. The TCP connection eventually times out.

The fix: ensure the TCP sender sends segments that fit the path MTU. The way to do this without PMTUD is MSS clamping — rewrite the MSS in the SYN to the value that fits the path.

VyOS configuration

On VyOS 1.4 and 1.5, MSS clamping is a property of an interface, not an action on a firewall rule. It lives under the interface’s ip and ipv6 sub-trees:

configure
set interfaces vti vti0 ip adjust-mss 1380
set interfaces vti vti0 ipv6 adjust-mss 1360
commit
save

Every interface type that carries the common ip sub-tree has the node — ethernet, a vif under one, bonding, tunnel, vti, wireguard — so the clamp goes on whichever interface is the constrained hop. For a tunnel that is the tunnel interface, not the LAN interface behind it.

Three properties of that node decide whether it does what you intended:

1. ip and ipv6 are separate nodes carrying separate values. They are not aliases and one does not imply the other. Clamping v4 and forgetting v6 produces a dual-stack router where IPv4 transfers work and IPv6 transfers hang, which is a much harder ticket to read than “everything is broken”.
2. The value is either a number or the literal clamp-mss-to-pmtu. The number is the MSS you calculated: inner MTU minus 40 for IPv4, minus 60 for IPv6. Use a number whenever you know the overhead and want it stable.
3. It rewrites the MSS option in TCP SYN packets leaving that interface. It does not touch established connections, it does not touch non-SYN packets, and it does not touch anything that is not TCP.

There is no tcp-mss leaf on a firewall rule and no per-interface firewall in name ... binding on 1.4 or 1.5 — both are pre-1.4 constructs, and the interface firewall bindings were replaced by set firewall ipv4 forward filter with action jump. If you find MSS clamping written as a firewall rule in an old config archive, the 1.2/1.3 form was set firewall options interface <name> adjust-mss; 1.4 moved it onto the interface and 1.5 kept it there.

The operator confirms the committed value from configuration mode:

show interfaces vti vti0

When to use MSS clamping

The MSS clamping decision tree:

flowchart TD
  S["PMTUD fails or is unreliable"]
  S --> Q1{Cause?}
  Q1 -->|ISP filters ICMP| M1["Clamp on the WAN interface"]
  Q1 -->|Tunnel inner MTU| M2["MSS = inner MTU - 40 (IPv4)"]
  Q1 -->|Partner network filter| M3["Clamp on the boundary interface"]
  Q1 -->|IPv6 minimum violation| M4["MSS = max(1280, path MTU) - 60"]
  Q1 -->|Unknown| M5["Clamp + tcp_mtu_probing"]

The five scenarios:

1. ISP filters ICMP. The most common scenario. The operator cannot fix the ISP; MSS clamping is the local fallback. Configure on the WAN interface.

2. Tunnel inner MTU. The tunnel has a lower inner MTU than the wire MTU. Configure ip adjust-mss on the tunnel interface itself, calculated from the tunnel’s MTU rather than the wire MTU.

3. Partner network filters ICMP. A business partner’s network filters ICMP; the operator cannot fix the partner. Clamp on the interface facing the partner.

4. IPv6 minimum violation. The path MTU is below 1280 (rare; only on radio links or constrained segments). IPv6 requires 1280 as a floor, so the link itself has to fragment below that; clamp at 1280 - 60 = 1220 and treat the link as broken until it can carry 1280.

5. Unknown / intermittent. Some PMTUD failures are intermittent (e.g., one ISP link has ICMP filtered; the other does not). The operator clamps and enables tcp_mtu_probing on the endpoints for the cases where the clamp is too aggressive.

MSS validation

Configuration mode tells you what you asked for; only a capture tells you what left the router. The operator validates by inspecting SYN packets on the wire:

$ sudo tcpdump -i eth0 -nn -S 'tcp[tcpflags] & tcp-syn != 0'

The output shows the SYN packets with their MSS option. With the clamp in place on the outgoing side:

12:34:56.789 IP 10.0.0.1.54321 > 198.51.100.1.443: Flags [S], seq 12345, win 65535, options [mss 1380,nop,wscale 7], length 0

The mss 1380 confirms the rewrite happened. On the interface the SYN arrived on, the original value is still there:

12:34:56.789 IP 10.0.0.1.54321 > 198.51.100.1.443: Flags [S], seq 12345, win 65535, options [mss 1460,nop,wscale 7], length 0

1460 is what a host on a 1500-byte Ethernet MTU advertises by default.

Capture on both interfaces, in the same window, for the same connection:

# LAN side — the SYN as it arrived, before the clamp
$ sudo tcpdump -i eth0 -nn -S 'tcp[tcpflags] & tcp-syn != 0'
# MSS=1460

# Tunnel side — the same SYN as it leaves, after the clamp
$ sudo tcpdump -i vti0 -nn -S 'tcp[tcpflags] & tcp-syn != 0'
# MSS=1380

Two readings, not one. If the tunnel side still shows 1460, the clamp is on an interface this traffic does not leave through. And read the SYN-ACK in the same capture: if the server’s advertisement is still 1460 on its way to your client, you have clamped one direction and left the other one to stall.

Production failure modes

The MSS clamping failure modes the operator encounters:

• Clamp on an interface the traffic does not leave through. The node is committed, show interfaces confirms it, and the capture on the constrained interface still shows 1460. Fix: put adjust-mss on the interface that faces the bottleneck — for a tunnel, the tunnel interface.
• Only one direction clamped. The client’s SYN is rewritten; the server’s SYN-ACK is not, so large segments still arrive from the far side and die at the tunnel. Fix: clamp on both ends, or on both the tunnel interface and the interface the SYN-ACK leaves through.
• MSS too aggressive. The MSS is smaller than the path actually requires. Throughput suffers (small TCP segments have more per-segment overhead). Fix: raise the MSS to the real path MTU - 40.
• MSS too lenient. The MSS is larger than the path allows. Large packets are dropped at the bottleneck and the original stall returns. Fix: lower the MSS to the real path MTU - 40.
• clamp-mss-to-pmtu on an interface whose MTU is not the constraint. The value is derived from the local interface MTU, so a 1500-MTU WAN interface clamps to 1460 and the remote bottleneck is untouched. Fix: use the numeric form.
• Clamping does not rescue non-TCP traffic. The MSS option is a TCP option; UDP, ICMP and QUIC never see it. A QUIC-based transfer over the same broken path still fails. Fix: the UDP protocol has to do its own path MTU handling, or the MTU itself has to be corrected.
• IPv6 MSS calculation error. The operator sets the IPv4 value (MTU - 40) on the ipv6 adjust-mss node. The MSS is 20 bytes too large and IPv6 keeps stalling while IPv4 is fine. Fix: MTU - 60 for IPv6.

Rollback

MSS clamping changes are simple to roll back:

• commit-confirm 5 — auto-rollback restores the previous interface configuration if you lose the session.
• delete interfaces vti vti0 ip adjust-mss (and the ipv6 node) then commit — removes the clamp.

One caveat on testing either direction of the change: the clamp acts on the handshake, so established connections keep whatever MSS they negotiated. Neither applying nor removing it is visible on a session that is already up — open a fresh connection before concluding anything from a capture.

Production discipline

Cross-course references

• Part LI-01 (LI-VyOS-MTU / MTU basics) covers the canonical MTU values.
• Part LI-02 (LI-VyOS-MTU / tunnel overhead) covers the overhead calculations for tunnels.
• Part LI-03 (LI-VyOS-MTU / PMTUD) covers Path MTU Discovery; MSS clamping is the fallback when PMTUD fails.
• Part LI-05 (LI-VyOS-MTU / MTU and fragmentation troubleshoot) covers the operational diagnostic for MTU mismatches.
• Part LI-06 (LI-VyOS-MTU / MTU validation) covers the clamp-mss-to-pmtu trap in full.
• Part XXXVII (XXXVII-VyOS-Firewall) covers the 1.4+ firewall model, including why per-interface rule bindings no longer exist.

Quiz