VXLAN underlay — underlay routing, MTU 1550, jumbo frames

The VXLAN underlay is the Layer 3 network that carries the encapsulated VXLAN packets between VTEPs. The underlay is typically an IP network within a data center or across data centers (via WAN or private interconnect). The underlay must be properly routed, must have sufficient MTU to carry VXLAN packets, and must provide the connectivity that VXLAN overlays build on.

This lesson covers the underlay routing requirements, where the MTU number comes from, routing protocol design, and the production failure modes.

Why the underlay matters

The underlay is the foundation of the VXLAN deployment. If the underlay is broken (routing loops, MTU too small, congestion), the overlay experiences failures that look like VXLAN problems but are actually underlay problems.

A typical underlay:

flowchart LR
  subgraph DC1["Data Center 1"]
    SPINE1["Spine 1"]
    SPINE2["Spine 2"]
    LEAF1["Leaf 1<br/>(VTEP)"]
    LEAF2["Leaf 2<br/>(VTEP)"]
    SPINE1 --- SPINE2
    LEAF1 --- SPINE1
    LEAF1 --- SPINE2
    LEAF2 --- SPINE1
    LEAF2 --- SPINE2
  end
  subgraph DC2["Data Center 2"]
    SPINE3["Spine 3"]
    SPINE4["Spine 4"]
    LEAF3["Leaf 3<br/>(VTEP)"]
    LEAF4["Leaf 4<br/>(VTEP)"]
    SPINE3 --- SPINE4
    LEAF3 --- SPINE3
    LEAF3 --- SPINE4
    LEAF4 --- SPINE3
    LEAF4 --- SPINE4
  end
  SPINE1 -- "Inter-DC link" --> SPINE3

The underlay is a routed IP network (typically OSPF or BGP). The VTEPs are positioned at the leaf switches; the spine switches route VXLAN traffic between leaves and between data centers.

Where the MTU number comes from

Do not memorise 1550. Derive it, because the number changes with the outer address family and with VLAN tagging, and an operator who only knows the number cannot tell when it is wrong.

VXLAN wraps a complete inner Ethernet frame in a UDP datagram. On an IPv4 underlay, everything between the underlay link’s payload and the overlay’s IP payload is:

The outer Ethernet header is not counted, because MTU measures the payload an interface will carry, not the frame on the wire. So an overlay running a normal 1500-byte IP MTU produces a 1550-byte packet on the underlay, and the underlay interface MTU must be at least 1550.

Two adjustments matter in real networks:

• An IPv6 underlay replaces the 20-byte outer IPv4 header with a 40-byte outer IPv6 header: 70 bytes of overhead, so 1570.
• An 802.1Q tag on the inner frame adds 4 more bytes: 1554 (or 1574 on IPv6). Many operators round to 1600 for exactly this reason, and because it leaves room for whatever the next encapsulation turns out to be.

# Configure the underlay interface MTU on every VTEP and every
# transit device in the path
set interfaces ethernet eth0 mtu '1550'

Jumbo frames must be configured end-to-end (every switch port, every router interface) on the underlay path. A single link with MTU 1500 causes drops.

flowchart LR
  VTEP1["VTEP 1"]
  SW1["Switch<br/>MTU 1550"]
  SW2["Switch<br/>MTU 1550"]
  SW3["Switch<br/>MTU 1500<br/>(breaks the path)"]
  SW4["Switch<br/>MTU 1550"]
  VTEP2["VTEP 2"]
  VTEP1 -- "MTU 1550" --> SW1
  SW1 --> SW2
  SW2 --> SW3
  SW3 -. "oversize packets dropped" .-> SW4
  SW4 --> VTEP2

Routing the VTEP loopbacks

Each VTEP sources its VXLAN packets from a loopback address. The underlay’s entire job is to make those loopbacks reachable between VTEPs.

That gives the design one hard constraint that is easy to get wrong: the underlay routing session cannot run over the loopbacks it is responsible for distributing. Peer over the point-to-point link addresses. In a Clos fabric this is the standard eBGP-per-link design of RFC 7938 — a distinct AS per leaf, one AS per spine layer, and no IGP underneath:

# Leaf 1 (a VTEP), peering with both spines over the link subnets
set interfaces loopback lo address '192.0.2.10/32'
set interfaces ethernet eth0 address '10.0.0.10/31'
set interfaces ethernet eth1 address '10.0.1.10/31'

set protocols bgp system-as 65110
set protocols bgp parameters router-id '192.0.2.10'
set protocols bgp neighbor 10.0.0.11 remote-as '65100'
set protocols bgp neighbor 10.0.0.11 description 'spine1'
set protocols bgp neighbor 10.0.1.11 remote-as '65100'
set protocols bgp neighbor 10.0.1.11 description 'spine2'

# Advertise the VTEP loopback — this is what the remote VTEPs need
set protocols bgp address-family ipv4-unicast network '192.0.2.10/32'

The network statement lives under address-family ipv4-unicast. The bare set protocols bgp network <prefix> form was VyOS 1.3 and does not exist on 1.4 or later.

For a two-VTEP deployment the underlay can be a pair of static routes, and there is nothing wrong with that:

set protocols static route 192.0.2.20/32 next-hop 10.0.0.11

Static routes work for point-to-point deployments; BGP or OSPF scales better for many VTEPs and reconverges without an operator.

Configuration and validation

The VXLAN interface is a Layer 2 device: it carries Ethernet frames for a VNI. It belongs in a bridge with the local access ports, and any IP address for that segment belongs on the bridge, not on the VXLAN interface. Putting the tenant address directly on vxlan10001 gives you a routed link between two VTEPs with nothing for the tenant hosts to attach to, which is not what a VNI is for.

configure

# Loopback — the VTEP source address
set interfaces loopback lo address '192.0.2.10/32'

# Underlay interface, sized for the encapsulation
set interfaces ethernet eth0 address '10.0.0.10/31'
set interfaces ethernet eth0 mtu '1550'

# Underlay routing
set protocols bgp system-as 65110
set protocols bgp parameters router-id '192.0.2.10'
set protocols bgp neighbor 10.0.0.11 remote-as '65100'
set protocols bgp address-family ipv4-unicast network '192.0.2.10/32'

# VXLAN interface for VNI 10001 — no address on this interface
set interfaces vxlan vxlan10001 vni '10001'
set interfaces vxlan vxlan10001 source-address '192.0.2.10'
set interfaces vxlan vxlan10001 remote '192.0.2.20'
set interfaces vxlan vxlan10001 port '4789'
set interfaces vxlan vxlan10001 mtu '1500'

# The bridge is where the segment lives: the VXLAN interface plus
# the local access port (eth0 and eth1 are underlay uplinks here).
# The gateway address, if this router is the gateway for the
# segment, goes here.
set interfaces bridge br10001 member interface vxlan10001
set interfaces bridge br10001 member interface eth2
set interfaces bridge br10001 address '10.0.2.1/24'

commit
save

Two details in that block are worth stating plainly rather than leaving to the defaults:

• port '4789'. 4789 is the IANA-assigned VXLAN port and what RFC 7348 specifies, but the Linux VXLAN driver has historically defaulted to 8472, and a capture filter or firewall rule written for the wrong one finds nothing. Set the port explicitly, then confirm what is actually on the wire before you trust a filter.
• mtu '1500'. VyOS gives the VXLAN interface a default MTU chosen to survive a 1500-byte underlay, which is smaller than 1500. Once the underlay carries 1550 the overlay can run a normal 1500-byte MTU — but only if you say so. Check it with show interfaces vxlan vxlan10001 rather than assuming.

Validation, in the order that isolates a fault fastest:

# 1. Is the remote VTEP loopback in the table, and via the underlay?
show ip bgp summary
show ip route 192.0.2.20

# 2. Does the underlay carry a full-size encapsulated packet?
#    size 1522 + 8 (ICMP header) + 20 (IPv4 header) = 1550 on the wire.
ping 192.0.2.20 do-not-fragment size 1522 count 3 source-address 192.0.2.10

# 3. Is the VXLAN interface up, with the VNI, port and MTU you set?
show interfaces vxlan vxlan10001

# 4. Is the bridge forwarding, and has it learned remote MACs?
show bridge
bridge fdb show dev vxlan10001

# 5. Is anything on the wire?
monitor traffic interface eth0 filter 'udp port 4789'

Step 2 is the one people skip, and its failure mode is informative:

vyos@leaf1:~$ ping 192.0.2.20 do-not-fragment size 1522 count 3

PING 192.0.2.20 (192.0.2.20) 1522(1550) bytes of data.
From 10.0.0.11 icmp_seq=1 Frag needed and DF set (mtu = 1500)
From 10.0.0.11 icmp_seq=2 Frag needed and DF set (mtu = 1500)

--- 192.0.2.20 ping statistics ---
3 packets transmitted, 0 received, +2 errors, 100% packet loss

Read which address reports the problem. An error attributed to a remote address (“From 10.0.0.11 … Frag needed”) names the device whose MTU is too small — that is the hop to fix. A local error instead (“ping: local error: message too long”) means this router’s own outgoing interface MTU is the limit, and the fix is on the box you are typing on. If neither appears and the packets simply time out, some device is dropping the oversize frames without generating ICMP, which is the hardest variant and the reason to test with a known-good size first.

show bridge reports the membership. For the VXLAN device’s own forwarding entries — the remote-MAC-to-remote-VTEP mappings that make the overlay work — read them from iproute2 directly with bridge fdb show dev vxlan10001; the entries whose destination is the remote VTEP address are the ones that matter, and their absence on a flood-and-learn deployment means no frame has yet crossed in that direction.

A clean validation: the remote loopback is routed via the underlay, the 1550-byte probe succeeds, the VXLAN interface shows the VNI and MTU you configured, and the VXLAN forwarding database contains entries pointing at the remote VTEP.

Production failure modes

MTU mismatch

The underlay is left at 1500. A full-size overlay frame becomes a 1550-byte underlay packet and is dropped.

Diagnostic: ping 192.0.2.20 do-not-fragment size 1522 fails while a small ping succeeds. The overlay symptom is the giveaway: ARP and ping work, TCP connections establish and then stall on the first large transfer.

Fix: raise the MTU on every interface in the underlay path, including the transit devices you do not administer. Verify from both VTEPs — an asymmetric path can be large enough in one direction only.

The underlay session runs over the loopbacks it advertises

The operator configures the underlay BGP session between the VTEP loopbacks. The session cannot come up, because the route to the peer’s loopback is what the session was going to provide.

Diagnostic: show ip bgp summary shows the peer in Active and never established; show ip route 192.0.2.20 has no route.

Fix: peer over the point-to-point link addresses. Reserve loopback peering for the overlay (EVPN) sessions, which run on top of an underlay that already works.

eBGP session Established, no prefixes

FRR implements RFC 8212: depending on whether bgp ebgp-requires-policy is in force, an eBGP session with no inbound and outbound policy can be fully established and still exchange nothing. VyOS exposes it as set protocols bgp parameters ebgp-requires-policy.

Diagnostic: show ip bgp summary shows a healthy uptime and PfxRcd of 0. Confirm the setting with vtysh -c 'show running-config' rather than assuming the default.

Fix: attach a route-map to the neighbour’s address-family in both directions, or take the parameter out of force deliberately.

Loopback not advertised

The VTEP loopback is configured but never advertised, so remote VTEPs have no route to it. The encapsulated packets have nowhere to go, and — because the source address is unreachable — nothing comes back either.

Diagnostic: show ip route 192.0.2.20 on the local VTEP shows no route; on the remote VTEP, show ip bgp does not contain 192.0.2.10/32.

Fix: set protocols bgp address-family ipv4-unicast network '192.0.2.10/32' on each VTEP, and confirm the prefix appears in the remote table rather than only in the local configuration.

Underlay convergence is slower than the overlay expects

A link fails. The underlay reconverges in seconds; the overlay’s traffic is black-holed for that whole window, and to a tenant it looks like the application broke.

Fix: BFD on the underlay sessions, ECMP across at least two spines so that a single failure does not require reconvergence to restore a path, and no reliance on timer expiry for detection.

VXLAN blocked by the router’s own firewall

VXLAN is decapsulated by the router itself, so it is the input chain that matters on VyOS 1.4 and later — not a forward rule.

set firewall ipv4 input filter rule 40 action 'accept'
set firewall ipv4 input filter rule 40 protocol 'udp'
set firewall ipv4 input filter rule 40 destination port '4789'
set firewall ipv4 input filter rule 40 source address '192.0.2.0/24'

Match the port to the one configured on the VXLAN interface. A rule written for 4789 while the interface is actually using another port matches nothing, and the chain’s default action then drops the traffic — an accept rule that looks correct in show configuration and permits none of the traffic it names.

Rollback

# Snapshot before touching anything. `save` takes a path and is a
# configuration-mode command.
configure
save /config/vyos-vxlan-underlay-before.conf

# Review the candidate before committing
compare

# Back out the underlay change
delete interfaces ethernet eth0 mtu
delete protocols bgp address-family ipv4-unicast network 192.0.2.10/32
commit

# Or reload the snapshot
load /config/vyos-vxlan-underlay-before.conf
commit

Removing the MTU returns the interface to its default and immediately breaks any full-size overlay traffic; withdrawing the loopback breaks the VXLAN path entirely. Both are the kind of change to make under commit-confirm 5 if the path you are editing is the one carrying your session.

Production discipline

Cross-course references

• Part XLIV-01 (XLIV-VyOS-VXLAN / concept) covers the VXLAN protocol.
• Part XLIV-02 (XLIV-VyOS-VXLAN / VNI) covers VNI design.
• Part XLIV-04 (XLIV-VyOS-VXLAN / BGP EVPN) covers BGP EVPN as the control plane that runs on top of this underlay.
• Part XLII-04 (XLII-VyOS-IPsec / VTI) covers the same MTU arithmetic for an encrypted overlay.

Quiz