VXLAN troubleshooting — VNI mismatch, underlay MTU, EVPN routes that never arrive

VXLAN failures fall into a small number of categories, and each one is answered by a different command:

• The segment does not match. Different VNIs on the two VTEPs, or a VXLAN device that is not a member of the bridge it is supposed to serve.
• The underlay cannot carry the packet. VXLAN adds 50 bytes to every frame, and a path that cannot pass 1550 bytes drops the large ones silently while ping and ARP keep working.
• The EVPN control plane carries nothing. The BGP session is Established and no MAC ever arrives. This has three distinct causes and they need to be separated before anything is changed.
• Multicast never joins, on a flood-and-learn build, so BUM traffic goes nowhere.

The tools are show interfaces vxlan, show evpn vni, show bgp l2vpn evpn, bridge fdb show, and a capture on the underlay. This lesson is the diagnostic reference: the flow, the four failure modes, and — where VyOS gives you nothing to run — where the state actually lives instead.

The diagnostic flow

The systematic flow for any VXLAN failure:

1. Symptom — what the operator sees (no connectivity, slow throughput, only some VNIs work).
2. VTEP state — show interfaces vxlan on both VTEPs.
3. Underlay state — show ip route to verify the underlay is routing VXLAN packets.
4. Overlay state — bridge fdb show to verify local and remote MACs are learned.
5. EVPN state — for BGP EVPN, show bgp l2vpn evpn.
6. Capture on the wire — tcpdump to verify VXLAN packets on the underlay.

# The minimal VXLAN diagnostic set, run on VTEP1 (192.0.2.10),
# whose peer VTEP2 is 192.0.2.20.

# 1. VXLAN interface state: the VNI, the source address, and
#    whether the device is up.
show interfaces vxlan vxlan10001

# 2. Underlay reachability to the peer's VTEP address. This is the
#    address VXLAN encapsulates towards, so it is the one that must
#    be routed - not the tenant addresses inside.
show ip route 192.0.2.20

# 3. Bridge membership and forwarding database. Both are Linux
#    utilities run from the VyOS shell, not VyOS op-mode commands.
bridge link show
bridge fdb show dev vxlan10001

# 4. EVPN state, on an EVPN build.
show evpn vni
show bgp l2vpn evpn summary

# 5. Capture on the underlay.
sudo tcpdump -ni eth0 'udp port 4789' -c 10 -vv

Two of those are worth a note before you use them. bridge is iproute2, not VyOS: there is no show bridge fdb op-mode command, and the forwarding database is the kernel’s rather than something VyOS renders, which is exactly why it is the honest answer to “what does this VTEP believe right now”. And show evpn vni is FRR, reached through the VyOS shell like the other routing commands — it is the fastest single check on an EVPN fabric, because it reports what zebra found rather than what BGP was told.

Failure mode 1 — the segment does not match

Symptom: hosts on the two sides cannot reach each other at all. ARP goes unanswered; the VXLAN devices are up on both VTEPs and the underlay is fine.

Diagnostic:

# The VNI each VTEP believes in. Run this on both.
show interfaces vxlan vxlan10001
show configuration commands | match vxlan

# Bridge membership: the VXLAN device and the customer port must be
# ports of the SAME bridge, and both must be up.
bridge link show

# What the kernel has learned, per device.
bridge fdb show dev vxlan10001

# The inner traffic, after decapsulation.
sudo tcpdump -ni vxlan10001 -c 4

The interface name is not the VNI. vxlan10001 is a name a human chose, and VyOS does not derive anything from it — the VNI is the vni leaf. Two routers can both have an interface called vxlan10001 carrying different VNIs, and every command that prints the interface name will look consistent while the fabric is silently cut in two. Read the vni value, not the name.

Common causes, in the order they are worth checking:

1. The VNIs differ. VTEP1 has vni 10001, VTEP2 has vni 10002. Nothing errors: each VTEP sends packets the other decapsulates into a segment it does not serve, and drops.
2. The VXLAN device is not in the bridge, or the customer port is not. bridge link show lists the ports of each bridge; a VXLAN device that is up but not a bridge member is a tunnel that terminates nowhere. This produces exactly the same symptom as a VNI mismatch, and it is the more common of the two.
3. A VLAN-aware bridge is filtering the frame. Where the bridge has VLAN filtering enabled, a port that does not carry the tenant VLAN drops the traffic before VXLAN is ever involved.
4. The VNI was reassigned. A tenant was moved and one VTEP was not updated.

The discrimination is in the forwarding database. Local MACs and no remote MACs means the frames the far end sends are not being accepted into this segment — a VNI or membership fault. No MACs at all, not even local ones, means the problem is on this side of the tunnel: the customer port, not the overlay.

flowchart TD
  A["Symptom: no connectivity"] --> B{"VTEP1 sees the frame?"}
  B -- "no" --> C["Investigate local VM"]
  B -- "yes" --> D{"VXLAN packet leaves VTEP1?"}
  D -- "no" --> E["Investigate VXLAN encapsulation"]
  D -- "yes" --> F{"VTEP2 receives the packet?"}
  F -- "no" --> G["Investigate underlay routing"]
  F -- "yes" --> H{"VTEP2 has matching VNI?"}
  H -- "no" --> I["Fix VNI mismatch"]
  H -- "yes" --> J["Investigate frame delivery to local VM"]

Failure mode 2 — the underlay cannot carry the packet

Symptom: the overlay half-works. Ping succeeds, ARP resolves, DNS answers, and then a file transfer or a TLS handshake stalls. Small frames pass and large ones vanish with nothing logged anywhere.

This is the failure mode most often misdiagnosed as an application problem, because everything an operator normally uses to test connectivity is small.

The arithmetic, which the probe depends on

VXLAN over IPv4 adds 50 bytes to the frame it carries: 14 bytes of inner Ethernet header, 8 of VXLAN, 8 of UDP and 20 of outer IPv4. A tenant that wants an ordinary 1500-byte MTU therefore needs the underlay to carry 1550-byte IP packets end to end. Over IPv6 the outer header is 40 rather than 20, making the overhead 70. An inner 802.1Q tag adds another 4.

To prove the underlay carries 1550 bytes, the probe has to be 1550 bytes. ping counts its size as ICMP payload only, and adds 8 bytes of ICMP header and 20 of IPv4 — so the size that tests a 1550-byte path is 1522, not 1500. Getting this wrong by 28 bytes is how an underlay gets declared healthy while still being 28 bytes short.

# From the VyOS CLI. do-not-fragment is what makes this a test:
# without it the kernel fragments and the probe succeeds regardless.
ping 192.0.2.20 size 1522 do-not-fragment count 3

# Bracket the failure to find the real ceiling.
ping 192.0.2.20 size 1472 do-not-fragment count 3
ping 192.0.2.20 size 1600 do-not-fragment count 3

# The local MTU on each hop you own.
show interfaces ethernet eth0

Where the probe fails, the useful next question is which hop. A traceroute will not tell you: it does not probe with large packets. What narrows it down is testing hop by hop from the underlay routing table, and asking whoever owns the transit switches for their MTU — the intermediate device is the usual culprit and the one you cannot see from the VTEP.

Common causes:

1. The underlay is at the default 1500. Everything below 1450 of tenant payload works. This is the classic case.
2. One link in the path is at 1500 while the VTEPs are jumbo. Both routers look correct and the fabric is still broken.
3. The VTEPs disagree. VTEP1 at 1550 and VTEP2 at 1500 gives a fabric that works in one direction for large frames and not the other — which is much harder to recognise than a symmetric failure, so test both directions before concluding.
4. The overlay MTU was never lowered on a fabric whose underlay genuinely cannot go above 1500. The fix there is set interfaces vxlan vxlan10001 mtu 1450, which makes the tenant’s frames small enough to fit rather than pretending the underlay is bigger than it is.

Failure mode 3 — the EVPN control plane carries nothing

Symptom: BGP is Established, show bgp l2vpn evpn is empty or one-sided, and no remote MAC ever appears in the forwarding database.

“Route-type mismatch” is not what is happening, and it is worth saying plainly because the phrase circulates. EVPN route types are not negotiated and cannot be mismatched: every VTEP that speaks the L2VPN EVPN address family understands Type-2, Type-3 and Type-5, and a router that has nothing to advertise simply advertises nothing. What actually goes wrong are three separable faults, and they are checked in this order because each one makes the next impossible to assess.

1. The peer was never activated in the EVPN address family. VyOS emits no bgp default ipv4-unicast into FRR, so a neighbour participates only in the families named under it. Without set protocols bgp neighbor 192.0.2.20 address-family l2vpn-evpn, the session reaches Established and carries no EVPN routes at all. The tell is that show bgp summary lists the peer and show bgp l2vpn evpn summary does not — the second command only shows peers activated in that family.

2. advertise-all-vni is missing. The node is set protocols bgp address-family l2vpn-evpn advertise-all-vni, and it is the single line that turns EVPN on: it makes bgpd ask zebra for the kernel’s VNI list and begin originating routes. Without it the session is healthy, the VXLAN device exists, and the router originates nothing. show evpn vni printing an empty list on a router that has VXLAN interfaces is this fault, and it is the commonest of the three.

3. The route-targets no longer overlap. FRR derives them automatically: the export RT is the low 16 bits of the local AS then the VNI (65000:10001 on AS 65000, VNI 10001), and the import RT is derived as a wildcard, *:10001, which matches that VNI from any AS. That asymmetry is why an untouched fabric interoperates across ASNs. Configure an explicit set protocols bgp address-family l2vpn-evpn vni 10001 route-target import ... and you replace the wildcard with a literal — after which every peer whose export RT does not match it goes dark, including peers that were working a moment earlier. The vni node exists only to override the derivation; it is not something a working fabric needs.

# 1. Is the peer activated in this address family at all?
show bgp summary
show bgp l2vpn evpn summary

# 2. Did EVPN turn on? This lists the VNIs zebra found, with the
#    MAC count and the remote-VTEP count for each.
show evpn vni
show evpn vni 10001

# 3. What is in the EVPN table, and specifically the MAC/IP routes?
show bgp l2vpn evpn
show bgp l2vpn evpn route type macip
show bgp l2vpn evpn route type multicast

# 4. Which route-targets is this router importing?
show bgp l2vpn evpn import-rt

# 5. Did any of it become forwarding state?
show evpn mac vni 10001
bridge fdb show dev vxlan10001

# 6. The configuration, on both ends.
show configuration commands | match l2vpn-evpn

Read show evpn vni before anything else on an EVPN fabric. It answers all three questions above in one line per VNI: an absent VNI means advertisement is off or there is no VXLAN device; a present VNI with zero remote VTEPs means Type-3 routes are not being imported, which points at route-targets or activation; a present VNI with remote VTEPs and no MACs means the control plane works and the fault is below it.

flowchart TD
  A["Symptom: remote MACs never learned"] --> B{"Peer listed in<br/>show bgp l2vpn evpn summary?"}
  B -- "no" --> C["Activate it:<br/>neighbor X address-family l2vpn-evpn"]
  B -- "yes" --> D{"show evpn vni<br/>lists the VNI?"}
  D -- "no" --> E["advertise-all-vni missing,<br/>or no VXLAN device in the kernel"]
  D -- "yes" --> F{"Remote VTEP count<br/>above zero?"}
  F -- "no" --> G["Type-3 not imported:<br/>compare import-rt with the<br/>peer's export RT"]
  F -- "yes" --> H{"MACs in<br/>show evpn mac vni?"}
  H -- "no" --> I["Routes present, not installed:<br/>VNI or bridge membership"]
  H -- "yes" --> J["Control plane is healthy —<br/>go to MTU and the underlay"]

Failure mode 4 — Multicast issues (flood-and-learn)

Symptom: flood-and-learn VXLAN; unknown unicast is not reaching remote VTEPs; remote MACs are not learned.

Common causes:

1. The VTEP never joined the group. The kernel joins the VXLAN device’s multicast group on the underlay interface when the device comes up. If the join is not there, nothing downstream matters.
2. PIM is not configured on the underlay, so the join is never propagated beyond the first hop.
3. The groups differ. Each VNI is bound to a group; two VTEPs configured with different groups for the same VNI flood into separate trees and never hear each other.
4. There is no multicast routing at all between the VTEPs — common across a WAN or a cloud underlay, where the answer is not to fix multicast but to use head-end replication or EVPN instead.

# 1. Did this VTEP actually join the group? This is the kernel's
#    own membership list and it is the first thing to check.
ip maddr show dev eth0

# 2. PIM on the underlay, and the neighbours it has found.
show ip pim interface
show ip pim neighbor

# 3. The multicast routing state for the group.
show ip mroute

# 4. On the wire: IGMP membership reports and PIM hellos.
sudo tcpdump -ni eth0 'igmp or proto pim' -c 10

# 5. And the encapsulated BUM traffic itself, addressed to the group
#    rather than to a peer VTEP.
sudo tcpdump -ni eth0 'udp port 4789 and dst 239.1.1.1' -c 10

The last capture is the one that settles it. VXLAN BUM traffic in a flood-and-learn build leaves as a UDP 4789 packet addressed to the group. Seeing it on the sender and not on the receiver puts the fault squarely in the underlay’s multicast routing, which is a different team’s problem in most organisations and worth establishing before the conversation starts.

A complete diagnostic flow

Run it in this order on VTEP1, then run the same sequence on VTEP2 and compare. The order matters: each step assumes the one before it passed, and a check performed out of order produces evidence you cannot interpret.

# 1. Is the overlay device what you think it is?
show interfaces vxlan vxlan10001
bridge link show

# 2. Is the peer VTEP address routed through the underlay?
show ip route 192.0.2.20

# 3. Can the underlay carry a full-size encapsulated frame?
ping 192.0.2.20 size 1522 do-not-fragment count 3

# 4. On an EVPN build: is the control plane alive and originating?
show evpn vni
show bgp l2vpn evpn summary
show bgp l2vpn evpn route type macip

# 5. Did any of that become forwarding state?
show evpn mac vni 10001
bridge fdb show dev vxlan10001

# 6. The underlay on the wire: encapsulated packets leaving and arriving.
sudo tcpdump -ni eth0 'udp port 4789' -c 10 -vv

# 7. The overlay on the wire: the tenant's own frames, decapsulated.
sudo tcpdump -ni vxlan10001 -c 10

Steps 6 and 7 are a pair and the pairing is the point. Encapsulated packets leaving VTEP1 and not arriving at VTEP2 is an underlay fault. Arriving at VTEP2 and producing nothing on vxlan10001 is a VNI or bridge fault at the far end. Both captures showing traffic while the application still fails moves the investigation off VXLAN entirely.

Production failure modes

The underlay cannot carry 1550 bytes

The underlay is at the default 1500 and the tenant is at 1500 too. Small frames pass; anything approaching full size does not.

Diagnostic: ping 192.0.2.20 size 1522 do-not-fragment count 3 fails while size 1000 succeeds.

Fix: raise the underlay to 1550 or more on every hop, or lower the overlay with set interfaces vxlan vxlan10001 mtu 1450. Raising one VTEP and not the other creates a directional fault that is harder to find than the original problem.

The peer VTEP address is not routed

VXLAN encapsulates towards the peer’s VTEP address — the loopback, on a well-built fabric. If that address is not in the underlay routing table, nothing leaves.

Diagnostic: show ip route 192.0.2.20 returns no route, or a route pointing back through the overlay.

Fix: advertise the loopbacks in the underlay protocol. A route to the peer VTEP that resolves through a VXLAN interface is a recursion fault and will not carry traffic — the underlay must be reachable without the overlay.

An explicit route-target replaced the wildcard import

Somebody configured route-target import for one VNI to solve a specific interoperability problem. That replaced FRR’s auto-derived wildcard import (*:10001) with a literal, and every peer whose export RT does not match it stopped being imported.

Diagnostic: show bgp l2vpn evpn import-rt on the receiving VTEP, compared against the sending VTEP’s export RT. The routes are present in show bgp l2vpn evpn on the sender and absent on the receiver.

Fix: either give every VTEP the same set protocols bgp address-family l2vpn-evpn vni 10001 route-target both 65000:10001, or delete the explicit configuration everywhere and let the derivation apply again. What does not work is configuring it on one router.

The VXLAN device is up and belongs to nothing

show interfaces vxlan reports the device up, the VNI is right, and it is not a member of the bridge that carries the tenant port — or the tenant port is not.

Diagnostic: bridge link show. The VXLAN device and the customer-facing port must both appear as ports of the same bridge.

Fix: set interfaces bridge br10001 member interface vxlan10001. This one is worth checking early precisely because it is invisible in every VXLAN-specific command.

Multicast never joined

On a flood-and-learn build, the VTEP is not a member of the group, so BUM traffic is neither sent nor received where it should be.

Diagnostic: ip maddr show dev eth0 does not list the group; show ip pim neighbor shows no neighbours on the underlay interface.

Fix: enable PIM on the underlay, or stop depending on multicast — head-end replication and EVPN both remove the requirement entirely, and on any underlay you do not control that is the more durable answer.

Rollback

Everything above this section is read-only, so there is nothing to roll back until you change something. The two changes this lesson leads to — an MTU and a bridge membership — are both disruptive to the segment they touch, which is why the before-state is worth capturing even though the diagnosis was not.

Capture it with ordinary shell redirection. VyOS operational mode runs in a shell, so > works; there is no save pipe on an operational command, and show ... | save file is not a thing VyOS accepts.

show interfaces vxlan > /tmp/vxlan-before.txt
show evpn vni > /tmp/vxlan-evpn-before.txt
show bgp l2vpn evpn > /tmp/vxlan-bgp-before.txt
bridge fdb show > /tmp/vxlan-fdb-before.txt

Then take a configuration snapshot you can name later, and apply the change under a timer so a mistake reverts without you:

configure

# In configuration mode, save takes a path.
save /config/vxlan-before-change.conf

set interfaces vxlan vxlan10001 mtu 1450

# Read the change before applying it.
compare
commit-confirm 5

# Verify from a tenant host, not from the router, before confirming.
confirm
save

If the change was wrong, do nothing and let the timer expire; the router reverts on its own. If it has already been confirmed, go back to the file:

configure
load /config/vxlan-before-change.conf
compare
commit
save

Do not reach for op-mode rollback <revision> here. The VyOS documentation is explicit that it applies the stored revision by rebooting the router, which on a VTEP takes every other VNI on the box down with it — including the ones that were working. load from /config/archive/ or from your own snapshot, followed by compare and commit, is the in-place equivalent and the one to reach for.

Production discipline

Cross-course references

• Part LII-04 (LII-VyOS-Troubleshoot / subsystem by subsystem) covers the wider diagnostic methodology.
• Part XLIX-04 (XLIX-VyOS-Monitoring / VRRP telemetry) covers the monitoring integration.
• Part XLI-06 (XLI-VyOS-WireGuard / troubleshoot) covers a similar diagnostic approach for tunnel troubleshooting.

Quiz