VXLAN without EVPN — flood-and-learn, multicast underlay

VXLAN can be deployed without BGP EVPN, using the flood-and-learn mechanism defined in the original RFC 7348. Flood-and-learn uses multicast in the underlay to propagate unknown unicast, broadcast, and multicast traffic; VTEPs learn remote MACs by observing the source MAC of received frames. This is simpler than BGP EVPN but has limitations (requires multicast in underlay; doesn’t scale to many VNIs; inefficient for large deployments).

This lesson covers the flood-and-learn mechanism, multicast underlay configuration, the limitations, and when to use flood-and-learn vs BGP EVPN.

How flood-and-learn works

In flood-and-learn VXLAN, a VTEP that does not know the destination MAC of a frame floods the frame to all VTEPs in the same VNI. The mechanism:

1. Local learning. The VTEP learns MACs of locally attached VMs (from ARP requests, gratuitous ARPs, traffic received).
2. Flooding unknown unicast. When a frame arrives for an unknown MAC, the VTEP encapsulates it in VXLAN and sends to the multicast group for the VNI.
3. Multicast delivery. The underlay multicast routing (PIM) delivers the VXLAN packet to all VTEPs subscribed to the multicast group.
4. Remote learning. The remote VTEPs receive the VXLAN packet, decapsulate it, and learn the source MAC of the inner frame. The MAC is associated with the source VTEP.
5. Subsequent unicast. Once the MAC is known, subsequent frames use unicast VXLAN (point-to-point).

sequenceDiagram
  participant VTEP1 as VTEP 1 has VM_A1
  participant MCAST as Multicast Group 239.1.1.1
  participant VTEP2 as VTEP 2 has VM_A2
  Note over VTEP1: VM_A1 sends ARP for VM_A2<br/>destination MAC unknown to VTEP1
  VTEP1->>MCAST: ARP encapsulated in VXLAN<br/>destination 239.1.1.1
  MCAST->>VTEP2: VXLAN packet
  VTEP2->>VTEP2: Decapsulate and learn VTEP1 MAC
  VTEP2->>VTEP2: ARP reply from VM_A2
  VTEP2->>MCAST: ARP reply encapsulated in VXLAN
  MCAST->>VTEP1: VXLAN packet
  VTEP1->>VTEP1: Decapsulate and learn VTEP2 MAC
  Note over VTEP1,VTEP2: Subsequent frames use unicast VXLAN

The use of multicast is fundamental to flood-and-learn. Without multicast in the underlay, the unknown-unicast flooding cannot reach all VTEPs.

Configure flood-and-learn VXLAN

A typical flood-and-learn configuration:

configure
# Underlay interface — multicast VXLAN sends from this device
set interfaces ethernet eth0 address 10.0.0.1/24
set interfaces ethernet eth0 mtu 1550

# Tenant access port
set interfaces ethernet eth1 vif 101

# VXLAN interface with a multicast group for BUM traffic
set interfaces vxlan vxlan10001 vni 10001
set interfaces vxlan vxlan10001 group 239.1.1.1
set interfaces vxlan vxlan10001 source-interface eth0
set interfaces vxlan vxlan10001 mtu 1500

# The segment lives on the bridge. The VXLAN device is a port on it and
# carries no address; the tenant gateway, if this router is the gateway,
# goes here.
set interfaces bridge br10001 member interface vxlan10001
set interfaces bridge br10001 member interface eth1.101
set interfaces bridge br10001 address 10.0.1.1/24

# Multicast routing on the underlay
set protocols pim interface eth0
set protocols pim rp address 192.0.2.100 group 239.0.0.0/8

commit
save

Four things in that block are load-bearing, and three of them are enforced by commit rather than by the CLI:

• source-interface, not source-address. A multicast VXLAN interface that does not name an underlying device is rejected with Multicast VXLAN requires an underlying interface: the kernel needs a device to send the group join and the encapsulated BUM traffic out of. The outer source address is then eth0’s, so the loopback-as-VTEP-identity pattern from the unicast lessons does not transfer here unchanged.
• group and remote are mutually exclusive. Setting both is refused with Both group and remote cannot be specified. An interface either floods to a multicast group or to a list of remote VTEPs; there is no hybrid, and choosing between them is the design decision this lesson is about.
• No address on the VXLAN device. It is a bridge member, and vyos-1x refuses an address on a bridge member: Cannot assign address to interface "vxlan10001" as it is a member of bridge "br10001"! The tenant gateway belongs on br10001, which is what makes the VNI a segment that local hosts can join rather than a routed link between two routers.
• PIM is configured, not assumed. There is no default that enables it. 239.0.0.0/8 is an any-source range, so the VTEPs also need a rendezvous point they agree on; the static rp address line is the least surprising way to say so, and its value must be identical on every VTEP in the group.

The overlay MTU is 1500 — the VyOS default for a VXLAN interface, and the value commit warns you about going below. The 1550 on eth0 is what absorbs the encapsulation, and Part XLIV-03 derives it.

When a frame for an unknown MAC arrives, the VTEP encapsulates it in VXLAN with destination = multicast group (239.1.1.1). The underlay multicast routing (PIM) delivers the packet to all VTEPs subscribed to the group.

Unicast flood with static entries

For deployments without multicast support, a “unicast flood with static entries” configuration is a workaround:

# No group here, so the loopback can be the VTEP identity again
set interfaces loopback lo address 192.0.2.10/32

# Configure the VXLAN with a list of remote VTEPs (unicast flood targets)
set interfaces vxlan vxlan10001 vni 10001
set interfaces vxlan vxlan10001 source-address 192.0.2.10
set interfaces vxlan vxlan10001 remote 192.0.2.20
set interfaces vxlan vxlan10001 remote 192.0.2.30
set interfaces vxlan vxlan10001 remote 192.0.2.40
set interfaces vxlan vxlan10001 mtu 1500
# All remote VTEPs listed; unknown unicast floods to all of them

# Same bridge arrangement as before: the address is on br10001, never on
# vxlan10001
set interfaces bridge br10001 member interface vxlan10001
set interfaces bridge br10001 member interface eth1.101
set interfaces bridge br10001 address 10.0.1.1/24

The VTEP floods unknown unicast to each remote VTEP. This is inefficient (the remote VTEP receives the flood even if it doesn’t have the MAC) but works without multicast.

Two differences from the multicast block are worth naming. There is no group, so source-address is available again and the loopback resumes its job as the stable VTEP identity. And the remote list is now configuration: every VTEP in the segment has to appear in it, on every other VTEP, which is the O(N^2) bookkeeping that BGP EVPN exists to delete.

The trade-off:

• Multicast flood-and-learn: efficient, requires multicast in underlay.
• Unicast flood with static entries: less efficient, no multicast required.
• BGP EVPN: most efficient, no multicast required.

Limitations of flood-and-learn

The flood-and-learn approach has several limitations:

1. Multicast in underlay required. If the underlay does not support multicast, flood-and-learn cannot work (without the static-entry workaround).
2. Inefficient flooding. Unknown unicast floods to all VTEPs (or all in the multicast group). For a deployment with 100 VTEPs in the same VNI, every unknown frame generates 99 unnecessary VXLAN packets.
3. Convergence slow. A new VTEP joining the multicast group does not have any MACs learned; all unknown unicast is flooded until the MACs are learned.
4. No inter-VXLAN routing without extra config. Inter-VXLAN routing (a Layer 3 router between VXLAN segments) requires an external router or a default gateway configured.
5. No multi-tenancy via route-targets. Each VNI is isolated by VNI, not by route-target; there is no equivalent to BGP EVPN’s route-targets.

For data centers with many VNIs and many VTEPs, BGP EVPN is preferred. For small deployments (a few VTEPs, a few VNIs), flood-and-learn is acceptable.

Validation

# Verify the VXLAN state
show interfaces vxlan vxlan10001
# Shows the interface state, multicast group, source interface;
# it must show no address of its own

# Verify the segment
show bridge
show interfaces bridge br10001
# br10001 holds vxlan10001, the access port, and the tenant gateway

# Inspect the FDB
bridge fdb show vni 10001
# Lists the MACs known to the VTEP
bridge fdb show dev vxlan10001
# The same view restricted to one VXLAN device: the entries whose
# destination is a remote VTEP address are the learned remote MACs

# Capture on the underlay
tcpdump -ni eth0 'udp port 4789 or igmp' -c 10 -vv
# VXLAN packets and IGMP multicast joins

# Verify PIM
show ip pim interface
# PIM interface state
show ip pim rp-info
# The rendezvous point the router believes in, per group range

# Test connectivity
ping 10.0.1.10
# Should succeed via VXLAN

# Test a broadcast
arping -U -I eth1.101 10.0.1.50
# Should be flooded via multicast

A clean validation: the VXLAN interface is up and carries no address; br10001 lists the VXLAN device, the access port and the tenant gateway; the FDB has local and remote MACs; PIM is enabled and agrees on a rendezvous point; ping to a remote VM succeeds; broadcast packets are flooded.

Production failure modes

Multicast not supported in underlay

The underlay (e.g., a private WAN between data centers) does not support multicast. Flood-and-learn cannot deliver unknown frames.

Diagnostic: unknown unicast is not received by remote VTEPs; the FDB does not have remote MACs.

Fix: switch to BGP EVPN or unicast flood with static entries.

Multicast group mismatch

VTEP1 is configured with multicast group 239.1.1.1; VTEP2 is configured with 239.2.2.2. The two VTEPs are in different multicast groups; they don’t see each other’s floods.

Diagnostic: unknown unicast from VTEP1 is not received by VTEP2; remote MACs are not learned.

Fix: align the multicast group on all VTEPs in the same VNI.

PIM not configured

PIM is not enabled on the underlay interface. The multicast group has no members; the flood is not delivered.

Diagnostic: show ip pim interface shows no PIM interface; IGMP joins are not processed.

Fix: enable PIM on the underlay interface.

The group was configured without a source interface

The VXLAN interface has group and no source-interface. The change never reaches the running configuration at all: commit fails with Multicast VXLAN requires an underlying interface, because the kernel has no device to send the group join out of.

Diagnostic: the failure is at commit time and names itself. The related one is quieter — group and remote configured together, refused with Both group and remote cannot be specified, which usually means somebody was migrating between the two flooding modes and left half of each.

Fix: name the underlay device with set interfaces vxlan vxlan10001 source-interface eth0, and delete whichever of group or remote does not belong to the mode you chose.

The tenant address was put on the VXLAN interface

set interfaces vxlan vxlan10001 address 10.0.1.1/24 is accepted at the CLI and rejected at commit with Cannot assign address to interface "vxlan10001" as it is a member of bridge "br10001"!. The dangerous fix is deleting the bridge to make the error go away: the command then commits, and the VNI becomes a routed link between VTEPs with no access port on it. Flood-and-learn has nothing left to learn, because no local host is on the segment to send a frame.

Diagnostic: show bridge lists nothing for the VNI, show interfaces vxlan vxlan10001 shows an address, and bridge fdb show dev vxlan10001 stays empty on both VTEPs.

Fix: put the address back on the bridge and put both the VXLAN device and the tenant access port into it.

No rendezvous point for the group

PIM is enabled on the underlay interfaces, but no RP is configured for the 239.0.0.0/8 range, or the VTEPs disagree about which address it is. An any-source group has no shared tree without one, so the flood is encapsulated, sent, and dropped in the underlay.

Diagnostic: show ip pim rp-info is empty or shows a different address on different VTEPs; tcpdump -ni eth0 'udp port 4789' shows the VTEP transmitting and no remote VTEP receiving.

Fix: configure the same set protocols pim rp address ... group ... on every VTEP, or move the design to an SSM range and configure it as such.

MAC table size exceeded

The number of MACs in the FDB exceeds the hardware limit. New MACs are not learned.

Diagnostic: bridge fdb show shows the FDB is at the limit; new MACs are not in the table.

Fix: increase the FDB limit (if hardware supports); use BGP EVPN for large-scale MAC distribution.

Rollback

# Enter configuration mode and write the running configuration to a
# file you can load back. `save` is a configuration-mode command that
# takes a path; operational mode has no `| save` pipe.
configure
save /config/pre-change-vxlan-flood-learn-TICKET.conf

# Remove the segment, then the tunnel — the order matters
delete interfaces bridge br10001
delete interfaces vxlan vxlan10001

# Read the diff before committing anything
compare
commit

# Or restore a previous configuration
load /config/pre-change-vxlan-flood-learn-TICKET.conf
commit
save

The rollback removes the bridge and the VXLAN interface; the flood-and-learn is disabled. Deleting interfaces vxlan vxlan10001 while br10001 still lists it as a member is refused with Interface "vxlan10001" cannot be deleted as it is a member of bridge "br10001"!, so release the membership in the same commit. The PIM configuration is left alone deliberately: it is underlay state that other services may be using, and removing it is a separate decision.

Production discipline

Cross-course references

• Part XLIV-01 (XLIV-VyOS-VXLAN / concept) covers the VXLAN protocol.
• Part XLIV-03 (XLIV-VyOS-VXLAN / underlay) covers the underlay requirements.
• Part XLIV-04 (XLIV-VyOS-VXLAN / BGP EVPN) covers BGP EVPN as the production standard.

Quiz