VXLAN VNI — 24-bit namespace, tenant segmentation, VNI assignment

The VXLAN Network Identifier (VNI) is the 24-bit field in the VXLAN header that identifies a virtual network. With 24 bits, VXLAN supports up to 16 million (2^24) unique virtual networks — vastly more than the 4094 VLANs of 802.1Q. Each VNI represents an isolated Layer 2 segment; traffic in one VNI does not reach hosts in a different VNI.

This lesson covers the VNI namespace, VNI assignment strategies for multi-tenant deployments, configuring multiple VNIs on a single VTEP, and the production failure modes.

The VNI namespace

The VNI is a 24-bit integer, so the field holds 0 to 16,777,215 (2^24 - 1).

RFC 7348 reserves none of them. It defines the VNI as “a 24-bit value used to designate the individual VXLAN overlay network on which the communicating VMs are situated” and stops there; the reserved fields it does name are the 24 and 8 spare bits elsewhere in the header, not a value inside the VNI. Skipping VNI 0 is a widespread and sensible convention — it is the value you get from an uninitialised field — but it is a convention, and a design document that cites the RFC for it is citing something the RFC does not say.

What does constrain you is the platform. VyOS validates the vni leaf against the range 0 to 16777214, so the largest value the 24-bit field can hold is not settable from the CLI at all. Allocate tenant VNIs well inside that range and the difference never surfaces; write an allocator that counts up to the ceiling and it surfaces on the last tenant you onboard.

Each unique VNI corresponds to a unique Layer 2 segment. Hosts in VNI 10001 cannot directly communicate with hosts in VNI 10002 (Layer 2 isolation); they would need a router (inter-VXLAN routing).

flowchart LR
  subgraph V10001["VNI 10001 (tenant A)"]
    A1["VM A1<br/>10.0.1.10"]
    A2["VM A2<br/>10.0.1.11"]
  end
  subgraph V10002["VNI 10002 (tenant B)"]
    B1["VM B1<br/>10.0.2.10"]
    B2["VM B2<br/>10.0.2.11"]
  end
  subgraph V10003["VNI 10003 (tenant C)"]
    C1["VM C1<br/>10.0.3.10"]
  end

Each tenant has its own isolated Layer 2 segment. A VTEP can terminate several VNIs at once, one per tenant, but it does not route between them on the strength of the VNI: on receipt it reads the VNI out of the VXLAN header, decapsulates, and hands the inner frame to the VXLAN device configured for that VNI. That is a demultiplexing step followed by a switching decision. Moving traffic from one tenant to another is a separate and deliberate act of routing, and the next section shows where on the VyOS configuration tree that routing can happen — which is not on the VXLAN interface.

VNI assignment examples

A typical VNI assignment for a multi-tenant data center:

The hierarchical scheme:

• 10000-89999: tenant VNIs (sequentially assigned).
• 90000-90999: management VNIs (DC operations).
• 91000-91999: inter-DC VNIs (spans).

This scheme prevents collision across data centers; tenants are assigned from the tenant range (10000+), management from the management range (90000+), etc.

Configure multiple VNIs on a VTEP

A VTEP can have multiple VXLAN interfaces, each with a different VNI. Each of those interfaces is a Layer 2 device carrying Ethernet frames for its VNI, so each one belongs in a bridge alongside the port the tenant’s own hosts are attached to. The gateway address from the table above goes on that bridge:

configure
# VTEP local IP — one loopback, reused as the source for every VNI
set interfaces loopback lo address 192.0.2.10/32

# Tenant-facing trunk: one VLAN per tenant
set interfaces ethernet eth1 vif 101
set interfaces ethernet eth1 vif 102
set interfaces ethernet eth1 vif 103

# VXLAN for tenant A (VNI 10001) — no address on this interface
set interfaces vxlan vxlan10001 vni 10001
set interfaces vxlan vxlan10001 source-address 192.0.2.10
set interfaces vxlan vxlan10001 remote 192.0.2.20
set interfaces vxlan vxlan10001 mtu 1500

# VXLAN for tenant B (VNI 10002)
set interfaces vxlan vxlan10002 vni 10002
set interfaces vxlan vxlan10002 source-address 192.0.2.10
set interfaces vxlan vxlan10002 remote 192.0.2.20
set interfaces vxlan vxlan10002 mtu 1500

# VXLAN for tenant C (VNI 10003)
set interfaces vxlan vxlan10003 vni 10003
set interfaces vxlan vxlan10003 source-address 192.0.2.10
set interfaces vxlan vxlan10003 remote 192.0.2.20
set interfaces vxlan vxlan10003 mtu 1500

# One bridge per VNI. The VXLAN device and the tenant's access port are
# both ports on it, and the gateway address for the segment lives here.
set interfaces bridge br10001 member interface vxlan10001
set interfaces bridge br10001 member interface eth1.101
set interfaces bridge br10001 address 10.0.1.1/24

set interfaces bridge br10002 member interface vxlan10002
set interfaces bridge br10002 member interface eth1.102
set interfaces bridge br10002 address 10.0.2.1/24

set interfaces bridge br10003 member interface vxlan10003
set interfaces bridge br10003 member interface eth1.103
set interfaces bridge br10003 address 10.0.3.1/24

commit
save

The configuration:

• One loopback IP (192.0.2.10) used as the VTEP source address for all three VXLAN interfaces.
• Three VXLAN interfaces (vxlan10001, vxlan10002, vxlan10003), each carrying one VNI and each holding no IP address of its own.
• Three bridges, each joining one VXLAN device to one tenant access port and carrying that tenant’s gateway address.

The overlay MTU is set to 1500 explicitly. That is also the VyOS default for a VXLAN interface, and commit warns if you go below it — the encapsulation overhead is the underlay’s problem to absorb, not the tenant’s. Part XLIV-03 derives the 1550-byte underlay that makes a 1500-byte overlay possible.

There are no static routes in that block, and their absence is the point. set interfaces bridge br10001 address 10.0.1.1/24 installs the connected route for 10.0.1.0/24 via br10001 on its own. A static route aimed at vxlan10001 would be aimed at a bridge port, which has no IP configuration of its own to route from.

VNI design considerations

When designing a VNI namespace:

1. Number of VNIs needed. Each tenant, each management network, each inter-DC link requires a VNI. Add 10-20% headroom for future expansion.
2. Hierarchical scheme. Use ranges for different purposes (tenants, management, inter-DC).
3. Documentation. Maintain a VNI registry (which VNI is assigned to which tenant).
4. VTEP table size. Each VTEP maintains a forwarding table (FDB) for each VNI. With 10,000+ VNIs and many MACs per VNI, the table can be large. Use hardware with sufficient TCAM.
5. Multicast groups. Each VNI may need its own multicast group for flood-and-learn; allocate sufficient multicast groups.

A poor VNI design (sequential without documentation) leads to:

• Silent collisions (two data centers assigning the same VNI to different tenants).
• Operational confusion (which VNI is which tenant?).
• Hard-to-debug issues (the wrong tenant’s traffic in the wrong VNI).

Validation

# List all VXLAN interfaces
show interfaces vxlan
# All VXLAN interfaces with their VNIs

# Inspect a specific VXLAN interface
show interfaces vxlan vxlan10001
# Detailed state

# Verify the kernel's view
ip -d link show vxlan10001
# Shows the VXLAN-specific interface details (VNI, source IP, etc.)

# Bridge membership: each VNI's device and its access port on one bridge
show bridge
show interfaces bridge br10001
# The tenant gateway address must appear here and on nothing else

# Remote MACs learned over one VNI, with the VTEP behind each
bridge fdb show dev vxlan10001

# Test each tenant's connectivity
ping 10.0.1.10
ping 10.0.2.10
ping 10.0.3.10
# Each should succeed within its tenant

A clean validation: the VXLAN interfaces are up with the correct VNIs and no addresses of their own; each bridge lists one VXLAN device and one access port and carries the tenant gateway; the source IPs are correct; traffic flows between hosts in the same VNI; traffic crosses between VNIs only where somebody configured a router to carry it.

Production failure modes

VNI collision

Two data centers assign the same VNI to different tenants. The VXLAN encapsulation carries the inner frame from tenant A in data center 1 to tenant B in data center 2 (because both use VNI 10001). The traffic is misrouted.

Diagnostic: a host in one data center receives Layer 2 frames from a tenant it does not belong to.

Fix: redesign the VNI assignment with hierarchical ranges; document the assignment; ensure no overlap.

VTEP source-address changes

The VTEP source-address is a physical interface IP. If the interface is rerouted or fails over, the source-address changes. Remote VTEPs that have cached the old source-address cannot send traffic to the new IP.

Diagnostic: the tunnel works initially, then breaks after a failover.

Fix: use a loopback IP for the VTEP source-address. The loopback is always up; the source-address remains stable.

Tenant traffic crosses VNIs

A misconfiguration (VTEP configured with the wrong VNI for a tenant) causes traffic from tenant A to be delivered to tenant B. The two tenants see each other’s traffic.

Diagnostic: a host sees Layer 2 frames from a different tenant.

Fix: correct the VNI on the misconfigured VTEP; ensure each tenant’s VNI is unique and documented.

The tenant address was put on the VXLAN interface

set interfaces vxlan vxlan10001 address 10.0.1.1/24 is accepted at the CLI and rejected at commit while the interface is a bridge member. The usual response is to delete the bridge, at which point the command commits and the segment quietly stops being one: the VXLAN interface becomes a routed endpoint and the tenant’s access port has nothing to attach to.

Diagnostic: show bridge lists no bridge for that VNI, show interfaces vxlan vxlan10001 shows an address, and the tenant’s local hosts reach nothing across the overlay even though the two VTEPs can see each other.

Fix: put the address back on a bridge holding both the VXLAN device and the access port. Nothing about the tunnel changes; what changes is that there is a segment on each end of it.

VNI registry out of date

The VNI registry (which VNI is which tenant) is not maintained. New VNIs are assigned without consulting the registry. Collision is inevitable.

Fix: maintain the registry; check before assigning; automate VNI assignment if possible.

Rollback

# Enter configuration mode and write the running configuration to a
# file you can load back. `save` is a configuration-mode command that
# takes a path; operational mode has no `| save` pipe.
configure
save /config/pre-change-vxlan-vni-TICKET.conf

# Remove the segments, then the tunnels — the order matters
delete interfaces bridge br10001
delete interfaces bridge br10002
delete interfaces bridge br10003
delete interfaces vxlan vxlan10001
delete interfaces vxlan vxlan10002
delete interfaces vxlan vxlan10003

# Read the diff before committing anything
compare
commit

# Or restore a previous configuration
load /config/pre-change-vxlan-vni-TICKET.conf
commit
save

The rollback removes the bridges and the VXLAN interfaces; the VNIs are no longer used. Deleting interfaces vxlan vxlan10001 on its own, while br10001 still lists it as a member, is refused — Interface "vxlan10001" cannot be deleted as it is a member of bridge "br10001"! — so release the membership, or delete the whole bridge, in the same commit.

Production discipline

Cross-course references

• Part XLIV-01 (XLIV-VyOS-VXLAN / concept) covers the VXLAN protocol.
• Part XLIV-03 (XLIV-VyOS-VXLAN / underlay) covers the underlay requirements.
• Part XLIV-04 (XLIV-VyOS-VXLAN / BGP EVPN) covers BGP EVPN as the control plane for VNI management.

Quiz