VRF concept — L3VPN, the kernel vrf driver, and how VyOS implements routing tables per VRF

A VRF is a routable slice of a single router. To packets inside the slice, the router looks identical to a stand-alone device with its own routing table, its own connected networks, and its own view of every external destination. To packets in a different slice, that router does not even exist. The Linux kernel implements this with the vrf driver (an l3mdev master), and VyOS 1.5 LTS sits FRR’s per-VRF routing instances on top of the kernel tables so the control plane matches what the data plane is actually doing.

This lesson is the foundation for the rest of Part XV. xv-02 covers the configuration commands. xv-03 covers running OSPF and BGP inside a VRF and moving routes across the VRF boundary on purpose. xv-04 covers IPv6 inside a VRF. xv-05 covers the operational command set the engineer uses when a VRF is misbehaving. xv-06 covers the anti-patterns that turn a multi-VRF estate into an incident factory.

What a VRF actually is

In operational terms, a VRF is:

• A routing table that is independent of the main routing table.
• A set of interfaces whose packets are looked up in that table.
• A control-plane context in which routing protocols (OSPF, BGP, IS-IS, static) install routes into that table and not the main table.

A packet that arrives on an interface bound to VRF CUST-A is looked up only in CUST-A’s routing table. The kernel does not consult the main table for that packet, even when CUST-A’s table has no route for the destination. The packet is rejected as unreachable, not silently forwarded out the global default route. The section “How traffic actually flows” below shows the exact rule chain that produces that behaviour, because it is a set of ip rule entries VyOS installs rather than an intrinsic property of the lookup.

flowchart TB
  subgraph ROUTER["VyOS router (one device, multiple VRFs)"]
    direction TB
    subgraph VA["VRF CUST-A"]
      TA["table 1001"]
      IA["eth1"]
      SA["OSPF / BGP / static"]
      TA --- IA
      TA --- SA
    end
    subgraph VB["VRF CUST-B"]
      TB["table 1002"]
      IB["eth2"]
      SB["OSPF / BGP / static"]
      TB --- IB
      TB --- SB
    end
    subgraph GL["default VRF (main table)"]
      TG["table 254"]
      IG["eth0 (uplink)"]
      SG["default route / eBGP to ISP"]
      TG --- IG
      TG --- SG
    end
  end
  HA["Host A 10.0.1.0/24"] --> IA
  HB["Host B 10.0.2.0/24"] --> IB
  ISP["ISP"] <--> IG

A VRF is not a virtual router. A virtual router separates the control plane AND the data plane AND the management plane — it consumes its own CPU, its own memory, its own configuration, and its own copy of the OSPF process tree. A VRF separates the routing table. The kernel vrf driver multiplexes the data plane across VRFs in software; FRR runs one zebra process that maintains a separate RIB per VRF.

The distinction matters for resource planning. A site router can run dozens of VRFs inside the memory footprint of one FRR process. The same router with a per-tenant virtualisation plan would consume one VM or one container per tenant, which is a different order of cost in CPU and memory.

L3VPN and where VRFs come from

The L3VPN (Layer 3 Virtual Private Network) is the architectural pattern that made VRFs operationally important. RFC 4364 codifies it for service providers: a provider offers a “VPN” service to many customers, where each customer’s sites appear to be on a private network even though they share the provider’s MPLS backbone.

The primitives are:

• Customer Edge (CE) router — the customer-side router at each site.
• Provider Edge (PE) router — the provider-side router that carries the customer’s routes in a VRF.
• VRF — one VRF per customer on each PE.
• Route Distinguisher (RD) — an 8-byte value prepended to the customer’s IPv4 or IPv6 routes so two customers who both use 10.0.0.0/8 produce distinct VPNv4 / VPNv6 NLRI in BGP.
• Route Target (RT) — an extended community that controls which VRFs import which routes out of the VPN RIB.
• MP-BGP VPNv4 / VPNv6 address family — the BGP address family that carries customer routes between PEs across the core.

On VyOS those last two are not free-standing commands. They live under the per-VRF BGP address family, and xv-03 configures them in full:

set vrf name CUST-A protocols bgp address-family ipv4-unicast rd vpn export '65000:1001'
set vrf name CUST-A protocols bgp address-family ipv4-unicast route-target vpn both '65000:1001'
set vrf name CUST-A protocols bgp address-family ipv4-unicast export vpn
set vrf name CUST-A protocols bgp address-family ipv4-unicast import vpn

A VyOS router at the customer edge is usually not an MPLS L3VPN PE. It is, however, routinely expected to separate tenants on a single device: the same kernel model, the same FRR per-VRF routing instance, with no MPLS, no RD and no RT. Read the RD/RT vocabulary as the provider dialect of the same idea, and do not configure it when there is no VPN RIB to export into.

flowchart LR
  subgraph PE1["PE-1"]
    PA["VRF CUST-A<br/>rd 65000:1001"]
    PB["VRF CUST-B<br/>rd 65000:1002"]
  end
  subgraph CORE["Provider core"]
    MPLS["label switching only<br/>no per-customer state"]
  end
  subgraph PE2["PE-2"]
    PA2["VRF CUST-A<br/>rt import 65000:1001"]
    PB2["VRF CUST-B<br/>rt import 65000:1002"]
  end
  PA -->|"MP-BGP VPNv4"| MPLS
  PB -->|"MP-BGP VPNv4"| MPLS
  MPLS --> PA2
  MPLS --> PB2

The Linux kernel vrf driver

The kernel implements VRFs as an l3mdev master. The model is:

• A master device of type vrf, created with ip link add NAME type vrf table ID.
• Slave interfaces enslaved to that master. A slave’s packets carry the master’s table id into the FIB lookup.
• Routes installed with that table id are the VRF’s routes.

VyOS creates the master device with the VRF name as the interface name. There is no prefix and no decoration: set vrf name CUST-A table 1001 produces a Linux interface literally called CUST-A.

# What the commit performs, at the kernel level
ip link add CUST-A type vrf table 1001
ip link set dev eth1 master CUST-A

That naming is why the VyOS validator constrains VRF names the way it does. A VRF name must be 15 characters or fewer (the kernel’s IFNAMSIZ limit minus the terminator), must consist of letters, digits, - and _, and must not begin with a string the box would read as an interface type — eth, bond, br, dum, tun, vti, wg, lo and the rest. set vrf name eth-mgmt is rejected for that reason, not for a policy reason.

A separate list of words is reserved outright, because they are ip command keywords or the name of the global instance: add, all, broadcast, default, delete, dev, down, get, inet, link, mtu, type, up, vrf. default matters most of the three dozen rules in this lesson — it is the name FRR uses for the global routing instance, and it is the value you will pass to a route-leaking command in xv-03 to mean “the main table”.

The name is a label; the kernel’s identity for the VRF is the table id. VyOS accepts a table id in the range 100 to 65535, requires it on every VRF, rejects 254 explicitly because that is the main table, and refuses to change an id once the interface exists — the kernel does not allow a live vrf device to be re-tabled, so a change means deleting and recreating the VRF.

set vrf name CUST-A table 1001
set vrf name CUST-A description "Customer A tenant"
set interfaces ethernet eth1 vrf CUST-A
set interfaces ethernet eth1 address 10.1.0.1/24

The set vrf name CUST-A table 1001 line creates the CUST-A l3mdev master with table 1001. The set interfaces ethernet eth1 vrf CUST-A line enslaves eth1 to it. The address line then installs its connected route in table 1001 rather than table 254, because the kernel derives the table from the slave’s master.

How VyOS wires a vrf name to an FRR routing instance

The full chain when the operator commits set vrf name CUST-A table 1001 and set interfaces ethernet eth1 vrf CUST-A:

1. The vrf node commits early — it carries priority 11, ahead of every interface node — so the master device exists before any interface is asked to join it.
2. The commit script runs ip link add CUST-A type vrf table 1001, adds 127.0.0.1/8 and ::1/128 to the VRF device so services bound to the loopback are reachable inside the VRF, applies the description as the interface alias, and brings the device up.
3. It adds an nftables element mapping the VRF name to its table id in inet vrf_zones ct_iface_map, which gives each VRF its own conntrack zone. That is what allows two VRFs to use overlapping address space without their connection-tracking state colliding.
4. The interface node then enslaves eth1, and the connected route for 10.1.0.1/24 lands in table 1001.
5. FRR receives the interface’s VRF membership from zebra and maintains a CUST-A routing instance. Anything the operator configures under set vrf name CUST-A protocols ... is rendered inside FRR’s vrf CUST-A block and installs into table 1001.

No prefix from CUST-A reaches the main table unless somebody configures a leak on purpose. xv-03 covers how that is done and, more usefully, what it looks like when a leak is configured and quietly does nothing.

Why the engineering boundary exists

VRFs are a primitive rather than a convenience because they enforce isolation without depending on operator discipline. An engineer cannot accidentally route a packet from eth1 (bound to CUST-A) into CUST-B’s subnet, because the lookup for an eth1 packet is directed at table 1001 by the l3mdev rule and stopped by the l3mdev unreachable rule if it misses. The boundary is enforced by the rule chain, not by somebody reading a config carefully.

sequenceDiagram
  autonumber
  participant H as Host in 10.1.0.0/24
  participant E as eth1 (slave of CUST-A)
  participant K as Linux routing policy
  participant V as table 1001
  participant Z as zebra (FRR)
  participant F as OSPF instance in CUST-A

  H->>E: packet 10.1.0.50 to 10.1.0.99
  E->>K: skb on eth1, master CUST-A
  K->>V: rule 1000 l3mdev, lookup table 1001
  V-->>K: connected route on eth1
  K->>E: forward
  E->>H: deliver
  F-->>Z: OSPF Hello / LSA inside vrf CUST-A
  Z->>V: install route in table 1001

Two separate physical routers enforce the same isolation through wiring: a packet on router A’s interface cannot reach router B’s interface. A VRF delivers the same guarantee on one device, with the l3mdev rules as the enforcement point.

How traffic actually flows

The decision path for a packet arriving on an interface, given the rule chain from the “Under the hood” box above:

flowchart TD
  A["Packet arrives on an interface"] --> B{"Interface enslaved to a VRF?"}
  B -- "master CUST-A" --> C["rule 1000 l3mdev: look up table 1001"]
  B -- "no master" --> M["rule 32765 local, then 32766 main"]
  C --> D{"Hit in table 1001?"}
  D -- yes --> F["Forward per table 1001"]
  D -- no --> E["rule 2000 l3mdev unreachable"]
  E --> G["Reject: unreachable"]
  M --> H{"Hit in local or main?"}
  H -- yes --> I["Forward per the main table"]
  H -- no --> G

The packet in a VRF context is never offered to the main table, because rule 2000 terminates the walk before rule 32766 is reached. There is no implicit “fall back to the global table” — building one is a deliberate act, and xv-03 shows the two supported ways to do it.

How the result is validated

# VyOS view
show vrf
show vrf CUST-A
show vrf CUST-A processes
show interfaces ethernet eth1

# Kernel view
ip vrf show
ip -d link show dev CUST-A
ip link show master CUST-A
ip route show table 1001
ip -6 route show table 1001
ip -4 rule show

# FRR view
vtysh -c 'show vrf'
vtysh -c 'show ip route vrf CUST-A'
vtysh -c 'show ip ospf vrf CUST-A neighbor'

The operational-mode form is show vrf CUST-A, with the VRF name as a positional value. There is no name keyword in the operational tree even though the configuration tree has one — show vrf name CUST-A is not a command.

A working VRF has:

• The master device present and up.
• Every intended slave interface reporting master CUST-A.
• The VRF listed by vtysh -c 'show vrf' as an FRR routing instance.
• Connected routes present in the expected table id and absent from table 254.
• Where routing protocols are configured, adjacencies visible only in the per-VRF view.

How it fails

The failure modes an engineer must recognise:

• The interface is not enslaved. The configuration says vrf CUST-A but ip -d link show dev eth1 reports no master. Traffic on the interface is routed by the main table. The isolation is a fiction, and nothing in the VyOS view says so.
• The table id is already taken. Two VRFs configured with the same table value fail validation with table id is not unique. The commit is rejected as a whole, so the box is left on the previous configuration.
• Somebody tries to re-table a live VRF. Changing table on an existing VRF is rejected: the id cannot be modified once the kernel device exists. The only route is to delete the VRF and add it back, which drops every interface binding and every route inside it.
• A routing protocol is configured at the top level. set protocols ospf area 0 network 10.1.0.0/24 outside vrf name CUST-A builds the adjacency on eth1 but installs into the main table. Packets still flow — the connected route is in table 1001 either way — so the symptom is not loss of connectivity but loss of isolation. xv-03 covers this in detail.
• A VRF is deleted while something still references it. VyOS refuses to remove a VRF that still has member interfaces, static routes, or policy routes, and refuses to remove one that another VRF imports from. The commit fails with the reference named; delete the reference in the same commit.
• Overlapping address space without the conntrack zone. Two VRFs using the same prefix rely on the per-VRF conntrack zone VyOS installs. If connection tracking is not in play for a flow, nothing separates two identical 5-tuples in different VRFs — which is a reason to keep NAT and stateful firewalling explicitly VRF-aware rather than assuming it.

Rollback

# Detach a single interface from a VRF
delete interfaces ethernet eth1 vrf

# Remove a VRF - the references must go in the same commit
delete interfaces ethernet eth1 vrf
delete vrf name CUST-A
commit

A safe rollback sequence:

1. compare to confirm the candidate diff is exactly the inverse of the forward change.
2. commit to apply the rollback.
3. Verify with ip -d link show that the master device is gone and no interface still reports it.
4. Verify with ip -4 rule show that the rule chain matches the box’s new state — with the last VRF removed, local returns to priority 0 and the l3mdev rules disappear.
5. save only after the rollback is confirmed working.

Production discipline

Cross-course references

• The Linux course’s V-Linux-NetConfig and XXI-Linux-NetAdvanced cover the same l3mdev primitives from the host perspective — the vrf master device, the table argument, and ip vrf exec.
• The OPNsense course covers a different isolation model (VLAN + interface + per-VLAN firewall) on FreeBSD; the operational goal is similar, the primitives are not.
• The VyOS lessons vyos-xiv-01-routing-table-concept through vyos-xiv-06-table-anti-patterns cover the multi-table primitives that a per-VRF table id shares a namespace with.
• The BGP course’s XXVII-BGP-BestPath and XXX-BGP-RouteReflectors discuss the MP-BGP VPNv4 / VPNv6 family that consumes the VRF concept in an L3VPN environment.

Quiz