VyOS XV · VRFsVRF

VRF troubleshooting — RIB versus FIB, `ip vrf exec`, and the leak that installs nothing

Advanced⏱ ~22 minip vrf exec VRFNAME ping DSTip vrf exec VRFNAME traceroute DSTip vrf exec VRFNAME ss -tunapip route show vrf VRFNAMEip -d link show SLAVEIFshow vrfshow ip route vrf VRFNAMEshow bgp vrf VRFNAME ipv4 unicast summarytcpdump -ni SLAVEIFvyos

What you'll learn

Separate FRR's RIB from the kernel FIB and know which command reads which, per VRF
Run reachability and socket diagnostics inside a VRF with `ip vrf exec`, and explain what that actually does to the process
Verify the four places VRF state lives — configuration, kernel, FRR, peer — in the order that localises a fault fastest
Recognise the signature of a route leak that installs nothing, and the signature of a leak that is missing its return direction
Express a per-VRF firewall policy on VyOS 1.5, where the firewall matches interfaces rather than VRFs

Prerequisites

Verified against VyOS 1.5.x LTS (circinus) · VyOS 1.4.x (sagitta) — legacy · FRRouting 10.x (VyOS 1.5) · Linux kernel 6.6 LTS (VyOS 1.5 base) · strongSwan 5.9.x (IPsec) · WireGuard 1.0.x (kernel module + userspace tooling)

Not yet marked complete on this device.

A VRF-localised routing failure has a distinctive shape. A packet enters the router on the right interface, the configuration reads correctly, every show command an operator reaches for by habit looks healthy — and the traffic goes nowhere. The reason it is hard is that VRF state lives in four places at once, and the operational commands most people know only read two of them.

This lesson is the playbook: the four sources of truth, the order to check them in, the one distinction that resolves most of these incidents on its own, and the failure signatures worth memorising.

The distinction that resolves most VRF incidents

Before any of the command sequences: FRR’s RIB and the kernel’s FIB are different things, and there are separate commands for each.

show ip route vrf CUST-A reads FRR’s RIB. It is a statement of what FRR believes and intends.
ip route show vrf CUST-A reads the kernel’s table 1001. It is what actually forwards packets.

In a healthy VRF the two agree, which is why the difference rarely comes up — and why it ambushes people when it matters. A route can sit in the RIB indefinitely without ever being offered to the kernel: FRR keeps routes whose next-hop it cannot resolve, marks them inactive, and never installs them. show ip route prints them. Traffic never sees them.

flowchart LR
  CFG["VyOS configuration<br/>set vrf name CUST-A ..."] --> FRR
  FRR["FRR RIB · vrf CUST-A<br/>show ip route vrf CUST-A"] -- "zebra installs<br/>only resolved routes" --> FIB
  FIB["kernel table 1001<br/>ip route show vrf CUST-A"] --> PKT["forwarded packets"]
  FRR -. "unresolved next-hop:<br/>stays here, inactive" .-> DEAD["never installed"]

Every diagnostic in this lesson eventually lands on this question: is the route in the RIB only, or in the FIB too? Ask it early.

Read-only / Safethe kernel's per-VRF forwarding table— Three routes the kernel will actually use: a connected route, a leaked route whose egress interface eth9 belongs to another VRF, and a BGP-learned route. Anything FRR knows about but is not here does not forward traffic.

$ ip route show vrf CUST-A

10.1.0.0/24 dev eth1 proto kernel scope link src 10.1.0.1
10.250.0.0/16 via 10.99.0.254 dev eth9 proto 196 metric 20
192.168.0.0/16 via 10.1.0.254 dev eth1 proto bgp metric 20

Illustrative output

The four sources of truth

A working VRF has four views that agree:

The VyOS configuration tree — what the operator typed.
The kernel — the vrf-NAME master device, the enslaved interfaces under it, and the routes in its table.
FRR — the per-VRF routing instance: its RIB, and the state of OSPF, BGP and static inside it.
The adjacent peer — the far end’s view of the same session.

Check them in that order. Each step is cheap, and each one rules out everything downstream of it. Jumping straight to the peer because the symptom looks like a peering problem is how a two-minute configuration-placement bug becomes an afternoon.

flowchart TB
  subgraph S1["1 · configuration"]
    C1["set vrf name CUST-A table 1001"]
    C2["set interfaces ethernet eth1 vrf CUST-A"]
    C3["set vrf name CUST-A protocols bgp system-as 65000"]
  end
  subgraph S2["2 · kernel"]
    K1["vrf-CUST-A master · table 1001 · UP"]
    K2["eth1 · master vrf-CUST-A"]
    K3["table 1001 holds the connected routes"]
  end
  subgraph S3["3 · FRR"]
    F1["vrf CUST-A instance exists"]
    F2["bgpd · peer Established in this VRF"]
    F3["zebra · RIB and FIB agree"]
  end
  subgraph S4["4 · peer"]
    P1["far end sees the session"]
    P2["far end advertises the prefixes"]
  end
  S1 --> S2 --> S3 --> S4

Step 1 — the configuration tree

show configuration commands | match "vrf"
show configuration commands | match "protocols"

The first tells you which interfaces claim a VRF and which VRFs exist. The second is the one that catches the commonest defect of all: a protocols stanza that sits at the top of the tree instead of under vrf name CUST-A.

That defect has a specific history. An engineer converts a working non-VRF router to a VRF design by adding vrf CUST-A to the interfaces — and leaves the existing protocols ospf or protocols bgp block where it was. The adjacency comes up, because the interface is still reachable. The routes install in table 254. The customer’s traffic, which is scoped to table 1001, never sees them. Nothing errors.

# This should return nothing on a router whose routing is all per-VRF
show configuration commands | match "^set protocols"

Step 2 — the kernel

Read-only / SafeVyOS view of the VRF masters— Each VRF, its master device state, and the interfaces enslaved to it. A VRF listed with no interfaces is a VRF nothing can enter.

$ show vrf

VRF name          state     mac address        flags                     interfaces
--------          -----     -----------        -----                     ----------
CUST-A            up        52:54:00:11:11:11  noarp,master,up,lower_up  eth1
SHARED            up        52:54:00:99:99:99  noarp,master,up,lower_up  eth9

Illustrative output

Then the two questions that show vrf does not answer directly:

# Is the interface really enslaved? (the configuration tree is not evidence)
ip -d link show eth1

# Does the per-VRF table hold what it should?
ip route show vrf CUST-A
ip -6 route show vrf CUST-A

ip -d link show eth1 prints master vrf-CUST-A on the second line when the binding took. If it does not, the configuration and the kernel disagree.

`ip link` surgery is a probe, not a repair

When the kernel and the configuration disagree it is tempting to fix the kernel: ip link set eth1 master vrf-CUST-A, and the symptom goes away.

It is a legitimate way to confirm a hypothesis. It is not a repair. VyOS owns that state; your change is not in the configuration, so it does not survive a reboot, and the next commit that touches the interface reconciles it back. You have converted an outage into an outage that will recur at an unpredictable time, with a router whose running state no longer matches its saved configuration.

The durable repair goes through the configuration:

configure
delete interfaces ethernet eth1 vrf
commit
set interfaces ethernet eth1 vrf CUST-A
commit
save

Then re-verify with ip -d link show eth1. If a commit genuinely failed to apply, the commit output said so — check show log and journalctl -u vyos-configd for the commit-script error rather than papering over it.

If the master device is missing entirely, ip -d link show vrf-CUST-A reports Device "vrf-CUST-A" does not exist. — that is a VRF that was never created, not one that is down. Look for a failed commit, not for a link to bring up.

Step 3 — FRR

FRR keeps a routing instance per VRF. Its own view of which ones exist is the first thing to read:

Read-only / SafeFRR view of the VRFs— FRR's own list, with the interface index of each master device and the table id it maps to. A VRF present in `show vrf` but absent here means zebra never learned about it.

$ vtysh -c 'show vrf'

vrf CUST-A id 5 table 1001
vrf SHARED id 7 table 9999
vrf default id 0 table 254

Illustrative output

Then the per-VRF protocol state:

show ip route vrf CUST-A
show ip route vrf CUST-A 192.168.0.0/16
show bgp vrf CUST-A ipv4 unicast summary
show bgp vrf CUST-A ipv4 unicast
vtysh -c 'show ip ospf vrf CUST-A neighbor'
vtysh -c 'show ipv6 ospf6 vrf CUST-A neighbor'

What the common findings mean:

The VRF is missing from vtysh -c 'show vrf'. Zebra has no instance for it. The configuration exists and the kernel device may exist, but FRR was not told. Look for a commit-script error.
show ip route vrf CUST-A is empty or thin. Either the routing protocol is not running inside the VRF — the placement bug from step 1 — or its session is not up.
A peer is absent from show bgp vrf CUST-A ipv4 unicast summary. The neighbour has no address-family ipv4-unicast node, so VyOS never rendered an activate line for it into FRR. Note the shape: an unactivated peer is missing from the family’s summary, not present in it with a zero count. A peer that is listed with 0 did negotiate the family and either received nothing or had everything filtered.
A route is in show ip route vrf CUST-A but not in ip route show vrf CUST-A. Its next-hop did not resolve. This is the leaking failure, and it has its own section below.

VyOS does not expose an `activate` leaf

FRR activates a peer for an address family with a literal neighbor X activate line. VyOS has no matching CLI leaf — creating set vrf name CUST-A protocols bgp neighbor X address-family ipv4-unicast is the activation, and VyOS emits the activate line for you.

So do not go looking for a missing activate in the VyOS configuration; look for a missing address-family node. To see what was actually handed to FRR:

vtysh -c 'show running-config' | sed -n '/^router bgp 65000 vrf CUST-A/,/^!/p'

Step 4 — the peer, tested from inside the VRF

Reachability tests are only evidence if they were scoped to the VRF. An unscoped ping from the router uses the global table and can easily reach a completely different host that happens to share the address.

ip vrf exec CUST-A ping 10.1.0.254
ip vrf exec CUST-A traceroute 192.168.0.5
ip vrf exec CUST-A ss -tunap

VyOS also wraps the common ones, so ping 10.1.0.254 vrf CUST-A does the same job for that command. ip vrf exec is the general form: it works for any program, including ones VyOS never wrapped.

What `ip vrf exec` actually does to a process

It is worth being precise here, because the mental model people usually carry — “it enters a namespace” — is wrong, and the wrong model predicts the wrong failures.

ip vrf exec NAME cmd does not use network namespaces and does not call setns(). VyOS’s VRFs are l3mdev masters inside a single network namespace; there is no separate namespace to enter. What ip vrf exec does instead is put the process into a cgroup v2 with a small eBPF program attached at socket-creation time. Every socket the process opens from then on is bound to the VRF master device, as though the program had called setsockopt(SO_BINDTODEVICE, "vrf-CUST-A") itself. Route lookups on such a socket are scoped to the master’s table.

Three consequences follow directly from that mechanism, and all three are things operators hit:

It needs cgroup v2 mounted and the privilege to create a cgroup. On a system without it, ip vrf exec fails outright rather than silently running in the global table.
It only affects sockets the process creates after it is applied. A daemon that already holds an open socket is unaffected by running a control tool under ip vrf exec.
It binds sockets to the VRF; it does not move interfaces or change routing. Anything that bypasses the socket layer — a raw packet tool binding to an interface by name, for instance — is unaffected by it and needs the interface named directly.

ip netns exec is a genuinely different tool operating on full network namespaces. The command lines look alike; the mechanisms have nothing in common. Reaching for ip netns exec on a VyOS VRF finds nothing to enter.

The leak that installs nothing

This is the failure that route leaking between VRFs produces most often, and it is worth recognising on sight because every surface an operator normally trusts says the configuration is fine.

Leaking a prefix from one VRF into another means telling FRR to resolve the next-hop in a different VRF than the one that holds the route. On VyOS that is the vrf leaf under the next-hop; it renders FRR’s nexthop-vrf:

# Correct: resolve 10.99.0.254 in SHARED, install the result in CUST-A
set vrf name CUST-A protocols static route 10.250.0.0/16 next-hop 10.99.0.254 vrf SHARED

Omit that leaf and the command still commits without complaint:

# Commits cleanly. Installs nothing.
set vrf name CUST-A protocols static route 10.250.0.0/16 next-hop 10.99.0.254

FRR now tries to resolve 10.99.0.254 inside CUST-A’s own table, where the address does not exist. It keeps the route in the RIB, marks it unusable, and never hands it to the kernel.

Read-only / Safea leak that did not resolve— Read the flags, not the presence of the entry. There is no `best` on the second line and no `*` on the next-hop, and the next-hop is marked inactive. This route exists only in FRR's RIB.

$ show ip route vrf CUST-A 10.250.0.0/16

Routing entry for 10.250.0.0/16
Known via "static", distance 1, metric 0, vrf CUST-A
Last update 00:04:52 ago
  10.99.0.254 inactive

Illustrative output

Compare with the same route when the vrf leaf is present:

Read-only / Safea leak that resolved— `best` on the second line, a `*` on the next-hop, an egress interface, and the VRF the next-hop was resolved in printed in parentheses. That parenthesised VRF is the signature of a working leak.

$ show ip route vrf CUST-A 10.250.0.0/16

Routing entry for 10.250.0.0/16
Known via "static", distance 1, metric 0, vrf CUST-A, best
Last update 00:00:18 ago
* 10.99.0.254, via eth9 (vrf SHARED), weight 1

Illustrative output

And the check that settles it either way:

ip route show vrf CUST-A

If the prefix is not in that output, it does not forward traffic, whatever show ip route said.

Three leak failures, three different signatures

Learn to tell these apart before you touch the configuration; they have different fixes and two of them look identical from a customer ticket.

The leak installs nothing. show ip route vrf CUST-A has the prefix with inactive against the next-hop; ip route show vrf CUST-A does not have it. The vrf leaf is missing from the next-hop, or the next-hop does not exist in the VRF you named. Fix the statement.
The leak works one way. Both tables have their route, ip vrf exec CUST-A ping reaches the shared service, and the service cannot answer. The return leak in the other VRF was never configured. Leaks are directional and each direction is a separate statement — a fact that survives no amount of staring at the forward direction.
The leak resolves and the packets still die. Both directions are in both kernel tables and traffic is dropped anyway. Now it is the firewall or connection tracking: the request egresses one interface and the reply arrives on an interface in a different VRF, so a stateful rule set that was written per interface never sees a matching flow. conntrack -L and a tcpdump on each interface separate this from the routing cases.

The order matters. Diagnosing 3 while the real defect is 1 is how people end up adding permissive firewall rules to a router that was never dropping anything.

The firewall, and what VyOS does not give you

A recurring wish is a firewall rule that matches “traffic in VRF CUST-A”. VyOS does not expose that. The firewall matches interfaces, interface groups, addresses and ports — and VRF membership is a property of an interface, not something the rule set can select on directly.

The supported way to express a per-VRF policy is therefore to name the VRF’s interfaces, and to keep that list in one place so it does not drift as the VRF gains members:

configure
# One group per VRF, maintained alongside the VRF itself
set firewall group interface-group CUST-A-IF interface eth1
set firewall group interface-group CUST-A-IF interface eth1.100

set firewall ipv4 name CUST-A-FWD default-action drop
set firewall ipv4 name CUST-A-FWD rule 10 action accept
set firewall ipv4 name CUST-A-FWD rule 10 state established
set firewall ipv4 name CUST-A-FWD rule 20 action accept
set firewall ipv4 name CUST-A-FWD rule 20 state related

set firewall ipv4 forward filter rule 100 inbound-interface group CUST-A-IF
set firewall ipv4 forward filter rule 100 action jump
set firewall ipv4 forward filter rule 100 jump-target CUST-A-FWD
commit
save

Two things to hold on to:

The rule set is reached by a jump from the base chain (forward filter for transit traffic, input filter for traffic to the router itself). The per-interface firewall in / out bindings that older material uses were removed in 1.4.
The established/related rules are not boilerplate. A stateful policy that permits the request and forgets the reply is the commonest way to produce a VRF that “routes but does not work”, and it is indistinguishable from a routing fault until you look at conntrack -L.

The inspection commands:

show firewall
show firewall ipv4 name CUST-A-FWD
show firewall group
conntrack -L

The full per-VRF command set

# Configuration
show configuration commands | match "vrf"
show configuration commands | match "^set protocols"

# Kernel
show vrf
ip -d link show vrf-CUST-A
ip -d link show eth1
ip route show vrf CUST-A
ip -6 route show vrf CUST-A
ip neigh show dev eth1
ip -6 neigh show dev eth1

# FRR
vtysh -c 'show vrf'
show ip route vrf CUST-A
show ip route vrf CUST-A 10.250.0.0/16
show ipv6 route vrf CUST-A
show bgp vrf CUST-A ipv4 unicast summary
show bgp vrf CUST-A ipv4 unicast
vtysh -c 'show ip ospf vrf CUST-A neighbor'
vtysh -c 'show running-config'

# Reachability, scoped to the VRF
ip vrf exec CUST-A ping 10.1.0.254
ip vrf exec CUST-A traceroute 192.168.0.5
ip vrf exec CUST-A ss -tunap

# Firewall and state
show firewall ipv4 name CUST-A-FWD
conntrack -L

# The wire
tcpdump -ni eth1 host 10.1.0.254
tcpdump -ni eth9 host 10.99.0.254

The captures are the only commands in that list that prove bytes moved. Everything above them proves intent. When the intent looks right and the customer is still down, go to the wire — on both interfaces, because a leaked flow uses two.

The diagnostic walk

flowchart TD
  S["Customer reports CUST-A broken"] --> C["Is any protocols stanza outside vrf name?"]
  C -- yes --> CFX["Move it under vrf name · commit · re-check"]
  C -- no --> K["show vrf and ip -d link: master up, interface enslaved?"]
  K -- no --> KFX["Repair through the configuration, not with ip link"]
  K -- yes --> R["Is the prefix in ip route show vrf CUST-A?"]
  R -- "in RIB only" --> RFX["Next-hop unresolved: add or correct the vrf leaf on the leak"]
  R -- "not in RIB either" --> P["Peer or protocol: session state in this VRF?"]
  P -- "peer absent from family summary" --> PFX["Add the address-family node under the neighbour"]
  P -- "peer present, 0 prefixes" --> PF2["Peer sent nothing, or inbound policy filtered it"]
  R -- yes --> T["ip vrf exec CUST-A ping the next-hop"]
  T -- fails --> W["tcpdump both interfaces: are the bytes there?"]
  T -- succeeds --> F["Return direction and firewall state"]
  W --> F
  F --> FFX["Missing return leak, or a stateful rule set that never sees the reply"]

Rollback

Troubleshooting changes need the same discipline as planned ones — more, because they are made under pressure.

# Undo a leak you added while testing
delete vrf name CUST-A protocols static route 10.250.0.0/16
commit
save

# Move a misplaced routing stanza (both halves in one commit)
delete protocols bgp
set vrf name CUST-A protocols bgp system-as 65000
commit
save

# Release an interface from a VRF
delete interfaces ethernet eth1 vrf
commit
save

The sequence, every time:

show configuration commands captured off-box before the first change.
compare before every commit, so you know the candidate contains what you think and nothing else.
commit-confirm 10 — a whole number of minutes — for anything touching a path your own session traverses. confirm makes it permanent; doing nothing rolls it back.
Verify with both show ip route vrf and ip route show vrf, not one of them.
save only after the verification is clean. An unsaved fix is a fix that ends at the next reboot.

Production discipline

VRF incident checklist

Configuration — show configuration commands | match "vrf", and confirm no protocols stanza sits outside a vrf name.
Kernel — show vrf, ip -d link show SLAVEIF for master, ip route show vrf VRFNAME for the FIB.
FRR — vtysh -c 'show vrf', show ip route vrf VRFNAME, per-VRF protocol summaries.
RIB against FIB — for the specific prefix that is failing, compare show ip route vrf VRFNAME PREFIX with ip route show vrf VRFNAME. Note whether the next-hop says inactive.
Leaks — confirm the vrf leaf on the next-hop, and confirm the return direction exists as its own statement.
Reachability — ip vrf exec VRFNAME ping, never an unscoped ping.
Firewall and state — show firewall ipv4 name ..., conntrack -L, with attention to established and related.
The wire — tcpdump on both the ingress and the egress interface, because a leaked flow spans two VRFs.
Apply the fix in configure, with compare and commit-confirm.
Re-verify and capture the evidence for the post-incident review, including which of the four views had been lying.

Cross-course references

The Linux course’s XXII-Linux-NetTroubleshoot covers ip route get, ip neigh and namespace tooling; ip vrf exec is the l3mdev-scoped sibling of those primitives.
vyos-xiv-06-table-anti-patterns covers the same RIB/FIB reasoning for policy routing outside VRFs.
vyos-xv-04-vrf-ipv6 covers the IPv6-specific diagnostics — ND filtering, link-local next-hops, route6 leaks.
vyos-xvi-05-leaking-troubleshoot takes the leaking failures in this lesson further, including BGP-based leaking.
The BGP course’s XXXI-BGPTroubleshoot and the OSPF course’s XXII-OSPFTroubleshoot cover the protocol diagnostics that apply unchanged inside a VRF once you have added the vrf selector.

Quiz

Knowledge check · 4 questions

Q1. A leaked prefix appears in `show ip route vrf CUST-A` but traffic to it is dropped. Which single command best distinguishes 'FRR intends this route' from 'the kernel will forward on this route'?

ip route show vrf CUST-A show ip route vrf CUST-A summary, because the summary counts only installed routes and the detailed view counts every candidate FRR has considered vtysh -c 'show vrf', which lists each VRF with its table id and would show a mismatch show configuration commands | match vrf
Q2. FRR will keep a static route in the RIB, and print it in `show ip route vrf CUST-A`, even when it has never installed that route in the kernel because the next-hop did not resolve.

True False
Q3. A shared-services leak into a tenant VRF is confirmed installed in the kernel, and a host on the tenant network still gets no reply from the shared DNS server. The forward direction is verifiably correct. Where is the fault, and how do you prove it rather than assume it?

The topology: - VRF SHARED, table 9999, eth9 enslaved, 10.99.0.1/24. The shared services live in 10.250.0.0/16 behind the services router at 10.99.0.254. - VRF CUST-A, table 1001, eth1 enslaved, 10.1.0.1/24. The tenant CE is at 10.1.0.254 and the tenant's hosts are in 10.1.0.0/16 behind it. - set vrf name CUST-A protocols static route 10.250.0.0/16 next-hop 10.99.0.254 vrf SHARED `ip route show vrf CUST-A` lists 10.250.0.0/16 via 10.99.0.254 dev eth9. A host at 10.1.16.50 gets no reply from 10.250.0.53. Nothing has been configured in SHARED beyond the interface.
Q4. A tenant VRF has a healthy OSPF adjacency and an empty per-VRF routing table. `show ip ospf neighbor` shows the adjacency; `show ip route vrf CUST-A` shows only connected routes. Explain the state and give the safe fix.

The router was converted from a single-table design. The operator added: - set vrf name CUST-A table 1001 - set interfaces ethernet eth1 vrf CUST-A The pre-existing `set protocols ospf area 0 network 10.1.0.0/24` block at the top of the tree was left untouched. The adjacency on eth1 is Full. The customer reports that none of the routes they advertise are reachable.

Passing score: 75%. Answers are checked in this browser.