medium riskservice affecting~45 min

Runbook: Troubleshoot VRF Routing

1 · Prerequisites

Confirm every item is in place before any state change.

VRF concept — L3VPN, the kernel vrf driver, and how VyOS implements routing tables per VRF
VRF configuration — set vrf name, table ids, attaching interfaces, addresses
VRF routing protocols — OSPF, BGP, and crossing the VRF boundary on purpose
IPv6 inside a VRF — link-local next-hops, OSPFv3 and BGP per VRF, leaking route6
VRF troubleshooting — RIB versus FIB, `ip vrf exec`, and the leak that installs nothing
VRF anti-patterns — VRFs for non-routing problems, overlapping space, leaks that install nothing, MTU
Troubleshooting leaking — routes missing, route-map blocking, asymmetric path
Evidence first — collect before changing, write down symptoms, no action without data
Return-path — forward and reverse, asymmetric routing, return-path sanity

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· The failing traffic is stated as a flow inside a named VRF before any command is run: which VRF, which source prefix, which destination prefix, and which interface the traffic enters on. "Customer A is down" is a ticket; "CUST-A, 10.1.0.0/24 to the shared services block, entering on eth1" is something a command can answer.
· It is established whether the destination is inside the same VRF, in another VRF reached by a leak, or in the global table. Those three have different failure modes and different first commands, and a leak is the only one of the three where a missing return statement is the likely cause.
· Whether this ever worked is established, and if so when it stopped. A VRF that has never carried traffic is usually a placement or binding defect; one that stopped is a change, a next-hop that went away, or a firewall rule that no longer matches.
· The scope is established: one prefix, one VRF, or every VRF on the router. Every VRF at once points at the router — a failed commit, a routing daemon that did not restart cleanly, a change to the global table that several VRFs lean on — not at any one tenant.
· The change record for the last 72 hours is checked, covering VRF membership of interfaces, leak statements in either direction, per-VRF routing protocol configuration, firewall interface groups, and any interface addressing change.
· Pre-diagnosis state is captured off-box and attached to the ticket: show configuration commands filtered to vrf, show vrf, the per-VRF FRR route table, and the kernel table for the same VRF. Both route views are captured, because the difference between them is frequently the whole diagnosis.
· Out-of-band console access is confirmed. VRF work touches interface bindings, and an interface binding change on the interface carrying the operator session removes the operator with no warning and no route back.
· It is agreed up front that ip link surgery is a probe and not a repair. Re-parenting an interface by hand makes the symptom disappear, does not survive a reboot, and is reconciled away by the next commit that touches the interface — which converts an outage into an outage that recurs at an unpredictable time.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Read the configuration first, and specifically look for a routing protocol stanza that sits at the top of the tree instead of under the VRF. This is the single commonest VRF defect in production: the adjacency forms, the routes install in the global table, and the tenant traffic scoped to the VRF table never sees them. Nothing errors.
2Confirm the kernel agrees with the configuration. show vrf lists each VRF, its master device state and the interfaces enslaved to it; a VRF listed with no interfaces is a VRF nothing can enter. Then confirm the specific interface binding directly, because the configuration tree is a statement of intent and not evidence that the intent was applied.
3Confirm FRR knows about the VRF at all. FRR keeps a routing instance per VRF and its own list of them; a VRF that exists in the configuration and in the kernel but is absent from FRR means the daemon was never told, and the cause is a failed commit rather than anything about routing.
4Ask the decisive question for the failing prefix: is it in the FRR RIB only, or in the kernel table too? The two have separate commands and they disagree in exactly the case that matters. A route whose next-hop FRR cannot resolve stays in the RIB, marked inactive, and is never handed to the kernel — so it prints in the command most operators run and forwards nothing.
5Read the flags rather than the presence of the entry. An entry with no best marker, no asterisk on the next-hop, and the word inactive beside it is a route that exists only in FRR. An entry with the best marker, an asterisk, an egress interface and — for a leak — the resolving VRF printed in parentheses is a route that forwards.
6If the prefix is a leak, check the direction that is not being reported. A leak is one prefix in one direction and the return is a separate statement in the other VRF. Configuring only the forward direction produces a failure that looks like a remote problem: the request arrives, the service answers, and the router has no route to send the answer back on.
7If the leak statement exists and installs nothing, check that the next-hop names the VRF it should be resolved in. Without that, FRR tries to resolve the next-hop inside the VRF that holds the route, where the address does not exist, and the statement commits cleanly while forwarding nothing.
8If the leak resolves and the prefix already exists in the destination VRF as a connected route, stop and recognise it: a connected route wins on administrative distance and the leaked route can never become best. That is an address-overlap problem, and it is a renumbering or translation project rather than a change.
9Test reachability only from inside the VRF. An unscoped ping from the router uses the global table and can succeed against a completely different host that happens to share the address, which is worse than no evidence because it looks like evidence.
10If routing is proven and traffic still fails, move to the firewall and connection tracking. VyOS matches interfaces, not VRFs, so a per-VRF policy is expressed through interface groups — and a leaked flow enters on one VRF interface and leaves on another, so the request is evaluated against one rule set and the reply against a different one.
11Prove bytes moved with a capture on both the ingress and the egress interface, because a leaked flow spans two of them and half the evidence is on the interface nobody captured. Everything above this step proves intent; only the capture proves delivery.
12Apply any fix inside configure, with compare read before the commit and commit-confirm 10 for anything touching an interface binding or a path the operator session traverses. Confirm only after both route views have been re-checked.
13Verify with both route views and one real application flow, then record in the ticket which of the four views had been lying. That sentence is the part of the incident the next person can use.

4 · Verification

Confirm the procedure actually fixed the problem.

✓No routing protocol stanza sits outside a VRF on a router whose routing is meant to be entirely per-VRF, confirmed by reading the configuration rather than by memory of what was deployed.
✓The VRF appears with its interfaces in show vrf, and the specific failing interface reports the VRF master device in the kernel link detail — the configuration and the kernel agree.
✓FRR lists the VRF in its own VRF table, with the table identifier the configuration allocated to it.
✓The failing prefix appears in the kernel routing table for that VRF, not only in the FRR RIB. This is the check that decides whether traffic can move at all, and it is the one most often skipped.
✓For a leaked prefix, the FRR entry carries the best marker, an asterisk on the next-hop, an egress interface, and the resolving VRF in parentheses — the signature of a leak that actually installed.
✓The return direction of any leak exists as its own statement in the other VRF, and its prefix has been checked in that VRF kernel table the same way.
✓Reachability is proven with a VRF-scoped test, never an unscoped one, and the result is recorded with the scoping command visible so a reviewer can see it was scoped.
✓Where the flow crosses VRFs, the firewall policy has been read as one policy covering both directions, and the established and related rules that carry the reply are confirmed present rather than assumed.
✓One real application flow succeeds end to end from a host inside the VRF, not only a router-sourced probe, and a capture on both interfaces shows the packets in both directions.
✓Any kernel state created by hand during the investigation has been removed and the equivalent change made through the configuration, with the result re-verified in the kernel.

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶Every diagnostic step in this runbook is read-only. Nothing needs rolling back until a fix is applied.
↶A fix applied with commit-confirm 10 and found to be wrong needs no action: leave it and the router reverts. That is especially valuable here, because the changes most likely to go wrong are interface bindings, which are also the changes most likely to remove the operator.
↶A committed and confirmed fix is reverted by deleting the specific nodes that were added, with compare read before the commit. Prefer that to a whole-configuration revert, which discards anything else committed in the same window.
↶Removing a VRF requires releasing its interfaces in the same candidate configuration. A commit that deletes a VRF while an interface still names it is rejected, and discovering that mid-rollback wastes the window.
↶Rolling back a leak removes reachability that something may now depend on — frequently the monitoring path for the VRF itself, which is to say the thing you were using to watch the rollback. Establish a second way to observe before removing a leak.
↶A configuration rollback restores a configuration, not a network. Sessions that dropped during the incident have to re-establish and neighbours have to re-converge; plan for that time rather than treating the commit as the end of the event.
↶Any interface re-parenting done by hand as a probe is undone by making the same change through the configuration and re-checking the kernel, not by leaving the manual state in place because it appears to work.

6 · Escalation

When the runbook isn't enough, contact:

· Escalate to the change owner when the evidence points at a routing stanza placed outside its VRF or a VRF binding that was never applied. Both are configuration defects with a specific author and a specific change, and correcting routing configuration owned by someone else, during an incident and without their context, replaces a known fault with an unknown one.
· Escalate to the owner of the other VRF when a leak is missing its return direction. The return statement lives in their VRF, it changes what their tenant can reach, and adding it unilaterally during a page is how one tenant acquires a path into another that nobody reviewed.
· Escalate to network design when the diagnosis is overlapping address space. A leaked prefix that loses to a connected route cannot be made to win by any configuration; the ways out are renumbering or translation at the boundary, and both are projects with owners rather than changes with a window.
· Escalate to the firewall or security owner when routing is proven correct and the flow is still dropped at a VRF boundary. The policy covering a leaked flow spans two rule sets, widening either one has a blast radius beyond this incident, and a permissive rule added under pressure outlives everyone who remembers why.
· Escalate to the platform owner when a commit appears to have partially applied — the configuration says one thing, the kernel or FRR says another. That is a commit-script failure, the router state is not what anyone believes it is, and continuing to configure on top of it compounds the problem.
· Hold rather than improvise when the only available action is one that leaves the router in a state the configuration does not describe: hand-built kernel state, a leak wider than the requirement, a firewall rule opened to prove a theory. Record the hold with an owner and an end time, and state plainly which tenant traffic stays broken while it holds.

A VRF-localised failure has a shape you learn to recognise. The packet enters on the right interface, the configuration reads correctly, every command an operator reaches for by habit reports health, and the traffic goes nowhere.

It is hard for one reason: VRF state lives in four places at once, and the commands most people know read two of them.

View	What it holds	A disagreement with the view above means
The configuration tree	What somebody typed	—
The kernel: master device, enslaved interfaces, per-VRF table	What the commit actually built	A commit that did not apply, or state changed outside the configuration
FRR: the per-VRF routing instance and its RIB	What the routing daemon believes and intends	FRR was never told the VRF exists
The kernel routing table for that VRF	What forwards packets	A route FRR holds and never installed — almost always an unresolved next-hop

Most of this runbook is walking those four in order. The rest is the boundary cases: a leak that commits and installs nothing, a leak that works in one direction, and a flow that routes correctly and is dropped by a firewall written per interface for a path that crosses two VRFs.

When this runbook applies

One tenant VRF is not carrying traffic while others are.
A prefix is present in show ip route for the VRF and traffic to it fails.
A shared service is reachable from some VRFs and not others.
A VRF was created or modified and never worked.
Traffic reaches a service across a leak and no reply arrives.

When it does not

A routing failure that has nothing to do with VRFs. If the failing flow stays inside one VRF and the fault is a missing route, a session that is down, or a policy that filtered a prefix, add the VRF selector to the commands and use the runbook for that subsystem. The VRF changes which table you query; it does not change the diagnosis.
A design question. “Should this tenant have a VRF?” is a real question with a real answer, and a page is the wrong time to ask it. A VRF earns its place when tenants need different default gateways, their own peering, or genuine routing-table isolation — not when the requirement, examined, is policy between known subnets.
A firewall problem on a single-VRF path. The symptom overlaps and the evidence does not.

Blast radius

Every diagnostic step here is read-only, which makes the ordering free. The risk is concentrated in two actions, and both are attractive precisely when the pressure is highest.

`ip link` surgery is a probe, not a repair

When the kernel and the configuration disagree, it is tempting to fix the kernel directly and re-parent the interface by hand. The symptom goes away immediately.

As a way to confirm a hypothesis, that is legitimate. As a repair it is not. VyOS owns that state. The change is not in the configuration, so it does not survive a reboot, and the next commit that touches the interface reconciles it away. You have converted an outage into an outage that will recur at an unpredictable time, on a router whose running state no longer matches its saved configuration — which is the worst possible starting position for whoever is on call when it does.

The durable repair goes through the configuration: release the interface, commit, bind it again, commit, save, and then re-read the kernel to confirm the binding took. If a commit genuinely failed to apply, the commit output said so, and the system log holds the commit-script error. Read that rather than papering over it — a router that silently fails to apply one node will silently fail to apply others.

The second is a leak added or widened to make a symptom go away. A leak is a deliberate hole between two routing tables. Widened during an incident, it is a hole nobody reviewed, in a design whose entire purpose was separation.

Step 1: the configuration, and the defect that hides in plain sight

Read-only / Safewhich interfaces claim a VRF, and where the routing lives

show configuration commands | match vrf
show configuration commands | match "^set protocols"

The second command is the important one, and the expected answer on a router whose routing is entirely per-VRF is nothing.

That defect has a specific history. An engineer converts a working non-VRF router to a VRF design by adding the VRF to the interfaces, and leaves the existing protocols ospf or protocols bgp block exactly where it was. The adjacency comes up, because the interface is still reachable. The routes install — into the global table. The tenant traffic, which is scoped to the VRF table, never sees them.

Nothing errors, nothing logs, and every protocol command reports health. It is the single commonest VRF defect in production and it is two minutes to find if you look for it first, which is why it is step one rather than step four.

The correct shape puts the routing instance inside the VRF:

Configuration changethe routing lives under the VRF, not beside it

set vrf name CUST-A table 1001
set interfaces ethernet eth1 vrf CUST-A
set interfaces ethernet eth1 address 10.1.0.1/24
set vrf name CUST-A protocols ospf parameters router-id 10.1.0.1
set vrf name CUST-A protocols ospf area 0 network 10.1.0.0/24

Step 2: the kernel

Read-only / Safethe VRFs, and whether the interface is really enslaved

show vrf
show interfaces

Read-only / Safethe binding, from the kernel rather than the configuration

# Substitute your own values before running:
SLAVEIF=eth1
VRFNAME=CUST-A

ip -d link show "$SLAVEIF"
ip route show vrf "$VRFNAME"
ip -6 route show vrf "$VRFNAME"

show vrf lists each VRF, its master device state and the interfaces enslaved to it. A VRF listed with no interfaces is a VRF nothing can enter, and that is a complete diagnosis on its own.

ip -d link show on the slave interface prints the master device it belongs to. That is evidence; the configuration tree is intent. When the two disagree, a commit did not apply, and the fix is in the configuration and the commit log rather than in the kernel.

If the master device does not exist at all, the kernel says so plainly. That is a VRF that was never created — look for a failed commit, not for a link to bring up.

Step 3: FRR

FRR keeps a routing instance per VRF, and its own view of which ones exist is the next thing to read.

Read-only / SafeFRR's own list, then the per-VRF protocol state

show ip route vrf CUST-A
show ip route vrf CUST-A 10.250.0.0/16
show bgp vrf CUST-A ipv4 unicast summary

Read-only / Safethe daemon directly, for the views VyOS does not wrap

sudo vtysh -c 'show vrf'
sudo vtysh -c 'show ip ospf vrf CUST-A neighbor'
sudo vtysh -c 'show running-config'

A VRF present in show vrf and absent from FRR’s list means zebra was never told about it. The configuration exists, the kernel device may exist, and the routing daemon has no instance for it — which is a commit failure wearing a routing costume.

A thin or empty per-VRF route table usually means one of two things: the routing protocol is not running inside the VRF, which is the placement defect from step 1, or its session is not up, which is the protocol’s own runbook with a VRF selector added.

Step 4: the question that decides everything

Is the failing prefix in FRR’s RIB only, or in the kernel table too?

Read-only / Safethe same prefix, asked of both

# Substitute your own values before running:
VRFNAME=CUST-A
PREFIX=10.250.0.0/16

sudo vtysh -c "show ip route vrf $VRFNAME $PREFIX"
ip route show vrf "$VRFNAME"

In a healthy VRF the two agree, which is exactly why the difference ambushes people when it matters. FRR keeps routes whose next-hop it cannot resolve: it marks them inactive and never offers them to the kernel. The command most operators run prints them. Traffic never sees them.

Read the flags, not the presence of the entry.

What the FRR entry shows	Meaning
No best marker, no asterisk on the next-hop, the word inactive	The next-hop did not resolve. This route exists only in the RIB
Best marker, an asterisk, and an egress interface	Installed, and forwarding
Best marker, asterisk, egress interface, and a VRF in parentheses	A leak that resolved — the parenthesised VRF is the signature

And the check that settles it either way is the kernel table. If the prefix is not there, it does not forward traffic, whatever the RIB said.

Step 5: leaks, and their three distinct signatures

Leaking a prefix between VRFs means telling FRR to resolve the next-hop in a different VRF from the one that holds the route. On VyOS that is a leaf on the next-hop naming the VRF to resolve in. Omit it and the statement still commits — and installs nothing, because FRR tries to resolve an address inside a table where it does not exist.

Configuration changeboth directions, every time

set vrf name CUST-A protocols static route 10.250.0.0/16 next-hop 10.99.0.254 vrf SHARED
set vrf name SHARED protocols static route 10.1.0.0/16 next-hop 10.1.0.254 vrf CUST-A

Three leak failures, three different signatures

Tell these apart before touching the configuration. They have different fixes, and two of them are indistinguishable from a customer ticket.

The leak installs nothing. The prefix is in the FRR RIB with the next-hop marked inactive, and absent from the kernel table. Either the VRF leaf is missing from the next-hop, or the next-hop does not exist in the VRF that was named. Fix the statement.

The leak works one way. Both tables hold their route, a scoped ping reaches the shared service, and the service cannot answer. The return leak in the other VRF was never configured. Leaks are directional and each direction is a separate statement — a fact that survives no amount of staring at the forward direction, because the forward direction is perfect.

The leak resolves and the packets still die. Both directions are in both kernel tables and traffic is dropped anyway. Now it is the firewall or connection tracking: the request egresses one interface and the reply arrives on an interface in a different VRF, so a stateful rule set written per interface never sees a matching flow.

The order matters. Diagnosing the third while the real defect is the first is how people end up adding permissive firewall rules to a router that was never dropping anything.

There is a fourth case that is not a leak failure at all, and it is worth recognising because no configuration will fix it. If the destination VRF already holds that prefix as a connected route, the leaked route can never win: connected sits at administrative distance 0 and a static at 1. The statement commits, the entry appears, the connected route stays best, and the traffic keeps going to the local segment. That is overlapping address space, and the ways out are renumbering one tenant or translating at the boundary. Both are projects.

Step 6: test reachability inside the VRF, or do not test at all

An unscoped ping from the router uses the global table. On an estate where two tenants legitimately use the same address ranges, it can succeed against entirely the wrong host — which is worse than no evidence, because it looks like evidence.

Read-only / Safescoped, and visibly scoped

sudo ip vrf exec CUST-A ping -c 3 10.1.0.254
sudo ip vrf exec CUST-A traceroute 10.250.0.53
sudo ip vrf exec CUST-A ss -tunap

VyOS wraps the common cases, so a VRF selector on ping does the same job for that one command. ip vrf exec is the general form and works for any program, including the ones VyOS never wrapped.

What `ip vrf exec` actually does to a process

The mental model people usually carry — “it enters a namespace” — is wrong, and the wrong model predicts the wrong failures.

It does not use network namespaces. VyOS VRFs are l3mdev masters inside a single network namespace, so there is no separate namespace to enter. What it does instead is put the process into a cgroup with a small eBPF program attached at socket-creation time: every socket the process opens from then on is bound to the VRF master device, and route lookups on such a socket are scoped to that master’s table.

Three consequences follow directly from that mechanism, and operators hit all three.

It needs the cgroup support present and the privilege to create a cgroup. On a system without it the command fails outright rather than silently running in the global table, which is the failure mode you want.

It only affects sockets the process creates after it is applied. A daemon that already holds an open socket is entirely unaffected by running a control tool for it under ip vrf exec.

It binds sockets; it does not move interfaces or change routing. Anything that bypasses the socket layer — a packet capture binding to an interface by name, for instance — is unaffected and needs the interface named directly.

ip netns exec is a genuinely different tool operating on full network namespaces. The command lines look alike and the mechanisms have nothing in common; reaching for it on a VyOS VRF finds nothing to enter.

Step 7: the firewall, and what VyOS does not give you

There is no firewall matcher for “traffic in VRF CUST-A”. VyOS matches interfaces, interface groups, addresses and ports, and VRF membership is a property of an interface rather than something a rule can select on.

The supported way to express a per-VRF policy is therefore to name the VRF’s interfaces, keep that list in one interface group, and maintain the group alongside the VRF so it does not drift as the VRF gains members. Rule sets are reached by a jump from the base chain — the forward chain for transit traffic, the input chain for traffic to the router itself.

Read-only / Safepolicy, groups, and the state that carries replies

show firewall
show firewall group
show firewall ipv4 forward filter

Read-only / Safewhether the reply is being tracked at all

sudo conntrack -L | head -40

Two things account for most firewall-shaped VRF incidents.

The established and related rules are not boilerplate. A stateful policy that permits the request and forgets the reply produces a VRF that routes and does not work, and it is indistinguishable from a routing fault until you look at the connection tracking.

A leaked flow crosses interfaces, so it crosses the assumptions. The request enters on a CUST-A interface and leaves on a SHARED one; the reply arrives the other way. If each VRF has its own rule set keyed to its own interface group, the request is evaluated by one and the reply by the other — and a policy that is correct for each VRF considered alone can drop the pair. The leak’s policy has to be designed as one policy covering both directions, and the time to do that is before the leak is configured rather than during the incident it causes.

Step 8: prove that bytes moved

Read-only / Safeboth interfaces, because a leaked flow uses two

monitor traffic interface eth1
monitor traffic interface eth9

Everything above this proves intent. The captures are the only commands that prove packets exist. When the intent looks right and the tenant is still down, go to the wire — on both interfaces, because capturing only the ingress side of a leaked flow answers half the question and feels like it answered all of it.

What each step costs if you get it wrong

Action	Cost when it is the wrong move
Reading only the FRR route table	A confident wrong answer: the route is there and forwards nothing
Skipping the misplaced-stanza check	Hours on a protocol that is working perfectly, in the wrong table
Re-parenting an interface with `ip link`	A fix that vanishes at the next reboot or the next commit
An unscoped ping as evidence	A successful reply from the wrong tenant’s host
Fixing the forward leak only	The service answers and the answer has nowhere to go
Adding a permissive firewall rule to test a theory	A hole in the separation the VRF design exists to provide
Widening a leak to make a symptom go away	A path between tenants that nobody reviewed and nobody will remove
Treating an address overlap as a leak bug	Time spent on a statement that cannot win, whatever it says

Escalation, and the option to hold

Escalate as soon as the evidence crosses a boundary you do not own, and send the evidence rather than the conclusion. A prefix present in the RIB with an unresolved next-hop, alongside the kernel table that does not contain it, is a complete and self-explanatory diagnosis. So is a forward leak that installs correctly next to a missing return statement in a VRF somebody else owns.

The return direction is the escalation people most often skip, because the fix looks trivial and the statement is one line. It is one line in another tenant’s routing table, changing what that tenant can reach. Adding it unilaterally at three in the morning is how one tenant acquires a path into another that no review ever saw.

Hold rather than improvise when the only available action leaves the router in a state its configuration does not describe. Hand-built kernel state, a leak wider than the requirement, a firewall rule opened to prove a theory: each removes today’s symptom and installs a fault that will surface without warning. Record the hold with an owner and an end time, and say plainly which tenant traffic stays broken while it holds.

Why this matters

VRF diagnosis is the clearest example in this course of a general truth: the command everyone knows is not always the command that answers the question. show ip route for a VRF reports what the routing daemon intends. The kernel table reports what forwards packets. They agree almost always, and the case where they differ is precisely the case you were paged for — which means the familiar command is at its least reliable exactly when it is most trusted.

The second habit is remembering that a VRF design’s defects are usually placement defects, not routing defects. A protocol stanza outside its VRF, a leak missing its return direction, a firewall rule set keyed to interfaces on a flow that crosses two of them: none of these is a hard problem, and every one of them presents as a healthy router with a dead tenant. They are found by looking in the right place first, and they are found nowhere else.

And the separation is the whole point. Every action available during a VRF incident that makes the symptom vanish quickly — a wider leak, a permissive rule, a hand-built binding — works by weakening the isolation the design was built to provide. An estate can be read by how many of those it has accumulated, and by the fact that not one of them appears in the document that describes what each tenant is supposed to reach.