high riskservice affecting~45 min

Runbook: Configure a VRF

1 · Prerequisites

Confirm every item is in place before any state change.

VRF concept — L3VPN, the kernel vrf driver, and how VyOS implements routing tables per VRF
VRF configuration — set vrf name, table ids, attaching interfaces, addresses
VRF anti-patterns — VRFs for non-routing problems, overlapping space, leaks that install nothing, MTU
Services, web UI, HTTP API — what is reachable and how
commit-confirm — the rollback safety net for remote changes
The routing-table reason this VRF exists is written down. "Isolation" on its own is not one.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Out-of-band console access is proven now, not assumed. If the interface being bound is on the path you are connected over, the console is the only path that survives the commit
· The table id is allocated from the 1000-and-above range and checked against every existing VRF: show vrf and show configuration commands | grep "vrf name"
· The table id does not collide with an operator-defined routing table in the 1 to 252 range, and is not one of the kernel-reserved 253, 254 or 255
· The VRF name fits the estate convention, contains no spaces or slashes, and is short enough that the kernel device name derived from it stays inside the 15-character interface-name limit
· Baseline captured off the router: show vrf, show configuration commands | grep vrf, show interfaces, show ip route, and ip -d link show type vrf
· The interface being bound is identified by name and confirmed against the patch record. Binding the wrong port moves a live subnet out of the global routing table
· The addressing plan for this VRF does not overlap any other VRF on this router, and the plan is recorded
· Every firewall rule and NAT rule that references the interface has been listed, because binding the interface does not rewrite them
· Whether this VRF needs a default route, and where it points, is decided before the change rather than discovered afterwards

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Open the out-of-band console session and confirm you get a prompt. Keep it open for the whole change.
2Capture the baseline: show vrf, show interfaces, show ip route, and the kernel view with ip -d link show type vrf and ip route show table all.
3Confirm the chosen table id is unused, in both the VyOS view and the kernel view. A table id that is free in one and taken in the other is the state that produces a half-applied commit.
4Enter configure. Declare the VRF first: set vrf name CUST-A table 1001 plus a description. Nothing is bound yet, so this step is harmless on its own.
5Bind the interface: set interfaces ethernet eth1 vrf CUST-A. This is the step that removes the interface's connected route from the global table.
6Address the interface inside the VRF: set interfaces ethernet eth1 address 10.1.0.1/24. Order matters; the address is what installs the connected route into the new table.
7Add the routing the VRF needs. Static routes go under set vrf name CUST-A protocols static ...; a routing protocol for this VRF goes under set vrf name CUST-A protocols ... and never at the global level.
8Run compare. Confirm the diff contains the VRF declaration, exactly one interface binding, and nothing that touches an interface you did not name.
9Apply with commit-confirm 10 and exit. Do not run confirm yet.
10Reconcile the two views. VyOS: show vrf and show vrf name CUST-A. Kernel: ip -d link show type vrf, ip -d link show dev eth1, ip route show table 1001. All three kernel facts must agree with the VyOS view.
11Confirm the isolation is real: the interface's subnet must be present in the VRF table and absent from the global table.
12Test forwarding from inside the VRF, sourced from the VRF interface, to the VRF gateway. Reachability from the global table is not evidence of anything.
13Re-check the firewall and NAT rules listed in the pre-checks. Confirm each still matches what it was written to match.
14If everything reconciles: configure, confirm, save.
15If it does not: capture the kernel state before touching anything, then let the timer expire or run rollback 1 and commit.
16After any rollback, check for an orphan VRF master left behind in the kernel and clean it up before the next attempt.

4 · Verification

Confirm the procedure actually fixed the problem.

✓show vrf lists the VRF with the expected table id and reports it up
✓ip -d link show type vrf shows a VRF master device carrying table id 1001, and it is in state UP
✓ip -d link show dev eth1 shows the interface enslaved to that master. A nomaster here means the configuration tree says one thing and the kernel says another
✓ip route show table 1001 contains the connected route for the interface subnet, and the default route if one was configured
✓The interface subnet is absent from the global table: ip route show table 254 does not list it. This is the evidence that the isolation exists rather than being claimed
✓A ping sourced from the VRF interface reaches the VRF gateway
✓show configuration commands | grep vrf matches the intended design, with exactly the interfaces named in the ticket bound
✓Every firewall and NAT rule listed in the pre-checks still matches the traffic it was written for
✓The management session used for the change is still up, or the console session is, and both were expected to be

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶Preferred path: do not run confirm. Let the commit-confirm timer expire; the router restores the previous configuration on its own.
↶Explicit path: rollback 1 then commit inside configure.
↶To detach only the interface and leave the VRF in place: delete interfaces ethernet eth1 vrf then commit. The interface returns to the global table and its addresses install there instead.
↶To remove the VRF and everything bound to it: delete vrf name CUST-A then commit. VyOS unbinds the slave interfaces before deleting the master.
↶A detach that leaves the address behind is a half-rollback: the address was placed in the VRF context, so after the unbind it may no longer be routable. Delete or replace the address as part of the same change.
↶Check for an orphan master after every rollback. rollback reverts the configuration tree; it does not delete the kernel VRF device. ip -d link show type vrf reveals it, and a later attempt to re-create the same VRF fails with RTNETLINK answers: File exists until it is removed.
↶Verify the post-rollback state in both views before declaring the router restored: show vrf for the VyOS view, ip -d link show type vrf and ip route show table 254 for the kernel view.
↶Capture the failing kernel state before rolling back. A half-applied VRF commit is diagnosable only from ip -d link show output taken while it is still broken.

6 · Escalation

When the runbook isn't enough, contact:

· Escalate to network engineering before confirming if show vrf and the kernel view disagree. A configuration tree that claims a binding the kernel does not have is an isolation failure, and traffic is going somewhere nobody has looked.
· Escalate to network engineering if the commit fails with RTNETLINK answers: File exists. That is orphan kernel state from an earlier attempt, and clearing it by hand on a production router is a change of its own.
· Escalate to the platform team if the router stops answering on both the data and the management path. Recover from the console session opened before the change.
· Escalate to the service owner before confirming if a firewall or NAT rule stopped matching. Binding an interface into a VRF changes which table its traffic is looked up in, and a policy that silently stops matching is worse than one that visibly breaks.
· Escalate to the network architect if the design needs traffic to cross between this VRF and another. Route leaking is a separate design decision with its own failure modes; do not improvise it inside a change window.
· Holding is a legitimate outcome: let the timer expire, keep the ticket open, and return with the table-id allocation or the addressing plan that was missing.

A VRF is a routing table plus the set of interfaces whose lookups use it. That is the whole model, and the single most important consequence is the one operators forget under pressure: the kernel never falls back. A packet arriving on an interface bound to CUST-A is looked up in CUST-A’s table and nowhere else. If that table has no route, the packet is dropped with “no route to host” — even when the global table has a perfectly good default sitting right there.

So this change has two halves. Creating the VRF is trivial and reversible. Binding an interface to it is neither: it takes that interface’s connected route out of the global table, and everything that used to reach the subnet through the global table stops.

When this runbook applies

A new tenant, customer or environment needs its own routing table on an existing router.
An interface is moving from the global table into an existing VRF.
A dedicated management VRF is being created so the management plane is not reachable from the forwarding plane.

When it does not apply

You want tenants separated by policy, not by topology. If the tenants share a routing topology and only differ in what they are allowed to reach, VRFs are the wrong primitive — you want VLANs and firewall rules. vyos-xv-06-vrf-anti-patterns opens with this because it is the commonest misuse: a multi-VRF estate built to solve a firewall problem, which then has to be operated forever.
Traffic needs to cross between VRFs. Route leaking is a design decision with its own failure modes, and improvising it in a change window is how a leak ends up asymmetric. Design it, review it, then run it as its own change.
A single host or a small subnet needs isolating. A firewall source-address match does that, without adding a routing table to every future troubleshooting session.
The VRF already exists and is misbehaving. That is a diagnosis. Use the VRF troubleshooting runbook.

Blast radius

One interface, one routing table — and every flow that currently reaches the interface’s subnet through the global table.

That second clause is the one to write on the ticket. Before you bind anything, answer: what reaches this subnet today, and by which route? If the answer includes “the monitoring system”, “the backup server” or “my SSH session”, those flows break at commit and need a plan, not a surprise.

Binding the interface you are connected over will cut your session

This is not a risk, it is the mechanism working correctly. Binding an interface to a VRF moves its connected route into the VRF table. Your SSH session is looked up in the global table. At commit, the return path for your own packets disappears.

There are exactly two acceptable answers:

Do the change from the console, and treat the network session as expendable.
Do not bind that interface. Management belongs on its own interface and, in a hardened build, its own VRF — with the SSH service bound into that VRF in the same change, so the service follows the interface. vyos-iii-06-services-and-apis covers that build.

commit-confirm helps, but only if you can still see the router to decide. If your only path is the interface you are binding, the timer expires while you are locked out and you learn the outcome afterwards. Open the console first.

Table-id allocation

The VRF name is a label; the table id is the identity. The kernel does its lookups by table id, and two VRFs that somehow share one are the same VRF as far as forwarding is concerned.

Range	Belongs to	Use for a VRF?
1 to 252	Operator-defined routing tables (policy routing)	No — collides with non-VRF tables
253, 254, 255	Kernel reserved (default, main, local)	Never
1000 and above	Convention for VRFs in this course	Yes

Allocating from 1000 upwards keeps VRF tables visually separate from policy-routing tables in ip route show table all, which is the view you will be reading at 03:00. A collision is not merely untidy: the second ip link add fails, and what it leaves behind is a partially created VRF that the configuration tree does not know about.

Prove the way back in

Read-only / Safeon the console path

show version
show interfaces

If the change touches the interface carrying your session, the console is not a precaution — it is the procedure. Get a prompt on it before you type configure anywhere.

Capture both views of the baseline

VyOS and the kernel are two separate sources of truth about VRFs, and the failure this runbook is most likely to hit is the two disagreeing. Capture both now so you have something to diff against.

Read-only / SafeVyOS view

show vrf
show configuration commands | grep vrf
show interfaces
show ip route

Read-only / Safekernel view

ip -d link show type vrf
ip route show table all | head -50

Read the kernel capture for two things: which table ids are already in use, and whether any VRF master device exists that show vrf did not mention. A master with no configuration behind it is orphan state from an earlier attempt, and it will make this change fail in a way that looks like a VyOS bug.

Build the candidate in dependency order

The order is not stylistic. The master device must exist before an interface can be enslaved to it, and the interface must be enslaved before an address on it lands in the right table.

Configuration changedeclare, bind, address

configure

# 1. Declare the VRF. Harmless on its own - nothing is bound yet.
set vrf name CUST-A table 1001
set vrf name CUST-A description "VRF for customer A"

# 2. Bind the interface. This is the step with the blast radius.
set interfaces ethernet eth1 vrf CUST-A

# 3. Address it inside the VRF. The connected route now installs in
#    table 1001 rather than the global table.
set interfaces ethernet eth1 address 10.1.0.1/24

# 4. Give the VRF the routing it needs.
set vrf name CUST-A protocols static route 0.0.0.0/0 next-hop 10.1.0.254

Routing protocols for a VRF live under the VRF, not at the global level

set protocols ospf ... at the global level runs OSPF against the global routing table, no matter which VRF the interfaces are bound to. The packets may still flow, because the connected routes are wherever the binding put them — but the routes the protocol learns land in the wrong table, and the isolation you configured is a fiction.

The rule is structural: anything that installs routes for this VRF goes under set vrf name CUST-A protocols .... This mirrors what FRR is doing underneath, where the same distinction is router ospf versus router ospf vrf CUST-A.

If you are configuring a routing protocol inside the VRF as part of this change, read vyos-xv-03-vrf-routing-protocols first — and check the per-VRF router-id, which must be unique across every VRF on the box and the global instance too.

Read the diff, then commit behind a timer

Read-only / Safecompare

compare

Read it for one thing above all others: exactly one interface binding, and it is the interface named in the ticket. An extra binding in this diff is an outage in the making, and it is far cheaper to see it here than in show interfaces afterwards.

Service impact possiblecommit with a leash

commit-confirm 10
exit

Point of no return. The binding is live. The interface’s subnet has left the global table.

Reconcile the VyOS view against the kernel view

This is the step that distinguishes a VRF change that worked from one that appears to have worked. VyOS can commit successfully and leave the kernel in a partial state: the master created but down, the slave never enslaved, the address added to an interface that is still in the global table.

Read-only / SafeVyOS says

show vrf
show vrf name CUST-A

Read-only / Safethe kernel says

# Substitute your own values before running:
IFACE=eth1
TABLE=1001

# Does a VRF master exist, and is it up with the right table id?
ip -d link show type vrf

# Is the interface actually enslaved to it?
ip -d link show dev "$IFACE"

# Are the VRF's routes where they should be?
ip route show table "$TABLE"

# And absent from where they should not be?
ip route show table 254

Four facts, and all four must hold:

A VRF master device exists and is state UP.
The interface’s master field names that device. If it says nomaster, the configuration tree and the kernel disagree and the isolation does not exist.
The connected route for the interface subnet is in the VRF table.
That same subnet is not in table 254.

Fact 4 is the one people skip, and it is the only one that proves the isolation rather than assuming it. A subnet visible in both tables means the interface is not really bound, and traffic is being forwarded by the global table while everyone believes it is contained.

Test from inside the VRF, then re-check the firewall

Read-only / Safethe VRF routing table

vtysh -c "show ip route vrf CUST-A"

Read-only / Safeforwarding from inside the VRF

# Substitute your own values before running. VRF_DEV must be the master
# device name that 'ip -d link show type vrf' actually printed, which is
# derived from the VRF name in the VyOS configuration - read it, do not
# assume it.
VRF_DEV=CUST-A
GW=10.1.0.254

sudo ip vrf exec "$VRF_DEV" ping -c 3 "$GW"

Reachability tested from the router’s global context proves nothing about a VRF, because that lookup never touches the VRF table. The test has to originate inside the VRF, so the lookup uses table 1001 and the bound interface. Testing from a host on the VRF subnet is the stronger version of the same check, because it exercises the path the customer will use.

Then go back to the firewall and NAT rules listed in the pre-checks. Binding an interface into a VRF does not rewrite them, and a rule that matched on something the VRF changed now silently stops matching. A policy that fails open is worse than one that fails closed, because nothing reports it.

Confirm, or hold

Configuration changemake it permanent

configure
confirm
save
exit

save matters more than usual here. Without it the VRF survives until the next reboot and then vanishes, taking a customer’s routing table with it, at a moment nobody is watching.

Holding is a first-class option. If the two views did not reconcile, do not fix forward under a running timer. Capture the kernel state, let the timer expire, and come back with a clean tree. The change owner decides by T+8 on a 10-minute timer, and “we did not understand the kernel state” is a complete and sufficient reason to hold.

Rollback

Service impact possibledetach one interface

configure
delete interfaces ethernet eth1 vrf
delete interfaces ethernet eth1 address 10.1.0.1/24
commit
save

Deleting the binding but keeping the address is the classic half rollback. The address was placed in the VRF’s context; after the unbind it sits on an interface in the global table where nothing was expecting it. Delete it or replace it with the global address it had before.

Common patterns

Symptom	Likely cause	Next check
Commit fails with `RTNETLINK answers: File exists`	Orphan VRF master from an earlier attempt	`ip -d link show type vrf`, compare against `show vrf`
`show vrf` looks right, traffic still uses the global path	Interface never enslaved at the kernel level	`ip -d link show dev eth1`, look for `nomaster`
Subnet visible in both the VRF table and table 254	Same cause as above; the isolation does not exist	`ip route show table 254`
Everything inside the VRF is unreachable off-subnet	No default route in the VRF table. The kernel does not fall back to the global table	`ip route show table 1001`
VRF master exists but slaves cannot forward	Master device is `state DOWN`	`ip -d link show type vrf`
Routing protocol runs but its routes are in the global table	Protocol configured at the global level instead of under `vrf name`	`show configuration commands` filtered for `protocols`, then compare against `show ip route vrf`
A firewall rule stopped matching after the change	The rule referenced something the binding changed	Re-read the rule against the traffic it was written for
Management session died at commit	The bound interface was the management path	Recover from console; re-plan the management path

Provenance

This runbook has not been executed end to end against a live router, which is why last_verified reads pending rather than carrying a date. Its command set comes from the course’s own VRF lessons — vyos-xv-01-vrf-concept, vyos-xv-02-vrf-config and vyos-xv-06-vrf-anti-patterns — from the management-plane material in vyos-iii-06-services-and-apis, and from the commit discipline in vyos-vi-02-commit-confirm. No command output is reproduced here, because none of it was captured.

Why this matters

A VRF gives you an isolation guarantee that is enforced by the kernel rather than by an operator reading a configuration carefully. That is genuinely valuable: nobody can misroute a packet between two tenants by being careless, because the lookup for that packet never touches the other tenant’s table.

The guarantee is only worth what its verification is worth. The configuration tree can say vrf CUST-A while the kernel says nomaster, and the difference between those two states is invisible on the page you configured and total in the data plane. One says the tenants are isolated; the other says everything has been going through the global table since the change, and no alarm exists for it.

The four-fact reconciliation above takes about ninety seconds. It is the entire difference between having isolation and believing you have it.