Lab: Route Leaking Between VRFs

Objective

By the end of this lab you will have built one VyOS router carrying three independent routing contexts, and you will have proved — from kernel routing tables and packet captures, not from the configuration — exactly which prefixes cross between them. Along the way you will deliberately write the leak that commits cleanly, reads correctly to a reviewer, and installs no route at all, because recognising that state in thirty seconds is the single most useful thing this part of the course teaches.

Architecture

One router, three contexts, three segments, three Linux guests.

                       br-shared  198.51.100.0/24
      svc  198.51.100.53   (shared service)
           198.51.100.254  (stands in for the core router)
           dummy0 10.99.0.10/32  (stands in for the package mirror)
                    |
                    | eth0  198.51.100.1/24      default context, table 254
              +-----+--------------------------------+
              |            r-leak (VyOS 1.5)         |
              +--+--------------------------------+--+
      eth1 10.10.0.1/24                     eth2 10.20.0.1/24
      vrf mgmt, table 1001                  vrf tenant-a, table 1002
              |                                     |
        br-mgmt 10.10.0.0/24              br-tenant 10.20.0.0/24
              |                                     |
        mgmt-host 10.10.0.10                 tenant-host 10.20.0.10

This is the reference estate from Part XVI, made concrete. Two things are deliberately compressed so the lab fits in four guests, and you should know about both: svc carries the service address and the core-router address on the same interface, and the package mirror 10.99.0.10 is a loopback-style address on svc rather than a host one hop beyond it. The routing behaviour you are studying is identical — the mirror is still off-segment as far as the router is concerned, and still needs the next-hop-address form of the leak — but there is one fewer physical hop than the lesson’s diagram implies.

Requirements

• A hypervisor that can run four guests with three isolated layer-2 segments between them: Proxmox VE, libvirt/KVM, or any equivalent. Nested virtualisation is enough; nothing here needs hardware pass-through.
• A VyOS 1.5 LTS (circinus) image, installed to disk on one guest, with 1 GB RAM and three NICs. Every configuration command in this lab is VyOS 1.5 syntax. It has not been executed end to end on a physical box for this writing, so treat the outputs as shapes to match rather than as captures.
• Three small Linux guests with iproute2, python3, curl and tcpdump. 512 MB each is plenty. Debian 13 was the assumption when the commands were written; anything with a recent iproute2 behaves the same.
• Console access to the VyOS guest. Tasks 2 and 9 change interface bindings and remove VRFs. If you are managing the router over one of the three segments in this lab, those steps will drop your session. Use the hypervisor console, not SSH over a lab segment.
• Roughly 8 GB of disk and 3 GB of RAM in total.

Scenario

You run a single edge router for a small estate. mgmt and tenant-a are separate routing contexts because they must not see each other — that decision is already made and is not up for renegotiation. What has just arrived is a ticket: the management hosts cannot reach the shared HTTP service on 198.51.100.53, cannot reach the package mirror on 10.99.0.10, and the change window is Thursday.

The tempting answer — one default route into each VRF — is on the table, and part of your job today is to find out precisely what it would cost. The rest is to build the narrow version, and to establish the verification habit that distinguishes a leak that works from a leak that merely commits.

Tasks

Task 1: Build the segments and record the starting state

Create three isolated bridges on the hypervisor. On Proxmox VE, three vmbr interfaces with no physical port attached is the simplest form; on libvirt, three isolated networks. The requirement is only that each bridge carries exactly the guests named in the Architecture table and nothing else.

Attach the guests: r-leak gets one NIC on each bridge, in the order br-shared, br-mgmt, br-tenant, so they enumerate as eth0, eth1, eth2. Each Linux guest gets one NIC on its own bridge.

Before configuring anything, capture what you are about to change. Run this on each of the three Linux guests:

# Substitute your own interface name if it is not ens18.
IFACE=ens18

mkdir -p "$HOME/vrf-leak-lab"
ip addr show > "$HOME/vrf-leak-lab/pre-lab-addr.txt"
ip route show > "$HOME/vrf-leak-lab/pre-lab-route.txt"
ip link show "$IFACE"

Now address svc. It plays three roles: the shared service, the core router the default context points at, and the mirror behind that core router.

IFACE=ens18

ip link set "$IFACE" up
ip addr add 198.51.100.53/24 dev "$IFACE"
ip addr add 198.51.100.254/24 dev "$IFACE"

ip link add dummy0 type dummy
ip addr add 10.99.0.10/32 dev dummy0
ip link set dummy0 up

# The two consumer subnets are reached through the router. Without these the
# service can receive a request and still have no way to answer it.
ip route add 10.10.0.0/24 via 198.51.100.1
ip route add 10.20.0.0/24 via 198.51.100.1

10.99.0.10 is a local address on svc, so no forwarding is enabled and none is needed. That is the compression named in Architecture: the router still has to resolve 198.51.100.254 to reach it, which is the only property the lab depends on.

Then mgmt-host:

IFACE=ens18

ip link set "$IFACE" up
ip addr add 10.10.0.10/24 dev "$IFACE"
ip route add default via 10.10.0.1

And tenant-host, identically but on its own segment:

IFACE=ens18

ip link set "$IFACE" up
ip addr add 10.20.0.10/24 dev "$IFACE"
ip route add default via 10.20.0.1

Finally, start the shared service on svc and leave it running in its own terminal for the rest of the lab:

python3 -m http.server 8080 --bind 198.51.100.53

You now have three segments that cannot yet talk to each other, because the router has no configuration.

Task 2: Build the three routing contexts

Everything below runs on the VyOS console. The vrf node is applied before every interface node regardless of the order you type the statements, so a single commit is correct here.

configure

set system host-name r-leak

set vrf name mgmt table 1001
set vrf name mgmt description 'management context'
set vrf name tenant-a table 1002
set vrf name tenant-a description 'tenant A context'

set interfaces ethernet eth0 address 198.51.100.1/24
set interfaces ethernet eth0 description 'shared services segment'

set interfaces ethernet eth1 vrf mgmt
set interfaces ethernet eth1 address 10.10.0.1/24
set interfaces ethernet eth1 description 'mgmt hosts'

set interfaces ethernet eth2 vrf tenant-a
set interfaces ethernet eth2 address 10.20.0.1/24
set interfaces ethernet eth2 description 'tenant-a hosts'

set protocols static route 10.99.0.0/24 next-hop 198.51.100.254

commit
save
exit

The last statement is not a leak. It is an ordinary static route in the default context, and it exists so that the default context genuinely reaches 10.99.0.10 the way a real estate reaches an off-segment service. You will leak that reachability later; you cannot leak what the source context does not have.

Now verify the enslavement, in the kernel rather than in the configuration. Four things must all be true, and any one of them can be false while the other three are true.

$ show vrf

Name      State    MAC address        Flags                     Interfaces
--------  -------  -----------------  ------------------------  ------------
mgmt      up       be:6f:2a:11:8c:04  noarp,master,up,lower_up  eth1
tenant-a  up       9a:41:d7:33:0e:b2  noarp,master,up,lower_up  eth2

ip -d link show dev mgmt
ip -d link show dev eth1
ip route show table 1001
ip route show table 254

Read them in that order. ip -d link show dev mgmt must report vrf table 1001; ip -d link show dev eth1 must report master mgmt and vrf_slave table 1001. Table 1001 must hold 10.10.0.0/24 dev eth1 and nothing else. Table 254 must hold the connected 198.51.100.0/24, the static 10.99.0.0/24 via 198.51.100.254, and — this is the part people skip — no trace of 10.10.0.0/24. Its absence from the main table is the evidence that isolation is real.

Task 3: Prove the isolation before you start dismantling it

A lab that begins after the interesting state has already been broken teaches nothing. Establish the negative result first, and write it down.

From mgmt-host:

ping -c 3 -W 2 198.51.100.53
curl --max-time 5 http://198.51.100.53:8080/

Both fail. From the router, ask the same question two ways. First in operational mode, where ping takes the routing context as an argument:

ping 198.51.100.53 vrf mgmt

Then in the kernel table the operational command is really asking about:

ip route show table 1001

The router’s own probe fails too, and table 1001 shows exactly one line. That pairing matters: the failure is not the host’s, not the service’s, and not a firewall’s. There is no route, and there is no route because a VRF miss does not fall through to the main table.

Task 4: Write the leak that commits and installs nothing

This is the failure the whole part exists to teach, so write it on purpose. The intuitive statement points the leaked route at the router’s own address on the shared segment:

configure
set vrf name mgmt protocols static route 198.51.100.53/32 next-hop 198.51.100.1
commit
exit

No error. Now read the configuration back:

show configuration commands | match 198.51.100.53

It says exactly what you meant. A reviewer approving this change by reading the diff would approve it. Now ask the kernel:

$ ip route show table 1001

10.10.0.0/24 dev eth1 proto kernel scope link src 10.10.0.1

Nothing was installed. FRR resolves a next hop in the routing context the route belongs to, and the route belongs to mgmt. Table 1001 holds one connected prefix, 198.51.100.1 is not in it, the next hop does not resolve, and the static is held inactive and never offered to the kernel. An unresolvable static route is a valid configuration state, so nothing errors.

The control-plane view says the same thing in a form that takes longer to read:

vtysh -c 'show ip route vrf mgmt 198.51.100.53/32'

A route that is selected and installed carries > and * in the code column. This one carries neither. Read the markers, not the sentence beside them — the wording of the unresolved-next-hop note differs between FRR releases, and the markers do not.

Remove it before continuing:

configure
delete vrf name mgmt protocols static route 198.51.100.53/32
commit
exit

Task 5: Make the leak real

The working form separates the two halves of a route entry: the prefix decides which table the route is installed in, and the vrf value decides where the next hop is resolved. 198.51.100.0/24 is connected on eth0, so there is no next-hop address worth naming and the interface form is correct.

configure
set vrf name mgmt protocols static route 198.51.100.53/32 interface eth0 vrf default
commit
save
exit

$ ip route show table 1001

10.10.0.0/24 dev eth1 proto kernel scope link src 10.10.0.1
198.51.100.53 dev eth0 proto static

That second line is the signature of a leaked route and nothing else produces it: an entry in mgmt’s table whose output device, eth0, is not a member of mgmt. If you prefer names to table ids, ip route show vrf mgmt prints the same table.

Now test it from mgmt-host and watch it fail anyway:

ping -c 3 -W 2 198.51.100.53

Task 6: Find the missing direction from the capture, not from a guess

The forward leak is installed — you have kernel evidence — and the traffic still does not work. The reflex at this point is the firewall. Resist it for ninety seconds and take the capture instead, because a one-way leak and a firewall drop produce the same symptom and only one of them is your change.

Open two terminals on the router and run a capture in each while mgmt-host pings:

$ tcpdump -ni eth1 icmp

IP 10.10.0.10 > 198.51.100.53: ICMP echo request, id 4231, seq 1, length 64
IP 10.10.0.10 > 198.51.100.53: ICMP echo request, id 4231, seq 2, length 64

$ tcpdump -ni eth0 icmp

IP 10.10.0.10 > 198.51.100.53: ICMP echo request, id 4231, seq 1, length 64
IP 198.51.100.53 > 10.10.0.10: ICMP echo reply, id 4231, seq 1, length 64

That pair is the whole diagnosis. The request crosses the leak, the service answers, the reply arrives on eth0 — and eth0 is in the default context, so the reply is resolved in table 254, which has never heard of 10.10.0.0/24. Confirm the negative directly:

ip route show table 254

A leak is one route in one direction. The return path is a separate statement in a different part of the configuration tree, because the default context has no vrf name node to hang it under:

configure
set protocols static route 10.10.0.0/24 interface eth1 vrf mgmt
commit
save
exit

Verify the mirror-image signature in the main table — a route in table 254 whose output device belongs to mgmt — and then re-test with the real service flow rather than with ping:

ip route show table 254

curl --max-time 5 http://198.51.100.53:8080/

Task 7: Leak a destination that is not on a connected segment

10.99.0.10 is not connected anywhere. The default context reaches it through 198.51.100.254, so the leak has to name that address, and the vrf value tells FRR which context to resolve it in.

configure
set vrf name mgmt protocols static route 10.99.0.10/32 next-hop 198.51.100.254 vrf default
commit
save
exit

The installed route now carries both a via and a foreign output device:

$ ip route show table 1001

10.10.0.0/24 dev eth1 proto kernel scope link src 10.10.0.1
198.51.100.53 dev eth0 proto static
10.99.0.10 via 198.51.100.254 dev eth0 proto static

Confirm from mgmt-host with ping -c 3 10.99.0.10. Then do the tenant side yourself — one forward statement and the matching reverse — using what you have learned rather than by copying:

configure
set vrf name tenant-a protocols static route 198.51.100.53/32 interface eth0 vrf default
set protocols static route 10.20.0.0/24 interface eth2 vrf tenant-a
commit
save
exit

Before you go on, confirm the isolation that has not changed. From tenant-host:

ping -c 3 -W 2 10.10.0.10

It fails, and it should. Five leak statements have been committed so far and none of them connects the two consumer VRFs. Table 1002 has a route to the shared service and to nothing else in mgmt.

Task 8: Measure what a default-route leak actually opens

Leaking 0.0.0.0/0 is the change that passes review most reliably, because it is one short line that reads like a sensible default route. Measure it rather than arguing about it.

First record the baseline you are about to change. From tenant-host, all three of these fail right now:

ping -c 2 -W 2 198.51.100.254
ping -c 2 -W 2 10.99.0.10
ping -c 2 -W 2 10.10.0.10

Now leak the default into tenant-a, with a confirm window because you are changing the tenant’s reachability wholesale:

configure
set vrf name tenant-a protocols static route 0.0.0.0/0 interface eth0 vrf default
commit-confirm 10

Stay in configure mode. A ten-minute timer is now running: unless you type confirm before it expires the router reverts this commit on its own. That is exactly the behaviour you want on a change this wide, and it means the measurement below has a deadline.

Re-run the same three probes from tenant-host. All three now succeed. Two of them are unsurprising: 198.51.100.254 is on the shared segment and 10.99.0.10 is reachable through the default context’s own static route, neither of which anybody wrote a tenant statement for.

The third is the finding. 10.10.0.10 is in mgmt, and tenant-a still has no route to mgmt — but the default route sends the packet out eth0 to svc, svc holds a route to 10.10.0.0/24 via the router, and the router resolves that in table 254, where your Task 6 reverse leak forwards it into mgmt. The two consumer VRFs now reach each other by hairpinning through the shared segment. No statement in the configuration says so.

Keep the change deliberately rather than letting the timer take it away — you want to remove it in a controlled commit, not to watch a rollback land mid-measurement:

confirm
exit

Now replace the default with the statement you actually meant, in one commit so the tenant is never left with neither:

configure
delete vrf name tenant-a protocols static route 0.0.0.0/0
set vrf name tenant-a protocols static route 10.99.0.10/32 next-hop 198.51.100.254 vrf default
compare
commit
save
exit

Re-run the three probes. 10.99.0.10 succeeds, 198.51.100.254 and 10.10.0.10 fail again. That is the difference between a leak and a merge, measured.

Validation

Work through each of these and confirm the stated result before calling the lab done.

• ip route show table 1001 on the router contains exactly three lines: the connected 10.10.0.0/24 dev eth1, 198.51.100.53 dev eth0, and 10.99.0.10 via 198.51.100.254 dev eth0. Nothing else.
• ip route show table 1002 contains the connected 10.20.0.0/24 dev eth2, 198.51.100.53 dev eth0, 10.99.0.10 via 198.51.100.254 dev eth0, and no default route.
• ip route show table 254 contains 10.10.0.0/24 dev eth1 and 10.20.0.0/24 dev eth2 — the two reverse leaks — alongside the connected shared segment and the 10.99.0.0/24 static.
• From mgmt-host, curl --max-time 5 http://198.51.100.53:8080/ returns the directory listing, and ping -c 3 10.99.0.10 succeeds.
• From tenant-host, the same two probes succeed, and ping -c 2 -W 2 10.10.0.10 fails.
• From tenant-host, ping -c 2 -W 2 198.51.100.254 fails. This is the check that distinguishes per-prefix leaking from a default route; if it succeeds, a 0.0.0.0/0 statement is still in the tree somewhere.
• show configuration commands | match nexthop returns nothing, and vtysh -c 'show running-config' shows one nexthop-vrf line per leak — six in total. Counting them against your leak register is the audit.

Expected Outcome

A single VyOS router with three routing contexts and six leak statements, four forward and two reverse. Both consumer VRFs reach the shared HTTP service and the package mirror. Neither reaches the other, and neither reaches any address on the shared segment that was not named in a statement. Every one of those claims is backed by a line — or a proven absence of a line — in a kernel routing table rather than by the configuration reading correctly.

Your journal should contain the Task 3 baseline failures, the Task 4 commits-but-installs-nothing pair, the Task 6 capture pair, and the Task 8 before/after probe matrix.

Troubleshooting

The commit is rejected naming a VRF and a table id. Table ids must be between 100 and 65535, must not be 254, and must be unique across VRFs. A VRF name must be 15 characters or fewer and must not begin with an interface-type string such as eth or br.

show vrf lists the VRF but its Interfaces column is empty. The interface node did not run. Check ip -d link show dev eth1 for a master field; if the configuration claims the binding and the kernel does not have it, force the node to run again by deleting and re-adding the binding in one commit.

The leak is in table 1001 and traffic still fails one way. You are in Task 6 without having done Task 6. Capture on both interfaces before changing anything else; the request-with-no-reply pattern is the reverse leak, and it is indistinguishable from a firewall drop until you look.

The leak is in table 1001 and traffic fails in both directions. Check the service host’s own routing table. svc needs 10.10.0.0/24 via 198.51.100.1; without it the reply is dropped before the router ever sees it, and the router capture on eth0 will show the request going out and no reply arriving.

ping works and curl does not. The route is fine and something above the network layer is not. Confirm the HTTP server is still running and bound to 198.51.100.53 rather than to 127.0.0.1.

A VRF refuses to be deleted. It still has member interfaces, static routes inside it, or policy routes naming it. Verification reads the candidate configuration, so delete the references and the VRF in the same commit rather than in two.

Cleanup

Cleanup here is not only removal: the VRF teardown moves live subnets between routing tables, and the routing-policy rules the first VRF installed are unwound when the last one goes away. Do it in order and check the kernel after.

Step 1. Remove the leaks in both directions, then the contexts, in one commit:

configure

delete vrf name mgmt protocols static route 198.51.100.53/32
delete vrf name mgmt protocols static route 10.99.0.10/32
delete vrf name tenant-a protocols static route 198.51.100.53/32
delete vrf name tenant-a protocols static route 10.99.0.10/32
delete protocols static route 10.10.0.0/24
delete protocols static route 10.20.0.0/24
delete protocols static route 10.99.0.0/24

delete interfaces ethernet eth1 vrf
delete interfaces ethernet eth2 vrf
delete vrf name mgmt
delete vrf name tenant-a

compare
commit
save
exit

Step 2. Confirm the box is back to one routing context:

ip -4 rule show
ip route show table 254
ip route show table 1001

ip -4 rule show should no longer carry the l3mdev rules the first VRF installed, and table 1001 should be empty. If either is not true, something still references a VRF; show vrf names it.

Step 3. Remove the addressing from the router, if you are returning the guest rather than deleting it:

configure
delete interfaces ethernet eth0 address
delete interfaces ethernet eth1 address
delete interfaces ethernet eth2 address
commit
save
exit

Step 4. On each Linux guest, stop the service, drop what you added, and diff against the Task 1 capture rather than trusting that you remembered everything:

IFACE=ens18

ip link del dummy0 2>/dev/null || true
ip addr flush dev "$IFACE"
ip route flush dev "$IFACE"

ip addr show > /tmp/post-lab-addr.txt
diff -u "$HOME/vrf-leak-lab/pre-lab-addr.txt" /tmp/post-lab-addr.txt \
  && echo 'ADDRESSING RESTORED'

If the guests originally took their addressing from DHCP, the flush above leaves them with none — re-run the DHCP client (dhclient "$IFACE" or systemctl restart systemd-networkd, depending on the distribution) and diff again. That is the case the diff exists to catch: cleanup that removes what the lab added and also removes what the guest arrived with is not cleanup.

Stop the python3 -m http.server process with Ctrl-C in its terminal. Step 5: delete the three bridges on the hypervisor, or leave them if you intend to run the next lab on the same wiring.

What You Learned

• A VRF miss does not fall through. Task 3 proved it as a result rather than asserting it: table 1001 had one route, and the shared service was unreachable from mgmt even though the router itself could reach it perfectly well from the default context.
• A leak can commit, read correctly, and install nothing. You wrote that state deliberately in Task 4 and confirmed it in the one place it is visible. The configuration is not the evidence; the receiving kernel table is.
• The vrf value on the next hop is the whole mechanism. Two statements differing only in that token produced an empty table and a working leak. nexthop-vrf is what the token renders to.
• Direction is a design decision, and it costs a separate statement. Task 6 showed the request crossing and the reply dying at the router, diagnosed from captures on both interfaces before any other subsystem was touched.
• There are two static forms, and which one is right depends on the destination. The interface form for a segment connected in the other context; the next-hop-address form for anything behind a router.
• The scope of a leak is the scope of its prefix. Task 8 measured a default-route leak opening a path between two VRFs that no statement described, by hairpinning through the shared segment — and per-prefix statements closing it again.