Lab: Policy-Based Routing

Objective

By the end of this lab one LAN subnet will leave the site by a second uplink while everything else keeps using the first, and you will have proved it from the kernel rather than from the configuration. That is the small half of the objective.

The larger half is that you will have seen a policy route fail twice without producing an error, and you will know which single command separates the two cases. Policy-based routing is the part of a VyOS configuration where “it commits” and “it works” diverge most sharply, because each of its three parts is independently valid on its own. Rules with no interface attachment are valid. An attachment pointing at a table with no routes is valid. Neither logs anything. Both produce a router that reviews perfectly and forwards exactly as it did before.

Architecture

One edge router with two uplinks, one provider that terminates both of them, and two LAN hosts that exist so that the source of a packet can differ while its destination does not.

   198.51.100.10/24        192.0.2.10/24
      +--------+             +--------+
      | voice  |             |  data  |
      |  eth1  |             |  eth1  |
      +---+----+             +----+---+
          |                       |
          +-----------+-----------+
                      |
             bridge LAB-LAN  (no uplink)
                      |
                 eth1 | 192.0.2.1/24
                      | 198.51.100.1/24
              +-------+--------+
              |     edge       |
              | eth2      eth3 |
              +--+----------+--+
   203.0.113.1/30 |          | 203.0.113.5/30
                  |          |
        bridge LAB-WAN-A   bridge LAB-WAN-B
                  |          |
   203.0.113.2/30 |          | 203.0.113.6/30
              +--+----------+--+
              |      isp       |
              | eth1      eth2 |
              |   dum0         |
              | 203.0.113.200/32
              +----------------+

   eth0 on all four VMs: your management LAN. Never touched.

Two design choices are doing the teaching here, and both are worth reading before you build.

The destination is identical for both classes. Every packet in this lab is headed for 203.0.113.200, and both uplinks reach it. A destination-based routing table therefore cannot express the policy at all: there is exactly one destination and two ways to it, and nothing in the packet’s destination says which one it should take. That is the condition PBR exists for, and it is why the lab uses one target address rather than two.

Two subnets share one LAN segment. edge carries both 192.0.2.1/24 and 198.51.100.1/24 on eth1. Secondary addressing on one wire is ordinary production practice, and here it saves a VLAN configuration that would teach nothing this lab is about. In a real deployment the voice subnet would be a VLAN of its own and the policy would be attached to eth1.20 as well as to eth1 — the shape of the set policy route ... interface line is the same either way.

Every address comes from the ranges RFC 5737 reserves for documentation, so a bridge accidentally attached to something real cannot collide with a live network.

Requirements

• A hypervisor that can run four VMs. The build commands below are written for Proxmox VE; any KVM/libvirt host works if you can give each VM one NIC on your management LAN and the rest on isolated bridges.
• 2 GB RAM and 8 GB disk per VM — 8 GB and 32 GB in total.
• The VyOS 1.5 LTS ISO in hypervisor storage. Everything here is written against the 1.5 (circinus) configuration tree. On 1.3 the interface attachment was spelled set interfaces ethernet eth1 policy route PBR-LAN; that node no longer exists and the block will be rejected outright, which is the good case.
• Console access to all four VMs. No task touches eth0 or any management route, so you should not lose your SSH sessions — but Task 7 deliberately blackholes one traffic class, and “should not” is not a plan.
• Roughly 110 minutes, of which the VM build is about 35.

Scenario

The site has two uplinks to the same provider. They are not interchangeable: WAN-A is the general-purpose link and carries everything today, WAN-B was bought for the voice platform and is supposed to carry the voice subnet and nothing else.

Nobody can express that with a routing table. The voice traffic and the web traffic go to the same places; what differs is who sent them. The request that lands on your desk is one sentence long — “put the voice VLAN on WAN-B” — and it turns into three separate configuration objects, each of which fails quietly when it is the one you forgot.

Tasks

Task 1: Build the starting state

Three isolated bridges first. This is configuration text for /etc/network/interfaces on the Proxmox host, not a command:

auto vmbr93
iface vmbr93 inet manual
    bridge-ports none
    bridge-stp off
    bridge-fd 0

auto vmbr94
iface vmbr94 inet manual
    bridge-ports none
    bridge-stp off
    bridge-fd 0

auto vmbr95
iface vmbr95 inet manual
    bridge-ports none
    bridge-stp off
    bridge-fd 0

Apply with ifreload -a, or reboot the node if the host has no ifupdown2. bridge-ports none is what makes each bridge isolated: nothing you do in this lab can reach a physical network.

# Run on the Proxmox host.
# Substitute your own values. The ISO volume ID must match what
# `pvesm list local` prints for your upload.
ISO=local:iso/vyos-1.5-lts-amd64.iso
STORE=local-lvm

qm create 213 --name edge --memory 2048 --cores 2 \
  --net0 "virtio,bridge=vmbr0,firewall=0" \
  --net1 "virtio,bridge=vmbr93,firewall=0" \
  --net2 "virtio,bridge=vmbr94,firewall=0" \
  --net3 "virtio,bridge=vmbr95,firewall=0" \
  --scsihw virtio-scsi-single --scsi0 "$STORE:8" \
  --ide2 "$ISO,media=cdrom" --boot order=ide2 --ostype l26

qm create 214 --name isp --memory 2048 --cores 2 \
  --net0 "virtio,bridge=vmbr0,firewall=0" \
  --net1 "virtio,bridge=vmbr94,firewall=0" \
  --net2 "virtio,bridge=vmbr95,firewall=0" \
  --scsihw virtio-scsi-single --scsi0 "$STORE:8" \
  --ide2 "$ISO,media=cdrom" --boot order=ide2 --ostype l26

for VMID in 215 216; do
  qm create "$VMID" --memory 2048 --cores 2 \
    --net0 "virtio,bridge=vmbr0,firewall=0" \
    --net1 "virtio,bridge=vmbr93,firewall=0" \
    --scsihw virtio-scsi-single --scsi0 "$STORE:8" \
    --ide2 "$ISO,media=cdrom" --boot order=ide2 --ostype l26
done

qm set 215 --name voice
qm set 216 --name data

for VMID in 213 214 215 216; do qm start "$VMID"; done

firewall=0 on every NIC matters more than it looks. The Proxmox per-interface firewall runs on the host; leaving it on for a routing VM means the host silently drops transit packets the guest expects to forward, and the symptom is indistinguishable from a routing bug inside VyOS.

Open each console, log in with the live-boot credentials, run install image, accept the defaults, then point the boot order back at the disk:

# Run on the Proxmox host once `install image` has finished on all four.
for VMID in 213 214 215 216; do
  qm set "$VMID" --boot order=scsi0
  qm set "$VMID" --delete ide2
done

Task 2: Address the topology and establish the baseline

Configure eth0 on each VM for your management LAN in whatever way your environment expects, then paste the rest. On edge:

configure
set system host-name edge
set interfaces ethernet eth1 address '192.0.2.1/24'
set interfaces ethernet eth1 address '198.51.100.1/24'
set interfaces ethernet eth1 description 'LAN - data and voice subnets'
set interfaces ethernet eth2 address '203.0.113.1/30'
set interfaces ethernet eth2 description 'WAN-A - general purpose'
set interfaces ethernet eth3 address '203.0.113.5/30'
set interfaces ethernet eth3 description 'WAN-B - voice platform uplink'
set protocols static route 0.0.0.0/0 next-hop '203.0.113.2'
commit
save

On isp:

configure
set system host-name isp
set interfaces ethernet eth1 address '203.0.113.2/30'
set interfaces ethernet eth2 address '203.0.113.6/30'
set interfaces dummy dum0 address '203.0.113.200/32'
set interfaces dummy dum0 description 'stands in for a destination on the Internet'
set protocols static route 192.0.2.0/24 next-hop '203.0.113.1'
set protocols static route 198.51.100.0/24 next-hop '203.0.113.1'
commit
save

Note what the isp routes say: both LAN subnets are reachable back through WAN-A only. That is deliberate, and Task 4 will make you look at it again.

On voice:

configure
set system host-name voice
set interfaces ethernet eth1 address '198.51.100.10/24'
set protocols static route 0.0.0.0/0 next-hop '198.51.100.1'
commit
save

On data, the same with 192.0.2.10/24 and next-hop 192.0.2.1.

Save a restore point on edge — everything from here is reversible in one command if you take it now:

configure
save /config/pre-lab-13.boot
exit

Now the baseline. From both voice and data:

$ ping 203.0.113.200 count 3

PING 203.0.113.200 (203.0.113.200) 56(84) bytes of data.
64 bytes from 203.0.113.200: icmp_seq=1 ttl=63 time=0.712 ms
64 bytes from 203.0.113.200: icmp_seq=2 ttl=63 time=0.559 ms
64 bytes from 203.0.113.200: icmp_seq=3 ttl=63 time=0.583 ms

--- 203.0.113.200 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2032ms

Both work. Now establish the evidence method you will use for the rest of the lab, because “it pings” does not tell you which uplink carried it. Open two SSH sessions to edge and run one capture in each:

# Terminal A on edge
sudo tcpdump -nni eth2 icmp

# Terminal B on edge
sudo tcpdump -nni eth3 icmp

Re-run the ping from each host. Every echo request should appear in Terminal A and none in Terminal B: the main table has one default route and it points at WAN-A. Write both results in the journal — this is the “before” half of every comparison that follows.

Task 3: Build the policy wrong twice, on purpose

A policy route is three objects. Build the first one alone and commit it, because the failure it produces is one you will otherwise meet for the first time in production.

configure
set policy route PBR-LAN description 'LAN traffic classes'
set policy route PBR-LAN rule 20 description 'voice subnet leaves by WAN-B'
set policy route PBR-LAN rule 20 source address '198.51.100.0/24'
set policy route PBR-LAN rule 20 set table '100'
commit
save

That commits cleanly. Re-run the ping from voice with both captures running: the traffic is still on eth2. Nothing has changed and nothing told you so.

Now read the three views, in the order the lesson gives them. Each answers a question the previous one cannot.

View 1 — is the policy attached at all?

$ show configuration commands | match 'policy route'

set policy route PBR-LAN description 'LAN traffic classes'
set policy route PBR-LAN rule 20 description 'voice subnet leaves by WAN-B'
set policy route PBR-LAN rule 20 set table '100'
set policy route PBR-LAN rule 20 source address '198.51.100.0/24'

There is no interface line. That is the whole diagnosis, and it is the line a configuration review misses most often because the rules themselves look complete. On 1.4 and later the interface is a property of the policy, not of the interface — the node under interfaces is gone.

View 2 — did the router install a selector?

ip rule show

0:      from all lookup local
32766:  from all lookup main
32767:  from all lookup default

Three default entries and nothing else. No selector for table 100 was installed, because nothing put the policy in the forwarding path.

Now add the attachment and watch view 2 change:

configure
set policy route PBR-LAN interface 'eth1'
commit
save

ip rule show

0:      from all lookup local
32765:  from all fwmark 0x7fffff9b lookup 100
32766:  from all lookup main
32767:  from all lookup default

Two things in that middle line are worth a minute. The selector sits above main at 32766, which matters because the first match wins and from all lookup main matches every packet — a selector below it would never be reached. And the match is on a firewall mark, not on the source address you wrote, which is the first sign that VyOS builds this out of two kernel primitives rather than one.

Re-run the ping from voice. It is still on eth2.

View 3 — is there anything in the table?

$ ip route show table 100

It returns nothing at all — no header, no error, one blank line. That is the second failure, and it is the most common of all of them: set table '100' means “consult table 100 for this packet” and nothing more. It does not create the table and it does not populate it. The rule fired, the kernel consulted the table, found nothing, and the lookup continued to main — which is why every other view looked healthy and the traffic went the old way.

Before moving on, write both failures into the journal with the one command that identified each. You will use that pair more than anything else in this lab.

Task 4: Populate the table and prove the diversion

configure
set protocols static table 100 route 0.0.0.0/0 next-hop '203.0.113.6'
set protocols static table 100 route 0.0.0.0/0 description 'voice class egress - WAN-B'
commit
save

Confirm the table before pointing any traffic at it. VyOS exposes the same thing two ways, and both are worth knowing:

$ show ip route table 100

Codes: K - kernel route, C - connected, S - static, R - RIP,
     > - selected route, * - FIB route

S>* 0.0.0.0/0 [1/0] via 203.0.113.6, eth3, weight 1, 00:00:14

ip route show table 100

default via 203.0.113.6 dev eth3

Now prove the whole path without sending a packet. ip route get performs a real lookup with the selectors you give it, which means it answers the question a ping cannot: would this packet be diverted?

# Run on edge.
VOICE_SRC=198.51.100.10
DATA_SRC=192.0.2.10
TARGET=203.0.113.200

ip route get "$TARGET" from "$VOICE_SRC" iif eth1
ip route get "$TARGET" from "$DATA_SRC" iif eth1

203.0.113.200 from 198.51.100.10 via 203.0.113.6 dev eth3 table 100 iif eth1
203.0.113.200 from 192.0.2.10 via 203.0.113.2 dev eth2 iif eth1

Two lookups, same destination, different uplinks, and the first one names the table that decided it. That is PBR working, stated by the kernel.

Then send real traffic and confirm it from the wire, both captures running. ping 203.0.113.200 count 3 from voice should appear only on eth3; the same ping from data should appear only on eth2.

Task 5: The exception rule, and what first-match really means

Ping data (192.0.2.10) from voice. It fails, and the reason is worth deriving before you read on.

Traffic from 198.51.100.10 to 192.0.2.10 arrives on eth1 and matches rule 20 — the rule says source 198.51.100.0/24 and says nothing about the destination. So the lookup runs against table 100, which holds one default route, and a packet destined for the LAN next door is handed to WAN-B. isp has no reason to send it back.

This is failure mode six from the multi-WAN lesson, and it is why every production policy route starts with an exception:

configure
set firewall group network-group INTERNAL-NETS description 'local networks'
set firewall group network-group INTERNAL-NETS network '192.0.2.0/24'
set firewall group network-group INTERNAL-NETS network '198.51.100.0/24'

set policy route PBR-LAN rule 10 description 'internal-to-internal stays on main'
set policy route PBR-LAN rule 10 destination group network-group 'INTERNAL-NETS'
set policy route PBR-LAN rule 10 set table 'main'
commit
save

main is a legitimate value for set table, and rule 10 is how you carve an exception out of a policy rather than bolting it on afterwards. Re-run the ping from voice to data: it works, and ping 203.0.113.200 count 3 still leaves by eth3.

Now change one thing and watch it break. Renumber the exception above the classifier — that is, put it below in evaluation order:

configure
delete policy route PBR-LAN rule 10
set policy route PBR-LAN rule 30 description 'internal-to-internal stays on main'
set policy route PBR-LAN rule 30 destination group network-group 'INTERNAL-NETS'
set policy route PBR-LAN rule 30 set table 'main'
commit

Rules are evaluated from 1 upward and the first match wins, exactly as in the firewall. Rule 20 now matches first for every packet from the voice subnet, so rule 30 never runs and inter-LAN traffic is broken again. An exception numbered after the classifiers has the same effect as no exception at all. Prove it, then put it back:

configure
delete policy route PBR-LAN rule 30
set policy route PBR-LAN rule 10 description 'internal-to-internal stays on main'
set policy route PBR-LAN rule 10 destination group network-group 'INTERNAL-NETS'
set policy route PBR-LAN rule 10 set table 'main'
commit
save

To prove which rule is firing rather than inferring it, turn on rule logging. Note the shape — in the policy route tree log takes an explicit enable, where the firewall tree uses a bare log node:

configure
set policy route PBR-LAN rule 20 log 'enable'
commit

Run monitor log on edge and generate traffic from voice. Each matching packet writes a line. Turn it back off before you move on — rule logging on a production edge is a firehose, and this is a habit worth building on a lab router rather than discovering on a live one:

configure
delete policy route PBR-LAN rule 20 log
commit
save

Task 6: The traffic the policy will never see

From edge itself, with both captures running:

ping 203.0.113.200 count 3

It leaves by eth2, not eth3, and no rule you write under policy route will change that. The attachment names an interface and the match is on arrival; traffic the router originates never arrives on any interface, so no policy route can see it. A ping from the router, a BGP session the router opens, an NTP poll, a DNS query from its own resolver — none of them are candidates for diversion.

This is the single most common “my PBR is not working” report, and it is usually not about policy route at all. Before reading a single rule, ask where the traffic originated. If the answer is “on this router”, the tree is policy local-route:

set policy local-route rule 100 source address '198.51.100.1'
set policy local-route rule 100 set table '100'

That pins traffic the router sends from 198.51.100.1 to table 100, matched by source address because there is no arrival interface to match on. policy local-route6 is the IPv6 counterpart.

Task 7: The failure this design cannot survive

Table 100 holds a static default route. Static routes do not have failure detection. Find out what that costs.

On isp, take the WAN-B side down. This simulates the far end of WAN-B failing while edge’s own link stays up, which is the case the design cannot see:

configure
set interfaces ethernet eth2 disable
commit

Now, from edge:

show interfaces ethernet eth3
show ip route table 100

eth3 is still up. The bridge keeps the local carrier alive even though the far end is gone — and that is not an artefact of the lab, it is the common production case. A provider-side failure, a fault beyond the first switch, a peer that has stopped forwarding: in all of them your own link light stays on. The connected route survives, so the static default via 203.0.113.6 is still installed and still selected.

Ping from voice: total loss. Ping from data: unaffected. One traffic class is blackholed and the router’s routing table looks completely healthy, because from the router’s point of view nothing has failed.

Try the obvious remedy and watch it not work:

configure
set protocols static table 100 route 0.0.0.0/0 next-hop '203.0.113.2'
set protocols static table 100 route 0.0.0.0/0 next-hop '203.0.113.2' distance '200'
commit

show ip route table 100 now lists a second candidate at distance 200, and the voice class still blackholes — because the distance-1 route via 203.0.113.6 is still installed. Administrative distance decides which route is selected; it has nothing to say about whether a route should be installed. Nothing in this table can observe that the next hop has stopped answering.

Remove the experiment and restore the link:

configure
delete protocols static table 100 route 0.0.0.0/0 next-hop '203.0.113.2'
commit
save

On isp: configure, delete interfaces ethernet eth2 disable, commit, save. Confirm the voice class recovers.

Validation

• show configuration commands | match 'policy route' on edge shows an interface 'eth1' line for PBR-LAN.
• ip rule show shows a selector for table 100 sitting above the main rule at 32766, and the mark matches 0x7FFFFFFF minus 100.
• ip route show table 100 returns exactly one route, the default via 203.0.113.6.
• ip route get 203.0.113.200 from 198.51.100.10 iif eth1 returns a next-hop of 203.0.113.6 on eth3 and names table 100; the same command with source 192.0.2.10 returns 203.0.113.2 on eth2 and names no table.
• With both captures running, three echo requests from voice appear on eth3 and none on eth2; three from data appear on eth2 and none on eth3.
• ping 192.0.2.10 count 3 from voice succeeds, which proves rule 10 is being evaluated before rule 20.
• ping 203.0.113.200 count 3 from edge itself appears on eth2, and your journal records why.
• The journal holds the three-view evidence set twice — once from the inert policy in Task 3, once from the working one in Task 4 — and a table registry naming table 100, its intent, and its uplink.

Expected Outcome

An edge router where one LAN subnet leaves by a second uplink and everything else does not, expressed as one named policy with a description on every rule, an exception rule ahead of the classifiers, and a table ID recorded next to the intent it serves.

More importantly, a journal that can distinguish three states that look identical from a client: a policy that was never put in the path, a policy in the path whose table is empty, and a policy that is working. The first is diagnosed by the absence of an interface line, the second by an empty ip route show table 100, and the third by ip route get naming the table.

The topology is self-contained and can be destroyed when you are done with it; nothing later in the course depends on it.

Troubleshooting

The policy commits but ip rule show has only the three default entries. No interface line. The rules are valid configuration that nothing evaluates. Add set policy route PBR-LAN interface 'eth1'. If you are working from an older runbook that says set interfaces ethernet eth1 policy route PBR-LAN, that is the 1.3 form and 1.5 rejects the path — which is the good failure, because at least you find out at commit.

ip rule show has the selector and ip route get still returns the main-table next-hop. Run ip route show table 100. Empty means the rule fired and the lookup fell through; that is a table problem, not a policy problem. A non-empty table with a mangle counter of zero is the opposite case — the rule never matched, so check the source prefix and the attached interface against where the traffic actually arrives.

Nothing about the policy appears in vtysh -c 'show running-config'. It never will. VyOS renders policy route to nftables plus a kernel ip rule, both below FRR, and it does not expose FRR’s own policy routing daemon. Searching FRR for it is an hour spent proving something was never going to be there.

The voice host reaches the Internet target but not the data host. Rule 10 is missing, or it is numbered above rule 20 instead of below it. Check the numbers, not the presence.

Both hosts still leave by eth2 after everything looks right. Check which interface the traffic actually arrives on. The policy is attached to eth1; if a host is reaching edge some other way — a stale route via the management LAN, for instance — the policy is correct and simply not in that packet’s path.

Traffic from edge itself ignores the policy entirely. That is correct behaviour, not a fault. See Task 6.

Cleanup

Everything in this lab is policy on isolated bridges, so cleanup means returning to a known state rather than restoring service. Work through these in order and stop at whichever one matches your intent.

Step 1. To keep the topology for further practice — the PBR break/fix scenarios use this same shape — stop here and save on each router. Note in the journal that edge is running PBR-LAN with the voice subnet on WAN-B, so the next person to open the configuration is not surprised by a second uplink carrying traffic.

Step 2. To remove the policy but keep the topology, take the diversion out of the path first and delete the objects afterwards:

configure
delete policy route PBR-LAN interface
commit
delete policy route PBR-LAN
delete protocols static table 100
delete firewall group network-group INTERNAL-NETS
commit
save

Deleting the interface node on its own is the surgical move worth remembering: it disables the diversion in one line and leaves the rules intact for inspection, which a full delete does not. Confirm with ip rule show that the selector is gone and with ping 203.0.113.200 count 3 from voice that traffic is back on eth2.

Step 3. To reset edge completely, load the file you saved in Task 2:

configure
load /config/pre-lab-13.boot
compare
commit
save

compare before commit is not optional here. It is the only thing standing between you and a stored revision that turns out not to be the one you thought.

Step 4. To remove the topology entirely, stop and destroy the four VMs on the hypervisor, then delete vmbr93, vmbr94 and vmbr95 from /etc/network/interfaces and run ifreload -a.

What You Learned

• A policy route is three objects, and each one fails silently alone. Rules with no attachment commit and are never evaluated. An attachment pointing at an empty table diverts the lookup into nothing and falls through to main. Neither logs anything, and both review clean.
• The configuration is the intent; the kernel is the state. Three views answer three different questions, in one order: show configuration commands for the attachment, ip rule show for the selector, ip route show table for the contents. Reading them out of order wastes the time you spend on the ones that could not have told you anything.
• ip route get is the cheapest proof there is. It runs a real lookup with the selectors you supply, so it answers “would this packet be diverted” before any user asks. Use it with iif, always — without it you are testing router-originated traffic, which is a different question.
• First match wins, so the exception goes first. Without an early set table 'main' rule, internal traffic between two local subnets is handed to a WAN default and dropped by somebody else’s router. The same rule numbered after the classifier is the same as no rule.
• The router’s own traffic is out of scope, by design. There is no arrival interface to match on, so policy route never sees it. That is what policy local-route is for, and it needs a source clause narrow enough not to move your management path.
• An alternate table has no failover unless you build one. A static route stays installed while its next hop is dead, and administrative distance cannot help because the problem is installation, not selection. The pinned class is the last thing tested and the first thing to break.