WAN policy routing — policy route rule-sets, alternate tables, and the mark that gets overwritten

Policy routing is the multi-WAN pattern that routes specific traffic to specific WANs on criteria other than the destination address. The criteria can be:

• Source address — guest VLAN traffic goes via WAN 2.
• Destination address or port — VoIP signalling goes via WAN 1.
• Connection mark — traffic a previous decision already classified stays on the WAN it started on.
• Protocol, DSCP, packet length, time of day — the rule-set has a long list of match criteria.

On VyOS 1.5 the whole thing is one tree: set policy route. A rule-set is a numbered list of rules, each with match criteria and an action; the action that matters here is set table, which sends the matched packet to an alternate routing table instead of the main one. The rule-set is then applied to the ingress interface the traffic arrives on.

That is four moving parts:

1. Alternate routing tables — set protocols static table <id> route ... gives each WAN its own default route.
2. A policy route rule-set — matches the traffic class and does set table <id>.
3. The interface binding — set policy route <name> interface <iface>, on the interface where the traffic enters the router.
4. The kernel ip rule — VyOS creates and maintains this for you. You do not write it.

The constraint that shapes the whole design

Before any configuration, read the one note in the VyOS documentation that determines what is and is not buildable:

flowchart LR
  PKT["Packet arrives on eth1"]
  PR["policy route PBR-LAN<br/>rules evaluated 1..999999"]
  M["rule matches:<br/>protocol / port / source / dscp"]
  ST["set table 100"]
  IPR["kernel ip rule<br/>maintained by VyOS"]
  T100["table 100<br/>default via WAN 1"]
  OUT["forwarded out eth0"]

  PKT --> PR
  PR --> M
  M --> ST
  ST --> IPR
  IPR --> T100
  T100 --> OUT

Configuring policy routing

1. Alternate routing tables

Each WAN gets a table with its own default route. Table IDs run 1-200.

set protocols static table 100 route 0.0.0.0/0 next-hop '203.0.113.1'
set protocols static table 200 route 0.0.0.0/0 next-hop '198.51.100.1'

The main table stays as it is — it is what everything not matched by a policy rule continues to use.

set protocols static route 0.0.0.0/0 next-hop '203.0.113.1'
set protocols static route 0.0.0.0/0 next-hop '198.51.100.1'

Confirm each alternate table independently before you point any traffic at it:

show ip route table 100
show ip route table 200

2. The policy route rule-set

set policy route PBR-LAN description 'Multi-WAN traffic classes'

set policy route PBR-LAN rule 10 description 'Inter-VLAN stays on main'
set policy route PBR-LAN rule 10 destination group network-group 'INTERNAL-NETS'
set policy route PBR-LAN rule 10 set table 'main'

set policy route PBR-LAN rule 20 description 'SIP signalling via WAN 1'
set policy route PBR-LAN rule 20 protocol 'udp'
set policy route PBR-LAN rule 20 destination port '5060'
set policy route PBR-LAN rule 20 set table '100'

set policy route PBR-LAN rule 30 description 'Guest VLAN via WAN 2'
set policy route PBR-LAN rule 30 source address '192.168.100.0/24'
set policy route PBR-LAN rule 30 set table '200'

Rules are evaluated from 1 upward and the first match wins, exactly as in the firewall. Rule 10 is the pattern worth copying: an early rule with set table 'main' is how you carve an exception out of a policy, and main is a legitimate value for set table. Without it, internal traffic between two local networks would be shoved at a WAN default route and dropped by the far end.

The network group it matches is an ordinary firewall group:

set firewall group network-group INTERNAL-NETS description 'Local networks'
set firewall group network-group INTERNAL-NETS network '192.168.100.0/24'
set firewall group network-group INTERNAL-NETS network '192.168.188.0/24'

3. Bind the rule-set to the ingress interface

set policy route PBR-LAN interface 'eth1'
set policy route PBR-LAN interface 'eth1.100'

Router-originated traffic needs a different tree

policy route is bound to an ingress interface, so it never sees traffic the router generates itself — a BGP session sourced from a loopback, an IPsec keepalive, a DNS query from the resolver, ping from the CLI. Those packets are handled by a separate tree:

set policy local-route rule 101 source address '203.0.113.254'
set policy local-route rule 101 set table '100'
set policy local-route rule 102 source address '198.51.100.254'
set policy local-route rule 102 set table '200'

This is the mechanism that makes a multi-homed router reply out of the interface a request arrived on, and the mechanism that pins one VPN tunnel to one provider and a second tunnel to the other. If you have configured policy route and the router’s own traffic is still leaving via the wrong WAN, policy local-route is the missing piece — not a bug in the rule-set you already wrote.

Production patterns

A latency-sensitive class on a dedicated WAN

set policy route PBR-LAN rule 20 protocol 'udp'
set policy route PBR-LAN rule 20 destination port '5060'
set policy route PBR-LAN rule 20 set table '100'
set protocols static table 100 route 0.0.0.0/0 next-hop '203.0.113.1'

Match the signalling port and you have pinned signalling. The media stream negotiates its own ephemeral UDP ports and will not match this rule — which is the usual reason “VoIP policy routing” half-works. Match on DSCP instead if the phones mark their media, or accept that you have pinned signalling only and say so in the runbook.

A bulk class on the cheaper WAN

set policy route PBR-LAN rule 30 source address '192.168.100.0/24'
set policy route PBR-LAN rule 30 set table '200'
set protocols static table 200 route 0.0.0.0/0 next-hop '198.51.100.1'

Source-address matching is the most reliable criterion available, because it does not depend on ports, protocol behaviour, or anything the application can change.

Sticky classification with a connection mark

Where a decision must survive for the life of a flow rather than being re-derived per packet, the rule-set can match on connection-mark:

set policy route PBR-LAN rule 40 connection-mark '10'
set policy route PBR-LAN rule 40 set table '100'

The mark itself has to be set somewhere that is not also doing set table — a policy route rule with set connection-mark, or a firewall rule — for the reason covered above. Keep the two jobs in separate rules and the model stays coherent.

Production failure modes

1. The rule-set is bound to the wrong interface. Bound to the WAN rather than the LAN, it never matches, and the traffic quietly uses the main table. Nothing errors.
2. The alternate table is empty. set table 100 with no routes in table 100 means the lookup finds nothing and the packet is dropped — a silent blackhole for exactly one traffic class, which is a miserable thing to diagnose. Always show ip route table 100 first.
3. The alternate table has no failover. A static default route in table 100 stays in table 100 whether or not WAN 1 is up. Policy-routed traffic keeps being handed to a dead next-hop while everything on the main table fails over cleanly. The pinned class is the last thing anyone tests and the first thing that breaks.
4. Return traffic is not symmetric. The policy applies to one direction on one ingress interface. If the far end replies to an address that routes back over the other WAN, the stateful firewall drops the reply — and the NAT translation the outbound packet used will not match either.
5. Router-originated traffic ignores the policy. Covered above: that is policy local-route, and its absence is the most common “the policy is not working” report that turns out not to be about policy route at all.
6. An exception rule is missing or numbered too high. Without an early set table 'main' rule, internal traffic between two local networks is sent to a WAN default route. Numbering that rule after the classifiers has the same effect as not having it.

Operational commands

# The tables the policy points at
show ip route table 100
show ip route table 200
show ip route

# What is configured
show configuration commands | match "policy route"
show configuration commands | match "protocols static table"

# The kernel's own view — the rules VyOS generated
sudo ip rule show
sudo ip route show table 100

# Flows and their marks
sudo conntrack -L

# Per-WAN counters and live capture
show interfaces ethernet eth0
sudo tcpdump -ni eth0
sudo tcpdump -ni eth1

Rule-level logging is available and is the fastest way to prove a rule is matching at all:

set policy route PBR-LAN rule 20 log 'enable'
set policy route PBR-LAN default-log

Note the shape: in the policy route tree log takes an explicit enable or disable, where the firewall tree uses a bare log node. They are different trees with different conventions, and the completion will tell you which you are in.

Rollback

compare
delete policy route PBR-LAN interface eth1
commit

Removing the interface binding is the surgical rollback: the rule-set stays defined for the next attempt, and all traffic returns to the main table immediately. Deleting the whole rule-set works too, but throws away the configuration you will want to look at while working out what was wrong.

After any change that alters which WAN a class uses, flush conntrack so existing flows are re-evaluated rather than riding out their timeouts on the old path:

sudo conntrack -F

That is a blunt instrument — it resets state for every flow on the router, not just the ones you changed — so treat it as a maintenance-window action.

rollback N reverts the whole configuration but reboots the router; on a live multi-WAN edge prefer the targeted delete above, and use commit-confirm when the change could cut off your own management path.

Production discipline

Cross-course references

• Part XIII-01 (XIII-VyOS-PBR / PBR concept) covers the policy-based routing concept.
• Part XXXVI-05 (XXXVI-VyOS-ECMP / fwmark) covers the fwmark integration with ECMP.
• Part XXXVII-04 (XXXVII-VyOS-Firewall / state tracking) covers connection marks.
• Part XXXIX-03 (XXXIX-VyOS-MultiWAN / load sharing) covers the active-active load distribution.

Quiz