Policy Route Sends Traffic to Wrong WAN

Reported symptoms

Three tickets, three queues, eleven days, and not one of them mentions the router.

The oldest is with the SaaS vendor. Nine days ago the finance team at the Northgate depot stopped being able to log in to the ERP. The page they get is an access-denied page rather than a wrong-password page, which the vendor’s first line read as a licensing problem and escalated accordingly. Finance have been working through the HQ VPN as a workaround, which is slow and which they are unhappy about.

The second is with the voice carrier. Call quality at Northgate has been worse since the voice-quality project completed than it was before the project started. Users describe choppy audio in the afternoons and during the evening. The carrier has been asked to investigate the new circuit and has so far reported no faults on it.

The third is with the backup team. The nightly off-site backup from the depot has overrun its window and been killed by the scheduler on six of the last eleven nights. It was reliable for two years before that.

Everything else at the site is normal. Browsing is fine, video calls to HQ are fine, and the site-to-site IPsec tunnel to HQ has not flapped once. That last fact has been cited in all three tickets as evidence that the depot’s connectivity is healthy.

One change touched the depot router in three months. Eleven days ago a policy route was added to send voice traffic out the new low-latency circuit. It was tested by placing a call and watching the packets leave the correct interface. It worked. The change was closed the same evening.

The estate

rtr-edge-1 is the VyOS 1.5 LTS router at the Northgate depot.

• eth0 faces the primary ISP. This is the circuit the depot has always used, it is the circuit whose public address the SaaS vendor allowlisted three years ago, and it is the source of the IPsec tunnel to HQ.
• eth3 faces the new circuit: 50 Mbit, low latency, bought for voice and sized for a few dozen concurrent calls. Its next-hop is 198.51.100.1.
• eth1 is the transit link to the campus layer-3 switch. Every depot subnet sits behind that switch and arrives at the router on this one interface: 10.10.0.0/24 voice, 10.10.20.0/24 data, 10.10.30.0/24 guest, 10.10.50.0/24 servers, plus a CCTV range.

Both WANs are source-NAT’d behind their own public address. The policy route added eleven days ago consists of a prefix-list naming the voice sources, a route-map that sets the voice circuit as the next-hop for anything matching it, and a binding of that route-map to eth1.

Evidence provided

vyos@rtr-edge-1:~$ ip rule show

0:      from all lookup local
100:    from 10.10.0.0/16 iif eth1 lookup 100
32766:  from all lookup main
32767:  from all lookup default

vyos@rtr-edge-1:~$ ip route show table 100

default via 198.51.100.1 dev eth3

vyos@rtr-edge-1:~$ ip route get 203.0.113.50 from 10.10.0.5 iif eth1

203.0.113.50 from 10.10.0.5 via 198.51.100.1 dev eth3 table 100

vyos@rtr-edge-1:~$ ip route get 203.0.113.50 from 10.10.20.44 iif eth1

203.0.113.50 from 10.10.20.44 via 198.51.100.1 dev eth3 table 100

vyos@rtr-edge-1:~$ tcpdump -ni eth3 'src net 10.10.20.0/24' -c 5

14:07:11.882 IP 10.10.20.44.51338 > 203.0.113.50.443: Flags [S], length 0
14:07:11.914 IP 10.10.20.17.44902 > 203.0.113.50.443: Flags [S], length 0
14:07:12.003 IP 10.10.20.44.51339 > 203.0.113.50.443: Flags [S], length 0

The route-map counter reads three orders of magnitude above what the depot’s call volume could produce. eth3 sits at its shaper ceiling every night during the backup window. eth0 is close to idle apart from the IPsec tunnel.

Work the evidence before reading on

Nothing here is failing. Every interface is up, every circuit is carrying traffic, and the router has no error to report.

1. Read the ip rule output and the intent side by side. The voice VLAN is 10.10.0.0/24. What does the rule say, and how many of the depot’s subnets does that cover?
2. The change was tested and the test passed. Work out precisely what the test proved, and then what it could not have proved no matter how carefully it was run.
3. Three tickets: a SaaS login refused before authentication, a backup that no longer fits its window, and voice that got worse after a voice project. What single property of a packet, if it changed, produces all three?
4. The IPsec tunnel never moved, and everyone has treated that as evidence of health. Look at the selector in the rule and work out why the tunnel is exempt. Is that reassuring or is it the most informative line in the output?
5. The router is behaving correctly and no component has failed. What does that tell you about which teams can possibly find this, and how long they will look before they do?

Before continuing: a selector is a statement about two sets, and only one of them was ever tested.

Root cause

1. One character of prefix length

The prefix-list feeding the route-map holds 10.10.0.0/16. The voice VLAN is 10.10.0.0/24.

In a configuration diff those two lines are nearly indistinguishable. As selectors they are not comparable at all. The /24 covers 254 addresses: the phones, and nothing else. The /16 covers 65,534, which at this depot means the data VLAN, the guest VLAN, the server subnet and the cameras.

Every one of those subnets arrives at the router on eth1, from the campus switch, so every one of them satisfies both halves of the rule selector — the source prefix and the input interface. Since eleven days ago, the entire depot has been egressing through a 50 Mbit circuit contracted to carry a few dozen calls.

2. Why the test could not have caught it

The change was tested. The engineer placed a call, captured on the voice circuit, saw the phone’s packets, and closed the ticket. That is a correct test, competently executed, and it proves the policy catches what it was meant to catch.

It cannot prove anything about what the policy leaves alone, because it never sends a packet that the policy is supposed to ignore. A selector that is too broad passes this test perfectly. So does a selector that is exactly right. The test does not distinguish them, and no amount of care in running it would change that.

This is the whole lesson of the file, and it generalises far beyond policy routing: whenever you write a match — a firewall rule, a prefix-list, a route-map, a NAT rule — you have made two claims, and the obvious test only exercises one of them.

3. Three symptoms, one changed egress

Nothing downstream of the router knows anything about routing policy. What they see is a changed source address and a changed amount of bandwidth, and each ticket is one consequence of that.

The SaaS vendor allowlists the depot by public source address. Traffic now leaves via the other circuit and is translated behind the other public address, so the vendor rejects the session before it reaches authentication — which is exactly why the error is an access-denied page and not a password error. The backup job does not fit through a voice-sized circuit and overruns. And call quality degraded because the circuit bought to isolate voice from everything else is now carrying everything else, so voice competes with a nightly backup for the same shaper. The project achieved the precise opposite of its goal.

Resolution

1. Establish the intended selector from the addressing plan, not from the rule. The voice VLAN is a /24; confirm that against the interface configuration on the campus switch rather than against the document that the prefix-list was typed from.
2. Check that the primary circuit can take the load back before you move it. The depot has been egressing through the voice circuit for eleven days, so the primary has no recent load history. Moving a saturated volume onto a circuit you have not sized turns a degraded site into a down one.
3. Decide the cutover time deliberately. Narrowing the selector moves the whole depot between WANs in one commit, and long-lived flows established over the voice circuit will change egress with their address translation already set up for the other one. Expect resets. The site is degraded rather than down, so an evening window is a legitimate choice - with an owner, an end time, and a note on all three tickets.
4. Stage the corrected prefix-list without committing, and read the change with compare. The failure you are fixing is a mistyped prefix length, so the diff is the last cheap chance to notice a second one.
5. Commit under commit-confirm with a short window. This is an egress-path change on a remote site router; if the new selector is also wrong, you want the router to undo it without you.
6. Watch the traffic move before confirming. Egress bytes should shift from the voice circuit to the primary within seconds, and the voice circuit should settle at something like call volume.
7. Reverse the workarounds. If the voice circuit address was added to the SaaS allowlist to unblock finance, remove it and re-test - an allowlist entry left behind hides the next occurrence of this exact failure.
8. Record the untracked next-hop as a separate change rather than fixing it here. The route-map hard-codes its next-hop with no tracking, so a voice circuit failure will hand voice traffic to a dead gateway instead of failing over. That deserves its own testing and its own window.

Verification

1. Run the negative case, which is the test the original change never ran. Resolve one address from every subnet the router serves: ip route get 203.0.113.50 from ADDR iif eth1. The voice source must resolve via the voice circuit; data, guest, server and camera sources must resolve via the primary ISP.
2. Include the iif argument on purpose. Without it the kernel answers for a locally-originated packet, and an interface-scoped rule does not apply to those, so the query answers a different question than the one you meant to ask and can return a reassuring wrong answer.
3. Confirm the selector itself. ip rule show prints the /24 rather than the /16, and there is exactly one policy rule rather than two.
4. Clear the route-map counters and let real traffic run. The matching count should now grow at something like call volume. A counter still climbing at site volume means the selector is still wrong.
5. Read the WAN counters as a ratio after the link has carried real traffic for a while. The proportion between the two circuits is the shape of the fix, and it is the number worth alerting on afterwards.
6. Prove the tickets, not the router. Log in to the SaaS ERP from a data-VLAN workstation. Query an external address-reflection service from that same workstation and confirm it reports the primary ISP address. Place a test call. Run the backup against real data rather than a token file.
7. Confirm the IPsec tunnel to HQ is still up. It was the one thing in this incident that was never wrong, and the fix had no business disturbing it.

Prevention

• Write the negative case into the change template. One address per subnet the router serves, resolved with ip route get ... iif, with the expected egress recorded before the change is made. It is a handful of lines, it takes a minute, and it is the only test that can distinguish a correct selector from an over-broad one.
• Generate policy prefix-lists from the addressing source of truth. A /16 and a /24 are one keystroke apart and read identically to a tired reviewer. A generated list cannot make this mistake; a typed one will eventually.
• Alert on the ratio of egress bytes between WANs, not on the absolutes. A selector that is too broad produces a step change in that ratio within minutes of the commit. This is the cheapest possible detector for the entire failure class, and it works even when nothing has failed.
• Treat a hard-coded policy next-hop as incomplete until it is tracked. Without tracking, the policy quietly becomes a black hole the moment its circuit fails, and the traffic class it was written to protect is the one that disappears.
• Reverse symptom-level workarounds as part of closing the incident. An allowlist entry added to unblock a user is indistinguishable from a deliberate design decision six months later, and it will hide the recurrence.
• Record the intent of every policy rule in its description. The next person to read this configuration in an incident has no way to know whether the /16 was a mistake or a deliberate choice made for a reason nobody wrote down.