Runbook: Investigate a Policy Route

1 · Prerequisites

Confirm every item is in place before any state change.

• PBR troubleshoot — show ip route, show ip rule, traceflow, asymmetry, counters
• PBR rules — the policy route tree, its interface, ip rule, and the table it points at
• Policy-based routing — concept, FIB vs RIB, table selection, why PBR exists
• PBR anti-patterns — PBR everywhere, conflicting route-maps, missing table, asymmetric PBR
• Routing table troubleshooting — ip route show table all, ip rule show, asymmetry
• Table ID namespace — reserved IDs, custom IDs, and VRF conflict
• Evidence first — collect before changing, write down symptoms, no action without data
• Return-path — forward and reverse, asymmetric routing, return-path sanity

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

• · The complaint is reduced to a tuple before any command is run: a source address, a destination, the interface the traffic enters on, and the path it is supposed to take. "PBR is broken" is not an investigation; "10.10.0.5 to 8.8.8.8 arriving on eth1 should leave via eth2 and is leaving via eth0" is.
• · The intent of the policy is written down from the configuration or the change record, not inferred from its name. A route-map called RM-VOICE-OUT tells you nothing about which prefixes it matches or which next-hop it sets.
• · The test source address is confirmed to be inside the matched prefix. Testing from the router itself, or from a management address, is the single most common way this investigation produces a false negative — router-originated traffic does not arrive on the bound interface and the iif match never fires.
• · Read-only baseline captured to a file first: show configuration commands | match policy, vtysh -c "show running-config", ip rule show, ip route show table all. Every diagnostic that changes anything destroys part of this.
• · vtysh -c "show route-map RM-NAME" counters read and recorded BEFORE anything is cleared. A counter that has been zero since boot is the strongest single piece of evidence in this runbook, and clear route-map counters erases it permanently.
• · Recent change checked with show system commit. A policy route that worked yesterday and does not today is a change, and the commit log names it faster than any packet capture.
• · Out-of-band console confirmed if the interface under investigation also carries your session. A PBR rule bound to the interface you are logged in over can strand you on the next commit.
• · The table IDs in use on this router established from ip route show table all, and checked against the reserved IDs (0 local, 253 default, 254 main, 255 local). A PBR table that collides with a VRF table ID is a different fault from a PBR table that is simply empty.
• · The owner of the far side identified — the upstream that owns the next-hop, and whoever owns the stateful firewall on the return path. Roughly half of PBR investigations end as a handover rather than a local change.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1. 1Capture the baseline to a file before anything else: show configuration commands | match policy, show policy route-map, vtysh -c "show running-config", ip rule show, ip route show table all, and the route-map counters. Everything after this step degrades the evidence.
2. 2State the expected path explicitly in the ticket — source prefix, ingress interface, table ID, next-hop, egress interface — before running a single diagnostic. Half of these investigations end when the operator discovers the expected path was never what the configuration says.
3. 3Rung 1, configuration: show policy route-map and show configuration commands | match policy. Confirm the route-map exists, that at least one rule has action permit with a set ip next-hop, that the prefix-list contains the prefix you are testing from, and that the interface binding line is present and names the interface the traffic actually arrives on.
4. 4Rung 2, FRR render: vtysh -c "show running-config" filtered for the route-map, for ip rule, and for table. The commit engine turns the VyOS tree into FRR configuration; if the rule and the table route are absent here, the configuration never became policy and no amount of packet capture will show why.
5. 5Rung 3, kernel installation: ip rule show. The PBR rule must appear at a priority numerically below 32766, with the expected source prefix and iif. If the FRR render has it and the kernel does not, the fault is netlink — check systemctl status frr, vtysh -c "show zebra client summary" and journalctl -u frr.
6. 6Rung 3b, the table: ip route show table <id> for the table the rule points at. An installed rule pointing at an empty table is a silent no-op — the kernel consults the table, finds nothing, and falls through to main. This looks identical from the outside to "the rule was never installed" and has a different fix.
7. 7Rung 4, route resolution: ip route get <dst> from <src> and, where the ingress interface matters, ip route get <dst> from <src> iif <ifname>. This performs the full rule-database walk for those selectors and names the table, next-hop and egress interface that the kernel would actually use. It is the most precise local check available and it costs nothing.
8. 8Read the route-map counters: vtysh -c "show route-map RM-NAME". A Statistics line at zero means the rule has never matched anything — a match-clause or binding problem — while a counter that climbs while traffic still takes the wrong path means the rule fires and the table it selects is wrong.
9. 9Rung 5, the packet: capture on the expected egress interface with tcpdump -ni <ifname> "src <address>" while a real host generates traffic from the matching source. Traffic sourced on the router with ping -I exercises the local output path, not the forwarding path, and is a different test — use it knowingly or not at all.
10. 10If the forward path is correct, test the return path before concluding the policy works. traceroute -s <src> <dst> from a host inside the source prefix, plus a capture on the interface the replies come back on. Asymmetry is the most common PBR outage and it looks like a working forward path.
11. 11If the rule fires, the table has a route, and packets still vanish, test the next-hop itself: ping <next-hop> and show ip route <next-hop>. A hard-coded next-hop in a route-map does not fail over when the upstream dies — the kernel keeps forwarding to a dead address.
12. 12Decide whether the fix belongs on this router at all. A policy that is installed, matching, and resolving to a live next-hop, with the packets leaving the expected interface, is a working policy — and the fault is then the return path, the far-end firewall, or the expectation itself.
13. 13Apply at most one change, with commit-confirm 5 if the interface under investigation carries your session, then confirm and save only after the verification below passes. Do not fix three things at once; you will not know which one worked.
14. 14Close with the rung that failed, the evidence at that rung, and the fix recorded together. "Rule was installed, table 100 was empty because the route-map rule had action deny" is reusable; "PBR fixed" guarantees the next person repeats the whole ladder.

4 · Verification

Confirm the procedure actually fixed the problem.

• ✓ip rule show contains the PBR rule at the expected priority, below 32766, with the source prefix and iif that the intent calls for
• ✓ip route show table <id> for the PBR table contains the expected route, and the next-hop in it answers ping
• ✓ip route get <dst> from <src> returns the PBR table, the intended next-hop, and the intended egress interface — and ip route get for a source deliberately OUTSIDE the matched prefix returns the main-table path, proving the rule is selective rather than universal
• ✓vtysh -c "show route-map RM-NAME" shows the Statistics counter climbing while test traffic runs, having been read before any clear
• ✓tcpdump on the expected egress interface shows the test traffic, and a simultaneous capture on the previously-wrong interface shows none of it
• ✓The return path is confirmed symmetric: replies arrive on the interface the policy intends, verified by capture rather than assumed from the forward result
• ✓A real application transaction from a real host inside the source prefix completes end to end — not just an ICMP echo, which many stateful paths treat differently from TCP
• ✓Traffic from every OTHER source prefix on the same ingress interface is confirmed unaffected, by ip route get from one of those addresses
• ✓The IPv6 side is checked with ip -6 rule show and ip -6 route show table all if the interface is dual-stack, because policy route-map and policy route-map6 are separate bindings and fixing one does not touch the other

5 · Rollback

If verification fails, undo the procedure in reverse order.

• ↶The ladder is read-only from rung 1 to rung 5 and there is nothing to roll back. Everything below needs undoing deliberately.
• ↶vtysh -c "clear route-map counters" has no undo and destroys the cumulative match history. Read the counters and record them before you clear, and clear only when you need a clean window to observe fresh traffic.
• ↶vtysh -c "debug zebra kernel" writes at volume to the system log. Turn it off explicitly when finished rather than leaving it on — it can fill a small /var/log on a router that is already having a bad day.
• ↶A wrong interface binding: delete interfaces ethernet <wrong-ifname> policy route-map <name>, then set interfaces ethernet <correct-ifname> policy route-map <name>, then compare, commit, save.
• ↶A binding added during the investigation and now unwanted: delete interfaces ethernet <ifname> policy route-map <name>, compare, commit, save. Deleting the binding removes the ip rule and the generated table entry with it, which is the whole point — but it also returns every matched source to the main-table path in one step, so confirm that path is healthy first.
• ↶A next-hop corrected on a route-map rule: set policy route-map <name> rule <n> set ip next-hop <address>, compare, commit, save. Read the compare output; a typo here silently blackholes the whole matched prefix.
• ↶Whole-tree revert: rollback N, then compare, then commit, then save. rollback loads the archived configuration into the candidate and changes nothing until commit — and it reverts the entire tree, not only the line you were thinking about.
• ↶Anything committed with commit-confirm reverts itself when the timer expires. If verification will take longer than the window, extend the window rather than confirming early to buy time.
• ↶Removing a PBR rule is itself a service-affecting change for the traffic class it carried. There is no read-only way to withdraw a policy route; the matched traffic moves to whatever the main table says the moment the rule leaves the kernel.

6 · Escalation

When the runbook isn't enough, contact:

• · Escalate to the upstream that owns the next-hop when the rule is installed, the table has a route, and the next-hop does not answer. A hard-coded next-hop that has failed is not a routing bug on this box and no local change will restore the path.
• · Escalate to whoever owns the stateful firewall on the return path when the forward path is correct and the replies are dropped. Asymmetric PBR is a design conflict between two correct configurations, and it is resolved by agreement, not by a knob on this router.
• · Escalate to network engineering when the router has multiple route-maps bound to the same interface. Which of them wins is a property of the generated rule priorities rather than of operator intent, and re-ordering them is a design change with its own change window.
• · Escalate to network engineering when the estate has reached PBR-everywhere — more policy rules than the kernel walks cheaply, or a forwarding path that no one can predict from the configuration. That is a routing-design problem wearing an incident costume, and patching one more rule into it makes the next incident worse.
• · Escalate to the VRF owner when a PBR table ID collides with a VRF table ID. The two namespaces share the same numeric space and the resolution belongs to whoever planned the ID layout.
• · Escalate to the change owner when the policy is installed and working exactly as configured, and the configuration does not match what anyone expected. That is a requirements failure, and changing the router before the requirement is agreed just moves the argument to a worse place.
• · Hold rather than force: a policy route that is degraded but carrying traffic, with a named owner and an agreed end time, is a better position at 03:00 than a rushed change to a forwarding policy nobody has fully read.

References

1. VyOS — Policy route-map
2. FRRouting — route-map
3. FRRouting — zebra
4. ip-rule(8)
5. rt_tables(5)
6. RFC 3704 — Ingress Filtering for Multihomed Networks