PBR troubleshoot — show ip route, show ip rule, traceflow, asymmetry, counters

When a PBR deployment is not working — voice traffic is taking the data provider, data traffic is taking the voice provider, the rule never fires, the rule fires but the next-hop is unreachable — the operator must walk a deterministic ladder to find the failure. The ladder is: configuration, FRR render, kernel installation, route resolution, packet capture. This lesson walks through each step of the ladder, the operational commands the engineer uses at each step, and the production failure modes the engineer must recognise.

The diagnostic ladder

flowchart TD
  A[PBR not working] --> B{Configuration<br/>correct?}
  B -->|no| FIX1[Fix configuration]
  B -->|yes| C{FRR render<br/>correct?}
  C -->|no| FIX2[Fix VyOS-to-FRR mapping]
  C -->|yes| D{Kernel rule<br/>installed?}
  D -->|no| FIX3[Fix rule installation]
  D -->|yes| E{PBR table<br/>populated?}
  E -->|no| FIX4[Fix route generation]
  E -->|yes| F{ip route get<br/>resolves correctly?}
  F -->|no| FIX5[Fix next-hop or table]
  F -->|yes| G{Packet capture<br/>shows expected path?}
  G -->|no| FIX6[Fix forwarding or return path]
  G -->|yes| H[Investigate asymmetry<br/>or upstream]

The ladder is deterministic. Start at the top; if the configuration is wrong, no amount of packet capture will fix it. If the configuration is right but the FRR render is wrong, the configuration tree has a syntax issue. And so on down the ladder.

Step 1 — verify the configuration

The first check is the VyOS configuration itself. The operator must confirm:

• The route-map exists with the expected match and set clauses.
• The prefix-list exists with the expected prefixes.
• The interface binding exists on the correct interface.

vyos@vyos:~$ show policy route-map
Route-map RM-VOICE-OUT:
  rule 10 {
    action permit
    match {
      ip {
        source {
          address {
            prefix-list PL-VOICE-SRC
          }
        }
      }
    }
    set {
      ip {
        next-hop 198.51.100.1
      }
    }
  }

vyos@vyos:~$ show configuration commands | match 'policy route-map'
set policy route-map RM-VOICE-OUT rule 10 action 'permit'
set policy route-map RM-VOICE-OUT rule 10 match ip source address prefix-list 'PL-VOICE-SRC'
set policy route-map RM-VOICE-OUT rule 10 set ip next-hop '198.51.100.1'
set interfaces ethernet eth1 policy route-map 'RM-VOICE-OUT'

The configuration looks right. The route-map has the expected match and set clauses; the interface binding is on eth1. Move to Step 2.

Step 2 — verify the FRR render

The second check is what FRR actually saw after the commit. The operator must confirm:

• The FRR route-map block matches the VyOS configuration.
• The FRR ip rule line is present.
• The FRR ip route ... table <id> line is present.

vyos@vyos:~$ vtysh -c 'show running-config' | grep -A 4 'route-map RM-VOICE-OUT'
route-map RM-VOICE-OUT permit 10
  match ip source-address prefix-list PL-VOICE-SRC
  set ip next-hop 198.51.100.1

vyos@vyos:~$ vtysh -c 'show running-config' | grep 'ip rule'
ip rule 100 from 10.10.0.0/24 iif eth1 lookup 100

vyos@vyos:~$ vtysh -c 'show running-config' | grep 'table 100'
ip route 0.0.0.0/0 198.51.100.1 table 100

The FRR render matches the VyOS configuration. Move to Step 3.

Step 3 — verify the kernel installation

The third check is what the kernel actually installed. The operator must confirm:

• The kernel ip rule show contains the PBR rule at the expected priority.
• The kernel ip route show table <id> contains the expected routes.

vyos@vyos:~$ ip rule show
0:      from all lookup local
100:    from 10.10.0.0/24 iif eth1 lookup 100
32766:  from all lookup main
32767:  from all lookup default

vyos@vyos:~$ ip route show table 100
0.0.0.0/0 via 198.51.100.1 dev eth2

The kernel has the rule at priority 100 and the table populated. Move to Step 4.

If the rule is missing from ip rule show, the issue is between the FRR render and the kernel installation. The operator should check journalctl -u frr for netlink errors and verify that the zebra daemon is running:

vyos@vyos:~$ systemctl status frr
vyos@vyos:~$ vtysh -c 'show zebra client summary'

Step 4 — verify the route resolution

The fourth check is whether the PBR table actually produces the expected next-hop for a specific source and destination. The ip route get command with a from <source> argument resolves the route as if the packet had arrived from the specified source:

vyos@vyos:~$ ip route get 8.8.8.8 from 10.10.0.5
8.8.8.8 from 10.10.0.5 via 198.51.100.1 dev eth2 table 100

The route resolves to the voice provider (198.51.100.1) via eth2. This is the expected PBR next-hop. Move to Step 5.

If ip route get returns the main-table next-hop, the PBR rule is not matching. The reasons are:

• The source does not match the prefix-list (typo in the prefix).
• The iif (input interface) does not match the binding.
• The PBR table is empty (FRR did not install the route).

The operator must re-verify the prefix-list content and the interface binding:

vyos@vyos:~$ show policy prefix-list PL-VOICE-SRC
Prefix-list: PL-VOICE-SRC
rule 10 {
    action permit
    prefix 10.10.0.0/24
}

vyos@vyos:~$ show configuration commands | match 'policy route-map'
set interfaces ethernet eth1 policy route-map 'RM-VOICE-OUT'

The prefix-list has the right prefix; the binding is on eth1. The mismatch must be elsewhere — most likely a typo in the route-map’s match clause or the operator testing with a different source IP.

Step 5 — verify with packet capture

The fifth check is whether a real packet takes the expected path. The operator must capture packets on the egress interface while sending a packet from the matching source:

vyos@vyos:~$ tcpdump -ni eth2 -c 5 'src 10.10.0.5'
tcpdump: listening on eth2, link-type EN10MB (Ethernet), capture size 262144 bytes

In another terminal, the operator sends a packet:

vyos@vyos:~$ ping -c 3 -I 10.10.0.5 8.8.8.8

If the packet appears on eth2 (the voice provider interface), the PBR is working at the data plane. If the packet appears on eth0 (the data provider interface), the PBR is not working and the operator must investigate further.

For end-to-end verification, the operator can use tracepath or mtr to confirm the path:

vyos@vyos:~$ tracepath -n -s 10.10.0.5 8.8.8.8

The -s 10.10.0.5 argument sets the source address so the tracepath packets take the PBR path. The output shows the hops; the first hop should be the voice provider’s gateway.

Per-rule counters — confirming the rule is firing

FRR’s show route-map shows the per-rule counters. The counters increment for every rule evaluation; the operator can compare them to the expected traffic volume to verify the rule is firing.

vyos@vyos:~$ vtysh -c 'show route-map RM-VOICE-OUT'
route-map RM-VOICE-OUT, permit, sequence 10
  Match clauses:
    ip source-address prefix-list PL-VOICE-SRC
  Set clauses:
    ip next-hop 198.51.100.1
  Statistics:
    Matching: 1234 packets, 98765 bytes

The Statistics line shows the matching packets. If the counter is zero, the rule is not firing. The most likely cause is a typo in the prefix-list or the interface binding.

The operator can also use the FRR clear route-map counters command to reset the counters and observe fresh traffic:

vyos@vyos:~$ vtysh -c 'clear route-map counters'

After clearing, the operator sends known-good traffic from the matching source and checks the counter again.

Asymmetry detection — the forward path is not the return path

PBR can create asymmetric routing: the forward path uses the PBR rule, but the return path uses the main table. Stateful firewalls on the return path drop the packets because no state exists.

sequenceDiagram
  participant Src as Voice source<br/>10.10.0.5
  participant FW1 as Forward firewall
  participant V as Voice provider
  participant Dst as Destination<br/>8.8.8.8
  participant FW2 as Return firewall
  participant FW2b as Return path
  Src->>FW1: SYN (src=10.10.0.5, dst=8.8.8.8)
  FW1->>V: PBR: forward via voice provider
  V->>Dst: SYN forwarded
  Dst-->>FW2b: SYN-ACK (src=8.8.8.8, dst=10.10.0.5)
  FW2b->>Src: Return path via data provider<br/>(no state in FW1)
  Src-->>FW1: ACK to FW1 (but no SYN state)
  FW1-->>Src: RST — stateful drop

The detection pattern: capture packets on the egress interface of the forward path and on the egress interface of the return path. If the packets appear on different interfaces, the forward and return paths are asymmetric.

vyos@vyos:~$ tcpdump -ni eth2 -c 5 'src 10.10.0.5'
# capture forward path (voice provider)

vyos@vyos:~$ tcpdump -ni eth0 -c 5 'src 10.10.0.5'
# capture return path (data provider)

If the return path is on eth0 but the forward path is on eth2, the routing is asymmetric. The fix is either:

• Symmetric PBR: apply the same route-map to the return path.
• Source-based routing on the return path so the response takes the same provider.
• Disable stateful firewall on the affected interfaces.

Debug at the kernel level — netlink

When the kernel is rejecting the ip rule installation, the operator can enable kernel netlink debug:

vyos@vyos:~$ vtysh -c 'debug zebra kernel'
vyos@vyos:~$ journalctl -u frr -f

The debug output shows the netlink messages between zebra and the kernel. If the rule installation is failing, the output shows the netlink error (e.g. ESRCH for “no such process” if the table does not exist, EINVAL for “invalid argument” if the rule syntax is wrong).

The operator can also use strace to trace the netlink communication:

vyos@vyos:~$ strace -f -e trace=sendmsg,recvmsg -p $(pidof zebra) 2>&1 | grep netlink

The strace output shows the raw netlink messages between zebra and the kernel. This is the lowest-level diagnostic and is typically used only when the higher-level diagnostics do not identify the issue.

How the configuration is validated

The validation command set for PBR troubleshoot:

show policy route-map
show configuration commands | match 'policy route-map'
vtysh -c 'show running-config' | grep -E 'ip rule|table'
vtysh -c 'show route-map RM-NAME'
ip rule show
ip route show table all
ip route get <src> <dst>
tcpdump -ni <ifname> 'src <src>'
tracepath -n -s <src> <dst>

A working PBR deployment has:

• The route-map in show policy route-map with the expected match and set clauses.
• The interface binding in show configuration commands.
• The FRR render with ip rule and ip route ... table <id>.
• The kernel rule at the expected priority in ip rule show.
• The PBR table populated with ip route show table <id>.
• The ip route get from the matching source returns the PBR next-hop.
• The packet capture shows the packet on the expected egress interface.

How it fails

The production failure modes the engineer must recognise:

• Configuration typo. A typo in the prefix-list prefix or the interface name. The configuration is valid; the rule never matches.
• Missing interface binding. The route-map exists; the binding is missing. ip rule show does not contain the PBR rule.
• Empty PBR table. The rule is installed but the table has no routes. The kernel consults an empty table and falls through.
• Wrong direction. The operator intends input-side PBR but the binding is on the egress interface. The rule never fires.
• Asymmetric return path. The forward path uses PBR; the return path uses the main table. Stateful firewalls drop the return packets.
• Hard-coded next-hop down. The PBR rule hard-codes a next-hop that has failed. The packets are forwarded to the dead next-hop.

Rollback

The recovery from a bad PBR troubleshooting outcome:

• Wrong binding: delete interfaces ethernet <ifname> policy route-map <name>; set interfaces ethernet <correct-ifname> policy route-map <name>; commit; save.
• Empty table: set policy route-map <name> rule <n> set ip next-hop <reachable>; commit; save.
• Asymmetry: apply the same PBR rule on the return path; or disable stateful firewall on the affected interfaces.
• Whole-tree rollback: rollback N; commit; save.

Production discipline

Cross-course references

The Linux course’s XIX-Linux-NetFoundations covers the kernel FIB and the rule selector. The VyOS lessons vyos-xiii-01-pbr-concept, vyos-xiii-02-route-maps, vyos-xiii-03-pbr-rules, and vyos-xiii-04-pbr-ipv6 cover the PBR foundation this lesson assumes. The lessons vyos-xii-06-static-route-troubleshoot and vyos-xxxi-04-route-not-installed cover the static and dynamic route troubleshoot patterns this lesson extends.

Quiz