Routing table troubleshooting — ip route show table all, ip rule show, asymmetry

A multiple-routing-table deployment is one of the easiest to misconfigure and one of the hardest to debug if the operator does not know the right command set. The routes exist; the rules exist; the configuration committed; but the packet does not follow the path the operator expected. The cause is usually one of three things: a rule that points at the wrong table, a table that has no matching route, or an asymmetric return path. This lesson covers the troubleshooting command set that exposes all three, the ip route get validation that confirms the intended RPDB walk, and the asymmetry-detection pattern that catches the most common production failure.

The troubleshooting command set

flowchart TD
  A[Multi-table incident] --> B[Local view]
  A --> C[Forward path]
  A --> D[Return path]
  B --> B1[ip rule show]
  B --> B2[ip route show table all]
  B --> B3[ip route get]
  B --> B4[show ip rule]
  B --> B5[vtysh -c 'show ip rule']
  C --> C1[traceroute -s <src> <dst>]
  C --> C2[tcpdump -ni <iface>]
  D --> D1[traceroute from destination]
  D --> D2[looking glass query]

The local view is what the local kernel and FRR know. The forward path is what the Internet actually carries for traffic from a representative source. The return path is what arrives back from the destination. The operator must check all three to localise a multi-table incident.

The local view — ip rule and ip route show table all

The first command is ip rule show. It lists every rule in the kernel’s RPDB:

vyos@vyos:~$ ip rule show
0:	from all lookup local
100:	from 10.10.0.0/24 lookup isp-a
200:	from 10.20.0.0/24 lookup isp-b
32766:	from all lookup main
32767:	from all lookup default

The walk order is priority ascending. For any given packet, the kernel evaluates the rules in this order; the first matching rule determines the table.

The second command is ip route show table all. It lists the routes in every table:

vyos@vyos:~$ ip route show table all
table local:
local 192.0.2.1 dev lo proto kernel scope host src 192.0.2.1
broadcast 192.0.2.255 dev lo proto kernel scope link src 192.0.2.1
table main:
10.20.0.0/16 via 192.0.2.2 dev eth0 proto static metric 1
192.0.2.0/24 dev eth0 proto kernel scope link src 192.0.2.1
table isp-a:
default via 198.51.100.1 dev eth0 proto static metric 1
198.51.100.0/24 dev eth0 proto kernel scope link src 198.51.100.2
table isp-b:
default via 203.0.113.1 dev eth1 proto kernel scope link src 203.0.113.2
table default:

This is the canonical inventory of what the kernel knows. The operator should compare the inventory against the design expectations: every custom table should have routes; every referenced table should have at least one rule; every rule should have a matching table with matching routes.

The VyOS shell shows the same data through show ip rule and show ip route:

vyos@vyos:~$ show ip rule
0:	from all lookup local
100:	from 10.10.0.0/24 lookup isp-a
200:	from 10.20.0.0/24 lookup isp-b
32766:	from all lookup main
32767:	from all lookup default

The VyOS view and the kernel view should agree. If they do not, the cause is a script that bypassed VyOS or a failed commit that left the live state inconsistent.

The local view — ip route get

The third command is ip route get <dst> from <src>. It performs the full RPDB walk for the specified source and destination and returns the table, the next-hop, and the egress interface:

vyos@vyos:~$ ip route get 8.8.8.8 from 10.10.0.5
8.8.8.8 from 10.10.0.5 via 198.51.100.1 dev eth0 table isp-a uid 0
    cache

This is the most precise validation. The operator can issue the command from the router with any source address and confirm the table the kernel would consult, the next-hop it would use, and the interface it would egress on. If the output does not match the design, the rule chain is wrong.

ip route get is also useful with the iif parameter:

vyos@vyos:~$ ip route get 8.8.8.8 from 10.10.0.5 iif eth2

The iif parameter overrides the input interface selection and forces the RPDB walk to start as if the packet arrived on eth2. This is useful when the operator wants to validate the walk for a specific ingress without actually generating traffic.

The forward path — traceroute -s

The forward path validation is traceroute -s <src> <dst> from a host on the corresponding LAN. The -s option forces the source address; the host sends the probe packets with the specified source.

host-a:~$ traceroute -s 10.10.0.5 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max
  1  198.51.100.1 (198.51.100.1)  0.456 ms
  2  198.51.100.1.isp-a.net  4.234 ms
  3  ...
  4  8.8.8.8

The first hop is the ISP-A gateway. If the first hop is the ISP-B gateway, the source-based routing is not working — either the rule is wrong, the route is in the wrong table, or the return-path asymmetry is causing the connection to fail despite the correct forward path.

A second traceroute -s <src> from a host on LAN-B confirms the ISP-B egress:

host-b:~$ traceroute -s 10.20.0.5 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max
  1  203.0.113.1 (203.0.113.1)  0.498 ms
  2  ...
  3  8.8.8.8

The two traceroute outputs must show different first hops — one ISP-A gateway, one ISP-B gateway. If both show the same gateway, the source-based routing is broken at the local level.

The return path — asymmetry detection

The return path validation is the most important troubleshooting step the operator often skips. The forward path is tested by traceroute -s from the source LAN; the return path must be tested by traceroute from the destination back to the source.

remote-host:~$ traceroute 10.10.0.5
traceroute to 10.10.0.5 (10.10.0.5), 30 hops max
  1  203.0.113.1 (203.0.113.1)  0.412 ms   <- ISP-B!
  2  ...
  3  10.10.0.5

The return path goes via ISP-B, but the forward path went via ISP-A. The routing is asymmetric. A stateful device on either path sees only one direction of the flow; the connection is broken.

The detection is two traceroute runs:

host-a:~$ traceroute -s 10.10.0.5 8.8.8.8
  1  198.51.100.1 (ISP-A)
host-b:~$ traceroute -s 10.20.0.5 8.8.8.8
  1  203.0.113.1 (ISP-B)
remote-host:~$ traceroute 10.10.0.5
  1  203.0.113.1 (ISP-B)   <- mismatch
remote-host:~$ traceroute 10.20.0.5
  1  198.51.100.1 (ISP-A)  <- mismatch

If the return paths are swapped relative to the forward paths, the routing is asymmetric. The fix is upstream — BGP, source- address binding, or negotiated return routes — not at the local router.

Common failure modes — diagnosis

The production failure modes the engineer must recognise:

• Missing route in custom table. ip route show table isp-a shows no default. ip route get 8.8.8.8 from 10.10.0.5 returns the wrong table (falls through to main). Fix: install the default in the custom table.
• Rule priority wrong. ip rule show shows the custom rule at priority 32768 instead of 100. The walk order means main is consulted first. Fix: lower the priority.
• Source selector wrong. ip rule show shows from 10.10.0.0/24 but the LAN is 10.10.0.0/16. Fix: correct the prefix.
• Table referenced but no routes. The rule references isp-a; the table has no routes; the lookup misses; the walk falls through. Fix: install routes in the table.
• Return-path asymmetry. traceroute -s exits via ISP-A, but traceroute from the destination returns via ISP-B. Fix: upstream — BGP announcement, source-address binding, or negotiated return routes.
• Router-originated traffic fails. BGP sessions from the router fail. The router’s source address is not in any custom rule. Fix: add a default in main or a rule for the router’s source.
• Alias missing. ip route show table <name> returns Invalid argument. The rt-table declaration is missing or malformed. Fix: declare the alias in set system ip rt-table.

The post-incident evidence package

A multi-table incident that goes to a post-incident review must have an evidence package. The package includes:

• ip rule show — the kernel RPDB at the time of the incident.
• ip route show table all — the full route inventory across every table.
• show ip rule (VyOS view) — the configuration-derived view.
• show configuration system ipv4 rule — the rules in the configuration tree.
• show configuration system ip rt-table — the rt-table declarations.
• show configuration protocols static — the static routes (in the main table by default; in custom tables if configured).
• ip route get <dst> from <src> — the RPDB walk output for the affected traffic.
• traceroute -s <src> <dst> from the source LAN — the forward path.
• traceroute from the destination — the return path.
• /var/log/frr/zebra.log — the zebra log for the affected period.
• The change request or commit log for the affected deployment.
• The timestamp of the incident and the timestamp of the recovery.

The evidence package is what the next operator reads. A good evidence package is reproducible — the next operator can run the same commands on a similar incident and find the same evidence.

How it fails

The production failure modes the engineer must recognise:

• Configuration not committed. The operator set the rules and routes in configure mode but did not commit. The show configuration shows the changes; ip rule show does not. Fix: commit; save.
• Configuration not saved. The operator committed but did not save. A reboot reverts to the saved config; the custom tables and rules disappear. Fix: save.
• Script bypassed VyOS. A startup script added rules and routes outside of VyOS. The kernel sees them; FRR does not. After a commit; save and reload, the script’s state is lost. Fix: remove the script and add the configuration to VyOS.
• Table ID collision. The operator declared a custom table at an ID that a VRF is already using. The commit fails with an error; the configuration is in the candidate but not active. Fix: choose a different ID.
• Wrong interface in connected route. The static route in the custom table has interface eth0 but the egress should be eth1. The route installs; the egress is wrong. Fix: correct the interface.

Rollback

The recovery from a multi-table incident depends on the failure mode. The canonical rollbacks:

• Wrong rule: delete system ipv4 rule <priority>; commit; save.
• Wrong route in custom table: delete protocols static table <name> route <prefix>; commit; save.
• Wrong rt-table declaration: delete system ip rt-table rt-table-id <id>; commit; save.
• Whole-tree rollback: rollback N; commit; save.

For emergency rollback, load <file>; commit; save replaces the candidate with a saved backup. For live-state rollback (a script that bypassed VyOS), the operator must remove the offending state with ip rule del and ip route del and re-apply the correct configuration through VyOS.

Production discipline

Cross-course references

The Linux course’s XXII-Linux-NetTroubleshoot covers the host- side equivalent of the troubleshooting command set. The PBR course’s XIII-VyOS-PBR covers the policy-routing troubleshooting in the broader sense. The BGP course’s XXXI-BGP-Troubleshooting covers route-selection failures that affect multi-table deployments where BGP provides the return path.

Quiz