Static route troubleshooting — show ip route, traceroute, looking-glass, asymmetric routing

A static route is the most explicit form of forwarding information, and the easiest to debug — once the operator knows the command set. The trouble is that a static-route problem is rarely local. A misconfigured static route on router A can manifest as a connectivity failure on router B or as an asymmetric path that breaks stateful devices between them. The troubleshooting command set must cover the local view, the transit view, and the remote view. This lesson walks through the canonical commands, the traceroute-and-looking-glass pattern for forward-path validation, the return-path validation that catches asymmetric routing, and the post-incident evidence package the operator must collect.

The troubleshooting command set

flowchart TD
  A[Static route incident] --> B[Local view]
  A --> C[Transit view]
  A --> D[Remote view]
  B --> B1[show ip route]
  B --> B2[show ip route static]
  B --> B3[ip route show]
  B --> B4[show configuration protocols static]
  C --> C1[traceroute]
  C --> C2[traceroute -s <source>]
  C --> C3[tcpdump -ni any]
  D --> D1[looking glass]
  D --> D2[ping from destination]
  D --> D3[show ip route on remote]

The local view is what the local router knows. The transit view is what the routers on the path know. The remote view is what the destination network knows. A connectivity issue can be in any of the three; the operator must check all three to localise the issue.

The local view

The local view starts with show ip route — the FRR view of the RIB:

vyos@vyos:~$ show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - table, v - VNC, V - VNC-Direct, A - Babel,
       D - SHARP, F - PBR, f - OpenFabric,
       > - selected route, * - FIB route

S>* 10.20.0.0/16 [1/0] via 192.0.2.2, eth0, 00:00:12
S   10.30.0.0/16 [1/0] via 192.0.2.3, eth1 inactive
C>* 192.0.2.0/24 is directly connected, eth0

The S>* 10.20.0.0/16 is a static route in the FIB. The S 10.30.0.0/16 is a static route that is not in the FIB — the inactive keyword tells the operator the route is in the RIB but not in the FIB.

The show ip route static filter is the second command — it limits the output to static routes only:

vyos@vyos:~$ show ip route static
S>* 10.20.0.0/16 [1/0] via 192.0.2.2, eth0
S   10.30.0.0/16 [1/0] via 192.0.2.3, eth1 inactive

The kernel view is the third command — ip route show. This is the canonical source of truth for what the kernel is forwarding:

vyos@vyos:~$ ip route show
10.20.0.0/16 via 192.0.2.2 dev eth0 proto static metric 1
192.0.2.0/24 dev eth0 proto kernel scope link src 192.0.2.1

The route is in the kernel. The proto static confirms it was installed by the static routing source.

The configuration view is the fourth command — show configuration protocols static. This is what the operator typed:

vyos@vyos:~$ show configuration protocols static
route 10.20.0.0/16 {
    next-hop 192.0.2.2
    description "CR-1234 upstream-primary to dc01-eu"
}
route 10.30.0.0/16 {
    next-hop 192.0.2.3
    description "CR-1235 upstream-backup to dc01-eu"
}

The configuration is what the operator wrote. The RIB and FIB should match. If the configuration has a route that the RIB does not, the operator has a typo or a disable attribute. If the RIB has a route that the kernel does not, the operator has a netlink or zebra error.

The transit view — traceroute

The transit view is traceroute — the ICMP-based path discovery that shows every router on the path from source to destination:

vyos@vyos:~$ traceroute 10.20.0.5
traceroute to 10.20.0.5 (10.20.0.5), 30 hops max, 60 byte packets
 1  192.0.2.2 (192.0.2.2)  0.456 ms  0.412 ms  0.398 ms
 2  198.51.100.1 (198.51.100.1)  4.234 ms  4.198 ms  4.176 ms
 3  203.0.113.5 (203.0.113.5)  12.345 ms  12.312 ms  12.298 ms
 4  10.20.0.5 (10.20.0.5)  18.456 ms  18.412 ms  18.398 ms

Each line is a router on the path. The first line is the local router’s next-hop. The middle lines are transit routers. The last line is the destination. The timing columns are the round-trip time for each probe.

traceroute has limitations. The tool uses ICMP echo with incrementing TTL, or UDP with incrementing TTL depending on the implementation. The middle routers may be configured to not reply to ICMP, or to reply with a different source address. The output may show * * * for routers that do not reply, even when the path is correct.

The traceroute -s <source> option uses a specific source address. This is critical when the local router has multiple source addresses (e.g. a VRF-bound interface). The default source may not be the one the operator expects.

The remote view — looking glass

A looking glass is a public (or private) service that lets the operator query the routing table and BGP state of a remote network. Major providers (Cogent, Hurricane Electric, NTT) and most Regional Internet Registries (RIPE, ARIN, APNIC) run looking glasses.

A typical looking-glass query asks for the BGP route to a prefix from a specific vantage point:

> show route for 10.20.0.0/16 from AS64500
BGP routing table entry for 10.20.0.0/16
  192.0.2.2 from 192.0.2.2 (192.0.2.2)
    Origin IGP, localpref 100, valid, internal
    Last update: 2d12h ago

The looking glass returns the BGP route as seen from the remote vantage point. The operator can confirm that the remote network has a path to the prefix.

A ping from the destination is the most direct remote validation:

remote-host:~$ ping -c 3 10.20.0.5
PING 10.20.0.5 (10.20.0.5) 56(84) bytes of data.
64 bytes from 10.20.0.5: icmp_seq=1 ttl=58 time=12.3 ms
64 bytes from 10.20.0.5: icmp_seq=2 ttl=58 time=11.8 ms
64 bytes from 10.20.0.5: icmp_seq=3 ttl=58 time=12.1 ms

--- 10.20.0.5 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss

If the destination can ping the source, the forward path is working. If the destination cannot, the forward path is broken somewhere.

Detecting asymmetric routing

Asymmetric routing is the pattern where the forward path (source to destination) takes one route, and the return path (destination to source) takes a different route. The forward and return packets traverse different routers, different links, and possibly different providers.

Asymmetric routing is not always a problem. The Internet is inherently asymmetric; routes do not have to match in both directions. The trouble comes when a stateful device (firewall, NAT, load balancer) sits on the path and expects to see both directions.

flowchart LR
  A[Source 10.0.0.5] -->|forward via ISP-A| B[Destination 10.20.0.5]
  B -->|return via ISP-B| A
  C[Firewall on ISP-A path] -->|sees forward| A
  D[Firewall on ISP-B path] -->|sees return| B

The firewall on the ISP-A path sees the forward packets but not the return. The stateful firewall rejects the return packets because they do not match a state; the connection is broken.

The detection is two traceroute runs — one from each end:

source:~$ traceroute 10.20.0.5
 1  192.0.2.2 (ISP-A)
 2  ...
 3  10.20.0.5

destination:~$ traceroute 10.0.0.5
 1  198.51.100.1 (ISP-B)
 2  ...
 3  10.0.0.5

If the two traceroute paths differ, the routing is asymmetric. The fix depends on the cause:

• Different providers: design the topology so the stateful devices are on the shared path.
• Different static routes: align the static routes so the forward and return paths use the same next-hop.
• Different dynamic protocols: align the routing policy so the same prefix is preferred in both directions.

Return-path validation

The return-path validation is the most important troubleshooting step the operator often skips. The forward path is tested by ping from source to destination; the return path must be tested by ping from destination to source.

The failure modes that asymmetric routing exposes:

• Forward works, return fails. The source can reach the destination; the destination cannot reach the source. The return path has no route, has a blackhole, or has a firewall drop.
• Forward slow, return fast. The forward path is congested; the return path is uncongested. The two paths have different load.
• Forward via ISP-A, return via ISP-B. The two paths use different providers. A stateful device on one path does not see the other direction.

The test is two ping runs and two traceroute runs. The operator must compare the four outputs:

• ping from A to B
• ping from B to A
• traceroute from A to B
• traceroute from B to A

If any pair differs asymmetrically, the routing is asymmetric. The fix is to identify the cause and align the paths.

The post-incident evidence package

A static-routing incident that goes to a post-incident review must have an evidence package. The package includes:

• The local configuration (show configuration protocols static).
• The RIB view (show ip route).
• The FIB view (show ip route static).
• The kernel view (ip route show).
• The traceroute from both ends.
• The ping from both ends.
• The looking-glass query for the affected prefix.
• The change request or commit log for the affected route.
• The timestamp of the incident and the timestamp of the recovery.

The evidence package is what the next operator reads. A good evidence package is reproducible — the next operator can run the same commands on a similar incident and find the same evidence.

How it fails

The production failure modes the engineer must recognise:

• Route not in configuration. The operator never wrote the route. The RIB, FIB, and kernel all show no route for the prefix. The fix is to write the configuration.
• Route in configuration but not in RIB. A typo or a disable attribute. The fix is to correct the typo or remove the disable.
• Route in RIB but not in FIB. A zebra error or a netlink error. The fix is to check the FRR log for the specific error and address it.
• Route in FIB but next-hop unreachable. A recursive route with a missing or wrong interface address. The route is inactive. The fix is to add or correct the address.
• Route correct but asymmetric. The forward path works; the return path takes a different route. A stateful firewall on one path rejects the other direction. The fix is to align the paths.
• Route correct but PMTUD broken. The route works for small packets; large packets are silently dropped. A firewall on the path is dropping the ICMP-unreachable. The fix is to find the firewall and reconfigure it.
• Route correct but MTU mismatch. The route works for standard packets; jumbo frames are dropped. The fix is to align the MTU on the path or to clamp the MSS at the TCP layer.

Rollback

The recovery from a static-routing incident depends on the failure mode. The canonical rollbacks:

• Wrong route: delete protocols static route <prefix>; commit; save.
• Wrong distance: set protocols static route <prefix> distance <correct>; commit; save.
• Wrong VRF: delete protocols static route <prefix> vrf; set protocols static route <prefix> vrf <correct>; commit; save.
• Whole-tree rollback: rollback N; commit; save.

For emergency rollback, load <file>; commit; save to replace the candidate with a saved backup.

Production discipline

Cross-course references

The Linux course’s XXII-Linux-NetTroubleshoot covers the host-side equivalent of the troubleshooting command set. The BGP course’s XXXI-BGP-Troubleshooting covers the looking-glass and asymmetric-routing patterns in depth. The Observability course’s XLIX-Monitoring covers the metrics-side evidence that complements the routing diagnostics. The OPNsense course’s XII-OPNsense-StaticRoutes covers the equivalent troubleshooting workflow on the OPNsense platform.

Quiz