Recursive Next Hop Unreachable · Break/Fix · VyOS

Reported symptoms

core1 is the DC-A distribution router. It sits in OSPF area 10 with two transit links towards the core, and it carries one static route that matters here: the partner supernet 10.0.0.0/8, pointed at 192.0.2.201. That address is a loopback on pe1, the partner-facing router in area 0. The static is recursive on purpose — pointing at a loopback rather than at an interface means the traffic follows whichever transit link the IGP currently prefers, and survives losing either one.

On Tuesday at about 06:45, every application in DC-A that talks to the partner estate began timing out. Timing out, not being refused: the sessions hang until the client gives up, which is what a far-end problem looks like.

Three more facts arrived over the next two hours and none of them pointed here:

The partner says their side is healthy. More usefully, their edge firewall shows no inbound connection attempts from us at all. Not blocked ones — none.
Security opened an unrelated ticket the same morning. The internet edge router is logging a steady stream of dropped egress packets from DC-A hosts to 10.x destinations. Read cold, that looks like something in DC-A beaconing to private address space.
Nothing changed on core1. show system commit goes back six weeks, and the routing team is confident, correctly, that they did not touch it.

And on the router itself, two checks were run early and both passed. The route is in show ip route 10.0.0.0/8. The next hop answers ping. By 09:00 the working theory was a partner-side or transit-side problem, because the local routing had been confirmed twice.

Evidence provided

Read-only / Safethe route everybody confirmed was present — read the end of the line

vyos@core1:~$ show ip route 10.0.0.0/8

Codes: K - kernel route, C - connected, S - static, R - RIP,
     O - OSPF, B - BGP, > - selected route, * - FIB route

S   10.0.0.0/8 [1/0] via 192.0.2.201 inactive

Illustrative output

Read-only / Safethree statics carry the selected-and-installed markers; one does not

vyos@core1:~$ show ip route static

S>* 172.16.40.0/22 [1/0] via 198.51.100.1, eth1, 41w2d
S>* 172.16.44.0/24 [1/0] via 198.51.100.1, eth1, 41w2d
S>* 192.168.90.0/24 [1/0] via 172.16.8.9, eth3, 22w4d
S   10.0.0.0/8 [1/0] via 192.0.2.201 inactive

Illustrative output

Read-only / Safethe check that closed the investigation for two hours

vyos@core1:~$ ping -c 3 192.0.2.201

PING 192.0.2.201 (192.0.2.201) 56(84) bytes of data.
64 bytes from 192.0.2.201: icmp_seq=1 ttl=63 time=0.784 ms
64 bytes from 192.0.2.201: icmp_seq=2 ttl=63 time=0.731 ms
64 bytes from 192.0.2.201: icmp_seq=3 ttl=63 time=0.744 ms

--- 192.0.2.201 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss

Illustrative output

Read-only / Safethe longest match for the next hop, and there is nothing more specific

vyos@core1:~$ show ip route 192.0.2.201

Routing entry for 0.0.0.0/0
Known via "ospf", distance 110, metric 10, best
Last update 02:41:19 ago
* 198.51.100.1, via eth1

Illustrative output

Read-only / Safeone Type-3 LSA in the area; the archived capture from last month listed sixty-one

vyos@core1:~$ show ip ospf database summary

       Summary Link States (Area 0.0.0.10)

Link ID         ADV Router      Age  Seq#       CkSum  Route
0.0.0.0         198.51.100.1    141  80000001   0x4d2a 0.0.0.0/0

Illustrative output

Read-only / Safewhere partner traffic is actually going, according to the thing that decides

vyos@core1:~$ ip route get 10.4.19.20

10.4.19.20 via 198.51.100.1 dev eth1 src 198.51.100.2 uid 1000
  cache

Illustrative output

Read-only / Safea different router, a different team, one minute before the first timeout

vyos@abr1:~$ show system commit | head -3

0   2026-08-18 06:40:22 by platform via cli   PLT-2291 shrink area 10 LSDB
1   2026-07-29 15:11:07 by platform via cli   ntp source change
2   2026-07-14 09:02:44 by netops via cli     add area 10 interface eth7

Illustrative output

Work the evidence before reading on

Two commands disagree about the same address, and both are correct.

Look at the first output again, character by character. What do the > and * markers mean, which of them is on the partner supernet line, and what is the last word on that line? “The route is in show ip route” answers a question — is it the question that was asked?
ping 192.0.2.201 succeeds and show ip route 192.0.2.201 returns the default route. Both are true at once. What operation does ping perform on that address, what operation does a recursive static perform on it, and are they the same operation?
The area’s link-state database went from sixty-one summary LSAs to one. Which routes did core1 lose, and which single route replaced all of them?
Traffic to the partner did not stop being generated. If the supernet is not in the FIB, what does a packet to 10.4.19.20 match instead, and does that explain the security ticket that arrived at 06:41?

Then the question that resolves it: the router can clearly send a packet to 192.0.2.201, so why will it not use 192.0.2.201 as a next hop?

Root cause

1. The area became totally stubby, and that is an ABR-only change

Area 10 has been a stub area for a long time; every router in it agrees on that, which is what the E-bit in the Hello packets negotiates. On Tuesday the platform team added no-summary on the area border router to shrink the link-state database on the branch routers that share the area.

That converts a stub area into a totally stubby area, and it is legal to configure on the ABR alone — the internal routers need no change and their adjacencies are unaffected, which is exactly why nobody thought of it as touching core1. What it does is stop the ABR flooding Type-3 summary LSAs into the area, with one exception: the default route. Sixty-one inter-area prefixes became one.

Among the sixty-one was 192.0.2.201/32, the loopback the partner supernet’s static route points at.

2. Resolution and reachability are different operations

A recursive static route has a next hop and no egress interface. Zebra has to find the egress by looking the next hop up in the routing table, and it applies a rule that surprises people the first time they meet it: it will not resolve a next hop through a default route.

That rule is deliberate. A default route says “everything I do not otherwise know about goes this way”, which is a statement about last resort, not about where a specific host is. Allowing recursion through it would mean a static route whose next hop is a typo — or a loopback that has been withdrawn — resolves anyway, installs anyway, and quietly forwards its traffic at the default gateway. The router refuses, marks the route inactive, and leaves it in the RIB where you can see it.

Forwarding does none of that. When you type ping 192.0.2.201, the kernel does a longest-prefix match against the FIB, matches 0.0.0.0/0, and sends the packet to the default gateway, which is in area 0 and knows exactly where pe1 is. The replies come back in under a millisecond. Every packet the next-hop test sends arrives, because the default route is a perfectly good route — for forwarding.

So the ping was not a lucky false negative. It was a correct answer to a different question. Reachability was never in doubt; resolvability was the whole fault.

3. `inactive` means in the RIB and not in the FIB

The route stayed in show ip route because the RIB holds what the router has been told. It never reached the FIB because zebra could not resolve it, and show ip route says so twice: the > and * markers are missing, and the line ends in the word inactive.

Three engineers looked at that output and read the prefix, which was the thing they were looking for. The markers are two characters wide and sit where the eye is not.

4. The traffic did not stop — it left

This is the part with the second ticket attached.

With the supernet out of the FIB, a packet to 10.4.19.20 no longer matches a /8. It matches the next-longest thing in the table, which is now the freshly injected default, and follows it towards the core and out through the internet edge. There the egress filter drops it as private-space destined traffic and logs it, which is correct behaviour and is why the estate did not actually leak partner traffic onto the internet.

It also means the fault presented in two places at once, in two tickets, owned by two teams, with no shared field between them except a timestamp.

Under the hood

Zebra resolves a route’s next hop by walking the same table it builds, and it asks three questions in order.

Is there a route covering the next-hop address? If not, the route is inactive and nothing else matters. Is that covering route the default? If so, refuse, unless the operator has explicitly enabled resolution via the default route — FRR exposes that as ip nht resolve-via-default, and it is off by default for the reason described above. Is the covering route itself resolved? Recursion can chain, and each level must terminate at a route with a real egress interface.

Only when all three pass does the route get an egress and enter the FIB, and only then does show ip route print > and * in front of it. The two markers mean different things and it is worth keeping them apart: > is the route zebra selected among the candidates for that prefix, and * is the one the kernel actually installed. A route can be selected and fail to install; a route that is neither, like this one, never got as far as being a candidate.

The same resolution machinery decides whether a BGP path is installable, which is why “the session is up and the prefix is in show ip bgp but not in show ip route” is the same fault wearing a different protocol. In both cases the fix is upstream of the route you are looking at: something has to put a covering route in the table.

Proving reachability when the question was resolvability

The next-hop ping is the most-run check in network troubleshooting and it is the one that failed here, because it was never testing the thing that had broken. A successful ping proves the FIB can carry a packet to an address. It proves nothing about whether the RIB can use that address as a next hop, and on this router those two facts had been different since 06:40.

The shape recurs well outside routing. A file that exists and is not read. A firewall rule that is present and sits below a terminating match. A DNS record that resolves from your laptop and not from the pod. In every case a check passes because it asks the subsystem that is working.

The discipline is to verify at the layer that makes the decision, and to say out loud which layer that is before you run anything. Here that is one command: show ip route 10.0.0.0/8, read to the end of the line. Ten seconds, and it was run at 07:10 by somebody who was looking for the prefix and found it.

Resolution

Correlate the two tickets before you change anything. The security ticket and the partner ticket are the same incident, and knowing that tells you the traffic is leaving the estate rather than being dropped locally — which is what decides whether this waits for a window.
Establish what else in the area depended on an inter-area prefix. core1 is one router in area 10, and every other router in it lost the same sixty summaries at the same moment. Check their static routes and any next hop that is not on a connected segment before you declare the blast radius to be one prefix.
Decide with the platform team, not instead of them. Reverting area-type stub no-summary on abr1 restores the summaries and fixes this within a convergence interval, but it undoes a change that was made for a real reason. That is a conversation, and it is a short one.
If the immediate revert is the agreed path, apply it on abr1 under commit-confirm, and watch show ip ospf database summary on core1 repopulate before you confirm. The routers inside the area need no change at all.
For the durable fix, move the dependency to where the information lives: carry the partner supernet static on abr1, which is in area 0, resolves 192.0.2.201 natively, and is already the router that area 10 default-routes towards. Area 10 keeps its small database and core1 carries no static at all.
Refuse the interface pin. next-hop 192.0.2.201 interface eth1, or on-link, asserts that the next hop is directly attached to that segment. It is a loopback two hops away. The route will install, the traffic will still not arrive, and the diagnostic evidence will be gone.
Refuse the helper static. Adding a route for 192.0.2.201/32 makes the recursion resolve, and it pins the supernet to one transit link at administrative distance 1 — precisely the pinning the recursive design existed to avoid. It will hold the supernet active and black-holing whenever that link is up and pe1 is not.
Refuse relaxing next-hop resolution. Permitting recursion via the default route would fix this static and arm every other one on the box, turning a loud local failure into silent egress. That is the symptom already costing you a security ticket, applied estate-wide.
Close the loop on the change record for PLT-2291, naming what the area-type change actually affected. The next person to shrink an LSDB should find this.

Verification

show ip route 10.0.0.0/8 shows S>* with a resolved egress interface after the via. The markers are the test; a line of output containing the prefix is not.
ip route get for a real partner address returns the transit interface, not the default gateway. The kernel decides where packets go, so the kernel is what you ask.
The check can fail. In the lab, withdraw the route covering the next hop and watch the static drop back to inactive, so you know what the failing state looks like on the command you are now relying on.
The partner sees inbound sessions. Their firewall counters are the only evidence independent of the estate that produced the fault, and "no inbound attempts at all" was the sharpest symptom in the whole incident.
The internet edge egress filter stops logging drops to 10.0.0.0/8. That log going quiet is the proof the leak closed, and it is observable from a router nobody touched.
Every other static on every router in area 10 shows S>*. One inactive route was the visible fault; the area change had the same reach on all of them.
The recursion still does its job. In a window, fail one transit link and confirm the supernet follows the IGP to the other. A recursive route nobody has watched move is an intention, not a verified behaviour.

Prevention

Alert on inactive statics. A static route exists to forward traffic, so one sitting in the RIB without a * is a defect whatever caused it. It is a single check against show ip route static, and it would have raised this at 06:41 rather than at 09:20.
Treat an OSPF area-type change as a forwarding change on every router in the area. no-summary is legal on the ABR alone and its blast radius is every route every internal router was resolving through an inter-area prefix, so the pre-change capture belongs on the internal routers and the post-change check is a count of inter-area routes on each of them.
Do not resolve a next hop across a boundary whose summarisation you do not own. Either the next hop is visible from inside the area, or the route belongs on a router where it is.
Make egress filters alert, not merely drop. Internal destinations arriving at the internet edge means something upstream has lost a route, and that signal existed two hours before anybody correlated the two tickets.
Separate the two questions permanently, in habit and in runbooks. “Can I reach it” and “can the router resolve it” are different commands, and only the second one is about this class of fault.

What if area 10 had never carried a default route?

The static would still have gone inactive — the covering route would have disappeared either way — but the incident would have been an hour long instead of a morning, and security would never have been involved.

With no default, a packet to 10.4.19.20 matches nothing. The kernel drops it locally and returns an ICMP network-unreachable, so the applications fail immediately and loudly with “no route to host” rather than hanging, and the failure names the router that is failing. Nothing leaves DC-A, so there is no egress-filter log, no beaconing-shaped security ticket, and no second team investigating in parallel.

The default route did not cause this fault. It did two things to it, both unhelpful: it swallowed the traffic so the failure became silent, and it made the next hop pingable so the check everybody ran came back green. A default route is a floor under your routing, and a floor is also the thing that stops you noticing you have fallen.

That is worth carrying into any estate that injects a default into an access area. It is the right design for reaching the internet and it is the reason a missing internal route stops being obvious, so the compensating control — alerting on inactive routes, or on internal destinations at the egress filter — is not optional decoration.

Why this matters

Recursive next hops are worth the trouble: pointing a route at a loopback instead of an interface is what lets the traffic follow the IGP across whichever transit link is alive, and it is a better design than the pinned alternative on every day except this one. The price is a dependency on another route continuing to exist, and that dependency is invisible in the line you typed.

The transferable habit is to read show ip route as three separate facts rather than one. Is the prefix there — is it selected — is it installed. The output puts all three on the same line and only the first one is the one people look for, which is why “the route is in the table” is among the most confidently wrong statements available in network operations.

And the boundary is the other lesson. This change was made by a competent team, on a different router, following a sound principle, and it was correct in isolation. It broke core1 because core1 was resolving something across a boundary that the ABR is entitled to summarise away, and nothing in either router’s configuration recorded that relationship. When a change is legal to make on one device and its effects land on another, the only thing standing between you and this incident is whether somebody took a before-and-after capture on the device that was going to be affected.

Reported symptoms

Evidence

Root cause

Remediation

Verification

Prevention

Reported symptoms

Evidence provided

Work the evidence before reading on

Root cause

1. The area became totally stubby, and that is an ABR-only change

2. Resolution and reachability are different operations

3. inactive means in the RIB and not in the FIB

4. The traffic did not stop — it left

Resolution

Verification

Prevention

3. `inactive` means in the RIB and not in the FIB