low riskinformational~45 min

Runbook: Troubleshoot a Missing Route

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· The destination is written down as a prefix, not as an application. "Users cannot reach the reporting system" is a report; 10.20.4.0/22 from 192.168.10.0/24 is a symptom you can run commands against, and converting one into the other is the first five minutes of this job.
· Which router is expected to ORIGINATE this route, and by which mechanism — static, OSPF, BGP, connected, redistribution — is established before any command is run. A surprising share of "missing route" tickets are routes that were never configured to exist on the router being examined.
· Whether this ever worked is established, and if so when it stopped. A prefix that has never been reachable is a configuration gap; one that stopped is a withdrawal, a filter change, or a next-hop that went away, and those have different first commands.
· The scope is established: one prefix, every prefix from one neighbour, or everything. One prefix points at policy or origination; everything from one neighbour points at the session; everything points at this router.
· The change record for the last 72 hours is checked on this router AND on the expected originator, including route policy, prefix lists, redistribution, interface addressing and any VRF work.
· The pre-diagnosis routing state is captured before anything is changed: show ip route summary, the relevant show ip route output and show configuration commands for the protocol involved, saved into the ticket.
· Out-of-band console access is confirmed if there is any chance a fix will touch the path the operator is connected over. A missing-route ticket that turns into a default-route change is exactly that case.
· It is agreed up front that adding a static route to make the symptom go away is a decision with an owner and an end date, not a diagnostic step. It hides the fault, wins over the dynamic route on administrative distance, and outlives everyone who remembers why it exists.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1State the destination as a prefix and the source it will be reached from. Everything below is a question about one specific lookup, and a lookup needs both.
2Ask the router what it would actually do, rather than reading the table and interpreting it: show ip route 10.20.4.7 performs a destination lookup and returns the entry that would be used. show ip route 10.20.4.0/22 is a different question — it asks for that exact prefix and returns nothing when a less specific route is doing the forwarding.
3Decide which of three states you are in, because they have nothing in common. ABSENT: no entry covers the destination except a default, or nothing at all. PRESENT BUT NOT CHOSEN: an entry exists without the selected marker, or reads inactive. PRESENT AND CHOSEN: the route is there and traffic still fails.
4Cross-check the four views for the prefix, in this order, and stop at the first pair that disagrees: configuration (show configuration commands | match ...), the FRR RIB (show ip route), the kernel FIB (ip route show), and the lookup the kernel actually performs (ip route get). Each adjacent pair that disagrees names a different fault.
5Confirm you are looking at the right table before concluding the route is absent. A route in a VRF is invisible to a default-table lookup: show ip route vrf VRFNAME. A route in a policy-routing table is invisible to both: show ip rule then ip route show table all.
6If the route is genuinely absent, go to the mechanism that should originate it and use its own diagnostic. Static: configuration, then next-hop reachability. OSPF: is it in the database but not the RIB, or not in the database at all. BGP: was it not received, not selected, or not installed — three different faults with three different evidence trails.
7For a static route that is configured and not installed, check the next hop rather than the route: a recursive next hop that no longer resolves leaves the route in the RIB marked inactive and out of the FIB, and the route itself looks perfect.
8For OSPF, separate the two layers before touching anything: show ip route ospf for the RIB view and show ip ospf database for the link-state database. In the database but not the RIB is summarisation, area type or filtering. Not in the database is an origination or flooding problem on another router, and nothing you do here will fix it.
9For BGP, walk the three points in order: show ip bgp neighbors NEIGHBOUR received-routes for what arrived, show ip bgp 10.20.4.0/22 for whether it was selected, and show ip route 10.20.4.0/22 for whether it was installed. Read the flags — a route marked as not valid usually means its next hop is unreachable, which is an IGP fault wearing a BGP costume.
10If the route is present and chosen and traffic still fails, stop looking at this prefix and look at three other things: a more specific route sending the traffic elsewhere, the return path, and any blackhole or reject route that is doing exactly what it was configured to do.
11Prove the return path rather than assuming it. Run a trace from each end and compare. Forward success with return failure is the signature that sends operators to the firewall for a routing fault.
12Name the fault as a layer before proposing a fix: origination, propagation, policy, selection, installation, or return path. The fix belongs to whichever runbook owns that layer, and it is applied with commit-confirm and a verification plan, not typed into a live router because the ticket is old.
13Re-verify with the same destination lookup used in step 2, from the same source, and then with one real application flow end to end. A restored routing table is not a restored service.

4 · Verification

Confirm the procedure actually fixed the problem.

✓show ip route 10.20.4.7 — the destination lookup, not the prefix lookup — returns an entry that is both selected and in the FIB, via the next hop the design intends.
✓The four views agree for the prefix: it is in the configuration or the expected protocol, in the FRR RIB, in the kernel FIB from ip route show, and ip route get for the real source and destination returns the same egress.
✓The route is in the table the traffic will actually consult — confirmed against show ip rule where policy routing exists, and against show ip route vrf VRFNAME where the interface is bound to a VRF.
✓No more specific prefix is sending this traffic somewhere else. Longest-prefix match runs before administrative distance, so a /24 from another source silently beats the /22 you just fixed.
✓The next hop is reachable in its own right, and for a recursive next hop the underlying route that resolves it is present and stable.
✓The return path is verified from the far end, not inferred. A trace from each end, compared, is the evidence; a successful ping from one side is not.
✓One real application flow succeeds from a host on the source network to a host on the destination network, not only a router-sourced ping.
✓If a static route was added as a temporary measure, it is recorded in the ticket with an owner and a removal date, and the underlying fault has its own ticket.
✓The diagnosis in the ticket names the layer that failed — origination, propagation, policy, selection, installation, return path — and the command output that established it.

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶The diagnosis in this runbook is entirely read-only. Nothing needs rolling back until a fix is applied, and the fix belongs to the runbook for the mechanism involved.
↶Any fix applied during this investigation should have been applied with commit-confirm 10. If it made things worse and the window has not expired, do nothing: the router reverts on its own and that is the cheapest rollback available.
↶A committed and confirmed fix is reverted by deleting the specific nodes that were added, verified with compare before commit. Prefer that over a whole-configuration revert, which undoes work other people did in the same window.
↶A temporary static route added to restore service is the item most likely to be forgotten. It must be removed as part of closing the underlying fault: delete protocols static route 10.20.4.0/22, then commit and save, then re-verify that the intended dynamic route is the one being used.
↶If a BGP session was cleared during the investigation, note it. A hard clear tears down the session and withdraws every prefix learned from that peer for the duration of the re-establishment; it is a service event, not a diagnostic.
↶Any temporary firewall or policy change made to test a hypothesis is reverted in the same session it was made in, and its removal is verified rather than assumed.

6 · Escalation

When the runbook isn't enough, contact:

· Escalate to the owner of the originating router when the evidence shows the route is not being advertised. Once you can demonstrate the prefix is absent from what arrived — an empty received-routes for that prefix, or an absent LSA — the fault is upstream and no local change can create the route honestly.
· Escalate to the network engineering owner of the route policy when a prefix is arriving and being filtered. Prefix lists, route maps and area filters are usually shared, deliberate and load-bearing; widening one to fix a single prefix during an incident is how a leak happens.
· Escalate to the IGP owner when a BGP route is valid but not installed because its next hop is unreachable. That is an IGP convergence or filtering fault, it will affect far more than the prefix you were paged about, and it is fixed in the IGP.
· Escalate to the far-end network owner when the forward path is proven and the return path is not. Asymmetric or absent return routing cannot be diagnosed or fixed from one end, and stateful devices in the path will keep breaking connections until it is.
· Escalate to the change owner when the timeline points at a specific change and the change owner is not you. Reverting a route policy owned by someone else during an incident, without their context, replaces a known fault with an unknown one.
· Hold rather than improvise when the only available fix is a static route that overrides a dynamic protocol. Record the hold with an owner and an end time. If service pressure forces the static in anyway, that is a decision with a name attached and a removal date, not a fix.

“The route is missing” is a claim about a table. On a VyOS router there are four tables that can hold, or fail to hold, an answer for the same prefix, and they can disagree with each other in ways that each mean something different:

View	Command	What a disagreement with the view above means
What was configured or learned	`show configuration commands`	—
The FRR RIB	`show ip route`	Configured but not in the RIB: a typo, a disabled node, or a protocol that rejected it
The kernel FIB	`ip route show`	In the RIB but not the kernel: the route was not installed, usually an unresolved next hop
The lookup actually performed	`ip route get`	In the kernel but not used: policy routing, a VRF, or a more specific route winning

Walking those four in order is most of this runbook. The other part is the case that catches experienced operators: the route is present, correct, selected and installed, and the traffic still fails — because something more specific is winning, or because nothing is coming back.

When this runbook applies

A prefix that is expected in the routing table and is not there.
A prefix that is in the table and is not being used.
Traffic to a destination failing while the routing looks correct.
A route that disappeared after a change, a reconvergence, or a link event.

When it does not

A destination that has never been designed to be reachable from here. That is a design request, not an incident, and adding a route to satisfy it during a page is how estates acquire routing nobody can account for.
Traffic being dropped by a firewall or a NAT rule. The routing table is the wrong subsystem; the symptom overlaps but the evidence does not. Prove the route is present and chosen first, then move on.
A BGP session that is not up. Fix the session; the routes follow. The session runbook is the shorter path.

Blast radius

None, until the fix. Every command in this runbook reads state. That is worth stating explicitly, because it makes the sequencing free: there is no reason to skip a step to save time, and no reason to reach for a change before the evidence is complete.

The risk in a missing-route incident is concentrated in exactly one action.

A static route added to make the symptom go away is not a fix

It is the most available action in the room and the most expensive one.

A static route sits at administrative distance 1. OSPF sits at 110, iBGP at 200, eBGP at 20. So a static added during an incident does not co-exist with the dynamic route you were trying to restore — it beats it, permanently and silently. When the dynamic route comes back, traffic stays on the static. When the static next hop later fails, the router has a perfectly good dynamic route it will not use.

It also removes the symptom that was going to lead you to the cause. The ticket closes, the underlying fault is still there, and the next person meets it with one more undocumented static route in the way.

If service pressure genuinely requires it, that is a decision with an owner and a removal date written into the ticket, and the underlying fault keeps its own ticket. It is not a diagnostic step and it does not close anything.

Step 0: convert the report into a lookup

A routing lookup needs a destination and a source. “The reporting system is unreachable” gives you neither. Establish the destination prefix, the source prefix, and — before anything else — which router is supposed to originate this route and by what mechanism.

That last question is not bureaucracy. If nobody can say where the route is meant to come from, the most likely answer is that it was never configured, and every minute spent examining this router’s RIB is spent in the wrong place.

Step 1: ask the router what it would do

There are two different questions here and they are easy to confuse.

Read-only / Safedestination lookup, then exact-prefix lookup

# What would this router do with a packet for this address?
show ip route 10.20.4.7

# Does this exact prefix exist as an entry?
show ip route 10.20.4.0/22

# Everything static, and the whole table's shape
show ip route static
show ip route summary

The destination form returns whatever entry would actually be used, including a less specific route or the default. The prefix form asks for that exact entry and returns nothing when a supernet is doing the forwarding. An operator who runs only the prefix form on a router that is forwarding through a /16 concludes the route is missing when the traffic is being forwarded perfectly well — somewhere else.

Read the result for which of three states you are in:

What you see	State	Where to go
Nothing, or only a default	Absent	Step 3, then the originating mechanism
An entry without the selected marker	Present, not chosen	Another source won, or it is `inactive`
An entry marked `inactive`	Present, not installable	The next hop does not resolve
Selected and installed, traffic still fails	Present and chosen	Step 5

inactive is the most informative of these and the least noticed. It means the route is in the RIB and the router will not install it because the next hop cannot be resolved. The route itself is perfect; the thing it depends on is not. That is a very different fault from a route that was never configured, and the fix is somewhere else entirely.

Step 2: the four views, in order

Read-only / Safeconfiguration, RIB, kernel, actual lookup

show configuration commands | match "protocols static"
show ip route 10.20.4.0/22

Read-only / Safethe kernel's own answer

# Substitute your own values before running:
DST=10.20.4.7
SRC=192.168.10.5

ip route show
ip route get "$DST" from "$SRC"

ip route get is the command that ends arguments. It performs the lookup the kernel would perform for that source and destination — policy rules, VRFs, recursion and all — and returns the egress it would use. When the RIB says one thing and ip route get says another, the difference is not a bug, it is policy routing or a VRF, and step 3 is where you find it.

Why a route can be in the RIB and not in the FIB

FRR holds the RIB: every candidate route from every source. The kernel holds the FIB: the routes it will actually forward with. Installation from one to the other is not automatic — a route is installed only if it wins selection and its next hop resolves.

A recursive static route makes this visible. set protocols static route 10.20.0.0/16 next-hop 192.0.2.99 names a next hop with no egress interface. The kernel must look 192.0.2.99 up in the table, find a route that reaches it, and adopt that route’s egress. If the connected route to 192.0.2.0/24 disappears — an interface address removed, an interface down — the recursion fails, the route is marked inactive, and it leaves the FIB.

The route did not change. Nothing in show configuration changed. What changed is a different route, one nobody was looking at, and that indirection is why “check the next hop, not the route” is the highest-yield instinct in static-route diagnosis.

Step 3: are you even looking at the right table?

This step exists because skipping it produces confident wrong answers. A route in a VRF is invisible to a default-table lookup, and a route in a policy-routing table is invisible to both.

Read-only / Safethe other tables

# VRF-bound interfaces have their own routing table
show ip route vrf VRFNAME

# Policy routing: which rule fires, and what is in the table it selects
show ip rule
show policy route-map

Read-only / Safeevery table the kernel holds

ip rule show
ip route show table all

Two patterns account for most of this. A prefix that “vanished” after a VRF migration is usually intact in the VRF, with the operator querying the default table. And a policy-routing deployment where the marking rule stops firing sends traffic back to the default table, which may well have a route — just not the one the design intends. The symptom in that second case is not “missing route” at all; it is traffic leaving by the wrong path, reported as a routing failure.

Step 4: the route is genuinely absent — go to its source

Three mechanisms, three separate diagnostics. Use the one that matches how the route is supposed to exist.

Static

Read the configuration first, then stop reading the route and start reading its next hop. A configured static route that is not installed is almost always a next hop that does not resolve: a recursive next hop whose underlying route is gone, an interface with no address, or an interface that is administratively down.

OSPF: two layers, and only one of them is yours

An OSPF route is installed in two stages — the LSA has to reach this router’s link-state database, and SPF has to compute a route from it. The two fail for completely different reasons and the fixes live on different routers.

Read-only / Safedatabase first, then the RIB

show ip route ospf
show ip ospf database
show ip ospf neighbor
show configuration commands | match ospf

In the database, not in the RIB. The information arrived and SPF did not produce a route from it. Look at area summarisation on the area border router, at area type — a stub area does not carry external routes at all — and at any filter applied on the area boundary. This is diagnosable and often fixable locally.

Not in the database. Nothing arrived. The originator is not advertising, or the LSA is not flooding to here. That is another router’s problem, and the useful work from this seat is producing the evidence: which LSAs are present, which adjacency the route should have arrived over, and whether that adjacency is full.

BGP: three failure points, three commands

A missing BGP route failed at exactly one of three places, and the three have different evidence and different owners.

Read-only / Safereceived, selected, installed

show ip bgp summary
show ip bgp neighbors 198.51.100.2 received-routes
show ip bgp 10.20.4.0/22
show ip route 10.20.4.0/22

Where it stopped	What you see	Who owns it
Not received	Absent from `received-routes`	The peer, or an inbound filter dropping it
Received, not selected	In `received-routes`, absent or not best in `show ip bgp`	Local policy, or the best-path decision
Selected, not installed	Best in `show ip bgp`, absent from `show ip route`	Usually the IGP: the next hop is unreachable

The third row is the one that gets misdiagnosed most often, because every BGP command reports health. The route is valid and best; the kernel will not install it because the next hop does not resolve. That is an IGP fault, it will be affecting far more than the prefix you were paged about, and no amount of BGP configuration will fix it.

Step 5: the route is there and traffic still fails

Three causes, in order of how often they are the answer.

Something more specific is winning. Longest-prefix match runs before administrative distance. A /24 learned from anywhere beats the /22 you just restored, regardless of which protocol carried it or how much you trust the source. Run the destination lookup — not the prefix lookup — and read which entry actually answers.

The return path is missing or asymmetric. The single most skipped step in routing diagnosis. Forward success with return failure produces a symptom that looks exactly like a firewall drop, and a stateful device anywhere in the path will make it look even more like one. Prove it with a trace from each end and compare.

The route is doing precisely what it was configured to do. A blackhole or reject route for the prefix discards traffic by design. It will be in the table, selected, installed, and correct. Somebody put it there for a reason that may or may not still hold, and removing it is a change with an owner, not a fix.

What each step costs if you get it wrong

Action	Cost when it is the wrong move
Running the exact-prefix lookup only	Concluding a route is missing while a supernet forwards the traffic elsewhere
Skipping the VRF and policy-table check	A confident wrong answer, and a fix applied to a table nobody is using
Reading the route instead of its next hop	Rewriting a correct static route while the fault stays in the route that resolves it
Adding a static to restore service	Permanent override of the dynamic route at distance 1, invisible until the next failure
Widening a prefix list during the incident	A route leak, which is a larger and more public incident than the one you were paged for
Clearing a BGP session to “refresh”	Every prefix from that peer withdrawn for the duration of re-establishment
Declaring success on a routing table	A restored table and a service that still does not work, because nothing comes back

Escalation, and the option to hold

Escalate as soon as the evidence crosses a boundary you do not own. An empty received-routes for the prefix, or an absent LSA, is complete proof that the route is not arriving — and nothing you do locally can create it honestly. Send that evidence with the escalation rather than a conclusion.

Hold rather than improvise when the only fix available to you is one that overrides a routing decision somebody else owns. A static route over a dynamic protocol, a widened prefix list, a redistribution added to make a prefix appear: each removes today’s symptom and installs tomorrow’s incident. Record the hold with an owner and an end time, and be specific about what service impact continues while it holds — that is the information the person deciding actually needs.

Why this matters

Routing diagnosis rewards a particular kind of pedantry: knowing which table you asked, and which question you asked it. Most of the wasted hours in a missing-route incident come from three small confusions — the exact-prefix lookup mistaken for the destination lookup, the RIB mistaken for the FIB, and the default table mistaken for the table the traffic will consult. None of those are hard. They are just easy to skip when a ticket is old.

The other half of the value is knowing that the routing table is a means, not an end. A route can be present, selected, installed and entirely correct while the service stays down, because something more specific is winning or because nothing is coming back. “The route is there” is a fact about a table; “the application works” is the thing you were actually asked about, and only one of them closes the ticket honestly.

And the static route is always available. It always works. It is almost never the fix, and an estate can be read like tree rings by counting the ones that were added at 3am and never removed.