VRF Contains Missing Route

Reported symptoms

The ticket arrived at 09:20 yesterday and reads: our DC servers cannot reach any of our branch sites.

It is precise, and it is narrow. CUST-A’s servers in our data centre cannot reach anything in 10.1.16.0/20, which is every branch network the tenant owns. Not slow, not intermittent - gone, from every server, at once.

The tenant’s own first hypothesis was their firewall, and they had a good reason for it. The failures are instantaneous. Applications return an unreachable error in milliseconds rather than hanging for a connection timeout, which is what an administratively dropped packet looks like from an application’s point of view and is not what a broken WAN looks like.

Four things sit alongside that and none of them agree with it.

CUST-A reaches the shared DNS resolvers and the shared NTP servers without a problem, and has done throughout. Those live in a different VRF and get there by a configured leak. The tenant’s VRF is demonstrably carrying traffic.

CUST-B was migrated onto the same PE, on the same night, by the same engineer, in the same maintenance window. CUST-B is completely fine.

show ip ospf neighbor on the PE lists five adjacencies, all Full, none of which has reset in twelve days. Two engineers have now run that command and concluded OSPF on the router is healthy.

And the change record for yesterday is empty. Nothing was committed on this PE yesterday, or for the eleven days before it. The only entry in the change calendar is a planned decommission at a different site, signed off as removing nothing that was in use.

The tenant’s edge router, meanwhile, reports no OSPF neighbour at all, and the tenant has raised that against us as a fault.

The estate

rtr-pe-2 is a shared provider-edge router in a regional data centre, VyOS 1.5 LTS. It is a single box with several tenants on it and no MPLS core - the ordinary mid-market shape, where the VRFs are routing-table boundaries rather than L3VPN.

• CUST-A, table 1001. eth1 is enslaved to it and holds 10.1.0.1/24. The tenant’s own edge router, cust-a-edge, sits at 10.1.0.254 and is expected to advertise the branch supernet 10.1.16.0/20 to us over OSPF.
• CUST-B, table 1002. Same shape, different tenant, and its design uses eBGP to the tenant edge rather than OSPF.
• SHARED, table 1099. eth9 holds 10.99.0.1/24 and carries the DNS resolvers, the NTP servers and the monitoring collector. Each tenant VRF has a static route for the shared prefix with the next-hop resolved in SHARED.
• CUST-A has no default route inside its VRF. That is deliberate: the tenant’s internet egress is through their own firewall at another site, and we carry only their internal routing.

Twelve days ago CUST-A was migrated onto this PE from a dedicated single-tenant router. The migration plan included a two-week soak: the old router stayed live and kept advertising the branch prefixes into the tenant’s network, so that a problem on the new PE would degrade rather than fail.

The soak ended yesterday. At 09:00 the old router was powered off as planned, at a different site, with no change required on rtr-pe-2.

Evidence provided

vyos@rtr-pe-2:~$ show vrf

VRF name          state     mac address        flags                     interfaces
--------          -----     -----------        -----                     ----------
CUST-A            up        52:54:00:31:11:01  noarp,master,up,lower_up  eth1
CUST-B            up        52:54:00:31:11:02  noarp,master,up,lower_up  eth2
SHARED            up        52:54:00:31:11:99  noarp,master,up,lower_up  eth9

vyos@rtr-pe-2:~$ show ip route vrf CUST-A

C>* 10.1.0.0/24 is directly connected, eth1, 12d02h10m
L>* 10.1.0.1/32 is directly connected, eth1, 12d02h10m
S>* 10.99.0.0/24 [1/0] via 10.99.0.254, eth9 (vrf SHARED), weight 1, 12d02h10m

ip route show vrf CUST-A returns the same three entries. The branch prefixes are not in the RIB waiting to be installed; they are not anywhere.

vyos@rtr-pe-2:~$ show ip ospf neighbor

Neighbor ID     Pri State           Dead Time Address         Interface
10.255.0.11       1 Full/DR          37.104s 10.0.0.1        eth0
10.255.0.12       1 Full/BDR         34.882s 10.0.0.5        eth0
10.255.0.13       1 Full/DROther     38.213s 10.0.0.9        eth0
10.255.0.14       1 Full/DROther     33.907s 10.0.0.13       eth0
10.255.0.15       1 Full/DROther     36.551s 10.0.0.17       eth0

vyos@rtr-pe-2:~$ vtysh -c 'show ip ospf vrf CUST-A neighbor'

% OSPF is not enabled in vrf CUST-A

vyos@rtr-pe-2:~$ tcpdump -ni eth1 proto ospf -vv

11:04:12.881 IP 10.1.0.254 > 224.0.0.5: OSPFv2, Hello, length 48
      Router-ID 10.1.0.254, Area 0.0.0.0, Authentication Type: none (0)
11:04:22.884 IP 10.1.0.254 > 224.0.0.5: OSPFv2, Hello, length 48
      Router-ID 10.1.0.254, Area 0.0.0.0, Authentication Type: none (0)
11:04:32.879 IP 10.1.0.254 > 224.0.0.5: OSPFv2, Hello, length 48
      Router-ID 10.1.0.254, Area 0.0.0.0, Authentication Type: none (0)

vyos@rtr-pe-2:~$ show configuration commands | match '^set protocols'

set protocols ospf area 0 network 10.0.0.0/22
set protocols ospf area 0 network 10.1.0.0/24
set protocols ospf parameters router-id '10.255.0.2'

ip vrf exec CUST-A ping 10.1.0.254 returns zero loss. show system commit records one commit, twelve days ago at the migration cutover.

Work the evidence before reading on

Everything on this router is up, and the router has no complaint to make about anything.

1. The prefix is missing from show ip route vrf CUST-A and from ip route show vrf CUST-A. Those two commands read different things. What does their agreement eliminate, and what would their disagreement have meant instead?
2. show ip ospf neighbor printed five healthy adjacencies. Which routing context did that command read? Write down what you would have to type to ask the same question about the tenant.
3. The capture shows Hellos travelling in one direction only. Compare that with the area-mismatch case, where both routers send and both discard. What does a one-way capture tell you that a two-way capture cannot?
4. CUST-B was migrated the same night, by the same engineer, from the same kind of router - and is fine. What is different about CUST-B, and what does that difference predict?
5. Nothing was committed on this router yesterday. Restate that sentence as a question about scope: what changed yesterday, and where would you have to look to find it?
6. ip vrf exec CUST-A ping 10.1.0.254 succeeds. Name exactly what that proves, and name the thing everyone assumed it proved.

Before continuing: in a VRF design, every diagnostic command has a scope, and the default scope is the one that will lie to you. List the commands run so far and mark which context each one actually read.

Root cause

1. The stanza is in the right file and the wrong routing context

The tenant’s OSPF configuration sits at the top of the tree:

set protocols ospf area 0 network 10.1.0.0/24

and the interface it refers to is enslaved to the VRF:

set interfaces ethernet eth1 vrf CUST-A

Those two lines describe different routing contexts, and the configuration tree mirrors FRR’s. A top-level protocols stanza renders router ospf - the instance in the default context, operating against the main table. A per-VRF stanza renders router ospf vrf CUST-A, and that qualifier is what makes FRR allocate an instance inside table 1001.

Because eth1 belongs to CUST-A, the default instance’s network statement matches no interface that instance can see. And because there is no vrf name CUST-A protocols ospf node, no instance exists in CUST-A at all. OSPF is not running on that link, in either context.

The capture is the shape of exactly that. The tenant edge router sends a Hello to 224.0.0.5 every ten seconds, and nothing comes back, because there is no process on our side to answer. This is worth contrasting with the parameter-mismatch scenarios, where both routers send and both discard: two-way traffic means two processes are running and disagreeing about something, and one-way traffic means one of them is not running at all. The capture separates those two worlds in a single command, and it is the cheapest command in the incident.

2. Every habitual command read the wrong table

show ip ospf neighbor reads the default instance. So does show ip route. So does an unqualified ping. None of them errors on a router with VRFs; each answers its own question correctly, and its own question is about the main table.

That is why two engineers looked at a healthy five-adjacency neighbour table and concluded OSPF was fine. It was fine. It was also entirely irrelevant to a tenant whose interface is in table 1001.

The per-VRF forms are the whole diagnostic set, and they have to be used deliberately:

The last row is the one that quietly matters most. An unqualified ping from the router uses the global table and can reach a completely different host that happens to share the address - which is not a hypothetical on a box carrying overlapping tenant address space.

3. Absent is a different finding from unresolved

There are two ways for a prefix to be missing from a VRF, and they have different causes, different fixes, and one pair of commands that separates them.

• In the RIB, not in the FIB. show ip route vrf CUST-A has the prefix, with inactive against the next-hop and no best marker; ip route show vrf CUST-A does not have it. FRR knows about the route and could not resolve its next-hop. This is the classic leak defect, where the vrf leaf is missing from the next-hop and FRR is trying to resolve a SHARED address inside CUST-A’s own table.
• In neither. Nothing is producing the route. The question is not “why did it fail to install” but “what was supposed to generate it, and is it running”.

Here the two commands agree, so the second question is the right one - and asking it leads directly to the OSPF instance that does not exist. The leaked shared-services prefix in the same table is useful evidence in the same breath: it proves leaking on this box works, that the next-hop resolution machinery is healthy, and that whatever is wrong is specific to what OSPF was meant to supply.

4. The soak validated the path that was about to be removed

The migration was carefully planned and the plan is what hid the fault.

The old single-tenant router stayed live for a two-week soak and kept advertising 10.1.16.0/20 into the tenant’s network. Every acceptance test run during the soak - reachability from a DC server to a branch host, an application transaction, a traceroute - passed, because it was passing over the router being replaced.

The cutover test that was recorded on the PE itself was ip vrf exec CUST-A ping 10.1.0.254. It succeeded. It uses the connected route on eth1 and would have succeeded on a router with no routing protocol configured at all, which is precisely the router it was run on.

At 09:00 yesterday the old router was powered off. That was a change at another site, requiring nothing on this PE, correctly signed off as removing nothing in use - and it is the reason the change record for the day the service broke is empty.

Resolution

1. Confirm the placement before touching anything. show configuration commands | match "^set protocols" on a router whose routing is all per-VRF should return nothing; here it returns the core stanza plus one tenant line. That single command is both the diagnosis and the acceptance test you are about to add.
2. Work out what else the top-level stanza is carrying before you delete anything from it. On this PE it holds the five core adjacencies that every other tenant depends on. Deleting protocols ospf wholesale to "move it into the VRF" turns a one-tenant outage into an all-tenant outage, and it is the obvious next action.
3. Decide fix-now or hold, and record it. Fixing now is straightforward here: the tenant is hard down for branch traffic, nothing is currently flowing that a path change could disturb, and the confirm window bounds the risk. There is no traffic-migration cost to weigh because there is no traffic.
4. Record that the decommission at the other site was correct and is not to be reversed. Powering the old router back on would restore service and destroy the only clear signal anybody has about which path is actually carrying the tenant.
5. Stage the change without committing: delete only the tenant network statement from the default instance, and add a properly qualified instance under vrf name CUST-A protocols ospf.
6. Set the per-VRF router-id explicitly, under parameters. If you leave it to FRR it derives one from the addresses visible in that routing context, and a derived id can move when an interface is added or removed - which resets every adjacency the instance holds.
7. If the tenant link uses authentication, configure both halves: the area selects the authentication type and the interface carries the key. They are separate statements and a missing key produces an adjacency that never leaves Init.
8. Read the whole change with compare. The deletion and the addition only make sense together, and the diff is the one place both are visible at once.
9. Commit with commit-confirm. The window is not protecting you from the VRF work - it is protecting the core adjacencies from the deletion.
10. Watch the tenant adjacency reach Full before confirming. With default timers it forms in well under a minute; if it has not formed within a dead interval, you have not finished diagnosing.
11. Save once the adjacency is stable, and only then close the confirm window.

# On rtr-pe-2, in configuration mode. Delete only the tenant's network
# statement - the rest of the default instance carries the core
# adjacencies for every other tenant on this box.
delete protocols ospf area 0 network 10.1.0.0/24
set vrf name CUST-A protocols ospf parameters router-id 10.1.0.1
set vrf name CUST-A protocols ospf area 0 network 10.1.0.0/24
compare
commit-confirm 5

Verification

1. Qualify the command. vtysh -c 'show ip ospf vrf CUST-A neighbor' shows 10.1.0.254 in Full. The unqualified show ip ospf neighbor still shows the five core adjacencies and is not evidence about the tenant, before or after.
2. Check from the tenant end at the same time. The tenant edge router must list us as a neighbour in Full. Checking one end is the omission that created this incident.
3. Check twice, several minutes apart and longer than the dead interval. An adjacency caught between two resets looks healthy for a moment.
4. The prefix is in the RIB: show ip route vrf CUST-A contains 10.1.16.0/20 learned from OSPF, with a best marker and a resolved next-hop via eth1.
5. The prefix is in the FIB: ip route show vrf CUST-A contains it too. These are different questions and both have to be asked; a route in the first and not the second forwards nothing.
6. The deletion did no harm. show ip ospf neighbor still lists the same five core adjacencies, with the same neighbour ids, and none of them reset across the commit. This is the check for the risky half of the change.
7. The other tenants are untouched, read rather than assumed. show ip route vrf CUST-B and the BGP summary for that VRF are identical to their pre-change state.
8. Test scoped to the VRF, from the router: ip vrf exec CUST-A traceroute to a branch host. An unqualified traceroute uses the main table and can reach a different host entirely on a box with overlapping tenant space.
9. Test from the tenant, not from us: a real application transaction from a DC server to a branch host, confirmed by the tenant. That is the ticket, and router state is not it.
10. Monitoring is telling the truth again. Re-enable the branch checks suppressed for the migration window and confirm they clear on their own rather than because somebody closed them.

Prevention

• Turn the placement rule into a test. On a router whose routing is all per-VRF, show configuration commands | match "^set protocols" returns nothing. Put that line in the build acceptance criteria and in the post-commit validation, where it costs one second and catches an entire defect class.
• Never lift a stanza from a single-tenant router into a multi-tenant one. The configuration that was correct on a box with one routing table is exactly the configuration that is wrong on a box with several, and both forms are valid, so nothing will tell you.
• Run acceptance in the final state, not the transitional one. A soak behind the router being replaced validates the router being replaced. Either remove the old path for the test or re-run the suite at decommission.
• Treat a decommission as a change against everything that was soaking behind it. The empty change record on the day of the outage was not an accident; it was the direct consequence of filing the decommission against the site it happened at.
• Write down the per-VRF diagnostic set and use it. Every unqualified command answers a different question correctly, and will keep doing so for as long as people type it under pressure.
• Set a per-VRF router-id explicitly. A derived id moves when the visible addresses change, and an OSPF instance that changes router id resets its adjacencies.
• Alert on learned-route count per VRF. A tenant VRF holding only its connected routes is either newly built or broken, and the age of the VRF distinguishes them. This would have fired twelve days before the ticket.
• Give every migration suppression a mandatory expiry, so the checks that would have caught this switch themselves back on.