Troubleshooting leaking — routes missing, route-map blocking, asymmetric path

A ticket arrives: DNS is broken for the mgmt hosts. The leak was configured two weeks ago and nobody has touched it since — or so the change record says. What turns this into a twenty-minute fix rather than a half-day is not knowing more commands. It is running the commands in an order where each answer eliminates a subsystem.

Route leaking deserves its own troubleshooting lesson because its failures have an unusual property: the configuration is right there in front of you, and it is doing nothing. A leak can commit cleanly, survive review, appear in show ip route, and forward not one packet. Three of the five failures below have exactly that shape.

The topology, restated

• default — table 254. eth0 is the transit toward the shared services: 192.0.2.2/30, service router at 192.0.2.1, and the services themselves at 198.51.100.53 (DNS), 198.51.100.123 (NTP), 198.51.100.200 (syslog).
• mgmt — table 1001, eth1, 10.10.0.0/24.
• tenant-a — table 1002, eth2, 10.20.0.0/24.

The leak under investigation:

set vrf name mgmt protocols static route 198.51.100.53/32 next-hop 192.0.2.1 vrf default
set protocols static route 10.10.0.0/24 interface eth1 vrf mgmt

The discipline

1. Define the failure. “A DNS query from 10.10.0.50 to 198.51.100.53 gets no answer” is testable. “DNS is broken” is a mood.
2. Fix the direction. Every leak question is one-directional. Decide which way the packet you care about is travelling before you look at any table, because the return path is configured somewhere else entirely.
3. Take evidence in layers. Kernel FIB, then control-plane RIB, then firewall counters, then the wire. Each layer can only be trusted about itself.
4. Name a specific cause before changing anything. “The vrf leaf is missing on the mgmt leak” is a hypothesis. “Something is wrong with BGP” is not.
5. Change one thing, and write down what the evidence was. The next person to hold this pager is you, at 03:00, with less context.

flowchart TD
  S1["1. Define: which packet, which direction"] --> S2["2. Kernel FIB in the table that packet uses"]
  S2 --> S3["3. RIB, only if the FIB disagrees with the config"]
  S3 --> S4["4. Firewall counters on the path"]
  S4 --> S5["5. tcpdump, last, to settle what is on the wire"]
  S5 --> S6["6. One change, then re-run step 2"]

Failure 1 — the leak that commits and installs nothing

This is the signature route-leaking failure, and it accounts for more lost hours than the other four together. The vrf leaf was omitted:

# What was configured
set vrf name mgmt protocols static route 198.51.100.53/32 next-hop 192.0.2.1

# What was meant
set vrf name mgmt protocols static route 198.51.100.53/32 next-hop 192.0.2.1 vrf default

Zebra installs the route in table 1001 only if it can resolve 192.0.2.1. Without the vrf leaf it resolves that address inside table 1001 — mgmt’s own table, which has no route to the transit /30 and never will. The route stays in the RIB, unresolved, and is never offered to the kernel.

$ ip route show table 1001

10.10.0.0/24 dev eth1 proto kernel scope link src 10.10.0.1

Compare with the same table once the vrf leaf is present:

$ ip route show table 1001

10.10.0.0/24 dev eth1 proto kernel scope link src 10.10.0.1
198.51.100.53 via 192.0.2.1 dev eth0 proto static metric 20

The signature of a leaked route is an output device that does not belong to the VRF the table represents. eth1 is in mgmt; eth0 is not; table 1001 is mgmt’s. No ordinary static route configured inside mgmt could have produced that second line.

The control-plane view of the same failure is softer, which is why it is the second thing to look at rather than the first:

$ vtysh -c 'show ip route vrf mgmt 198.51.100.53/32'

VRF mgmt:
S   198.51.100.53/32 [1/0] via 192.0.2.1 inactive

Two details distinguish this from a working route. The code column carries no > (selected) and no * (installed in the FIB), and the next-hop is flagged as not resolving. A working leak in the same table reads S>* 198.51.100.53/32 [1/0] via 192.0.2.1, eth0 (vrf default) — note the parenthesised VRF, which is the control plane telling you where it resolved the next-hop.

The exact wording of the unresolved case differs between FRR releases. Treat the marker columns as a hint and the kernel table as the fact.

Failure 2 — the next-hop does not resolve in the table you named

The vrf leaf is present and correct, and the leak still does not install. Now the question is about the other table: does default actually have a route to 192.0.2.1?

show ip route vrf default 192.0.2.1
ip route get 192.0.2.1

If the transit interface is down, or the address was renumbered, or the leak names a next-hop that was never on that segment, the answer is no — and the failure is identical in appearance to Failure 1. The route sits unresolved, the kernel table is empty, and the vrf leaf you would have gone looking for is already there.

The distinguishing test is one command, in the source table:

$ ip route get 192.0.2.1

RTNETLINK answers: Network is unreachable

The order matters. Check whether the leak is expressed correctly (Failure 1) and whether the table it points at can resolve the next-hop (Failure 2) before you touch anything, because the fixes are unrelated: one is a configuration edit, the other is usually an interface, an address or an upstream problem that has nothing to do with leaking at all.

flowchart TD
  A["Traffic toward 198.51.100.53 from mgmt fails"] --> B{"Route in ip route show table 1001?"}
  B -- "Yes" --> Z["Routing is fine — go to Failure 3, 4 or 5"]
  B -- "No" --> C{"Does the config carry the vrf leaf?"}
  C -- "No" --> C1["Failure 1: add vrf default, re-check the kernel table"]
  C -- "Yes" --> D{"Can the named table reach the next-hop?"}
  D -- "No" --> D1["Failure 2: fix reachability in that table, not the leak"]
  D -- "Yes" --> E["BGP-carried leak: go to Failure 4"]

Failure 3 — the one-way leak

The leak toward the service is present in the kernel table. The host sends a query, the router forwards it out eth0, the service answers, and nothing arrives. Everyone in the channel says firewall.

It is usually not the firewall. A leak is unidirectional. Leaking 198.51.100.53/32 into mgmt tells the default table nothing about how to reach 10.10.0.0/24, so the reply is dropped for want of a route — in the table the reply is looked up in, which is the default one.

The test is a route lookup for the reply, in the return table:

$ ip route get 10.10.0.50

RTNETLINK answers: Network is unreachable

That single output separates a routing failure from a firewall failure in one step, and it does so before anyone opens the rule set. A firewall drop leaves a route in place and a counter that moves; a missing return leak leaves no route at all.

The fix is the second leak, configured in the default context and pointing into the VRF:

configure
set protocols static route 10.10.0.0/24 interface eth1 vrf mgmt
commit
save

eth1 is directly connected inside mgmt and there is no next-hop address worth naming, so the interface form is the right one here. Verify in the other kernel table:

ip route show table 254

sequenceDiagram
  autonumber
  participant H as Host 10.10.0.50 (mgmt)
  participant K1 as table 1001
  participant K2 as table 254
  participant S as DNS 198.51.100.53

  H->>K1: query, dst 198.51.100.53
  Note over K1: leaked route<br/>via 192.0.2.1 dev eth0
  K1->>S: forwarded out eth0
  S->>K2: reply, dst 10.10.0.50
  Note over K2: no route to 10.10.0.0/24<br/>without the return leak
  K2-->>S: dropped, unreachable

Failure 4 — the BGP import that imports nothing

When there are many prefixes, or the set of them is learned rather than known, the leak is BGP rather than static:

set vrf name mgmt protocols bgp address-family ipv4-unicast import vrf default

The most common way this does nothing is not a broken session, and here it is worth being precise about the mechanism: import vrf creates no session. It copies paths between two BGP RIBs inside the one FRR process. There is no neighbour to go Idle, no MD5 key to mismatch, and show bgp summary in either context is silent about the relationship. Anyone troubleshooting an import by staring at session state is looking at a thing that does not exist.

Two real causes, in order of frequency:

The source VRF’s BGP instance holds nothing. A connected route lives in zebra’s RIB, not BGP’s, until a network statement or a redistribute puts it there. Import a VRF whose BGP table is empty and you import nothing, correctly and silently.

# In the source context, put the prefixes into BGP first
set protocols bgp address-family ipv4-unicast redistribute connected

The import route-map denies them. The map applied to the importing address family is evaluated on the way in:

set vrf name mgmt protocols bgp address-family ipv4-unicast route-map vrf import LEAK-IN

The evidence is a comparison of two tables — what the source holds against what the importer accepted:

show bgp vrf default ipv4 unicast
show bgp vrf mgmt ipv4 unicast

If the prefix is in the first and absent from the second, the route-map is the subject. Read it and the prefix-list it references with FRR’s own view, which shows the rendered policy rather than the VyOS configuration that produced it:

vtysh -c 'show route-map LEAK-IN'
vtysh -c 'show ip prefix-list SHARED-SERVICES'

A prefix-list ends in an implicit deny. A prefix that is simply not listed is denied by that implicit rule, and nothing in the configuration will look wrong — there is no rule to read.

Failure 5 — the leak is fine and the boundary is not

Routing is proven: both kernel tables carry the leak, the query and the reply have paths. The flow still fails. Now the firewall is a legitimate suspect, and the way to interrogate it is counters, not reasoning.

show firewall ipv4 forward filter

The output lists each rule with its packets and bytes. That turns a long argument into a short observation:

• A rule you expect to match shows zero packets. Its match criteria do not describe your traffic. Under leaking the usual culprit is an interface match, so check the direction and the interface pair before anything else.
• A drop rule’s counter moves in step with the test. You have found the drop. Read that rule.
• Nothing moves anywhere. The traffic is not reaching this chain at all — go back to routing, or to tcpdump.

The stateful half of this failure is worth naming separately. A forward rule set that permits the request and relies on state established for the reply will fail if the established rule is absent or ordered after a drop:

set firewall ipv4 forward filter default-action drop
set firewall ipv4 forward filter rule 5 action accept
set firewall ipv4 forward filter rule 5 description 'Return traffic for established flows'
set firewall ipv4 forward filter rule 5 state established
set firewall ipv4 forward filter rule 5 state related

If the reply is being dropped for want of state while the request was accepted, the conntrack table will show the flow as unreplied:

show conntrack table ipv4

When it is not the leak at all: conntrack exhaustion

Occasionally “the leak broke” means the router ran out of connection tracking entries. Established flows keep working because they are already in the table; new flows are dropped; the failure is router-wide but the tenant who noticed first owns the ticket.

cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max
dmesg | grep -i conntrack

A count sitting at the maximum, and nf_conntrack: table full, dropping packet in the kernel log, are unambiguous. The remedies are to raise the table, or to find whatever is opening connections faster than they expire:

set system conntrack table-size 262144

Raise it knowing what it costs — each entry is memory the router does not get back — and treat a table that fills twice as a capacity question rather than a tuning question.

Failure 6 — MTU at the leak boundary

A leak joins interfaces that may have been provisioned by different teams. Small packets cross, large ones vanish, and the application gets blamed.

ping 198.51.100.53 vrf mgmt count 3 size 1200 do-not-fragment
ping 198.51.100.53 vrf mgmt count 3 size 1450 do-not-fragment

Two runs that differ only in size. If the first succeeds and the second does not, the path cannot carry the larger packet and the sender is not learning about it — which means the ICMP that carries that news is being dropped somewhere on the path.

set firewall ipv4 forward filter rule 6 action accept
set firewall ipv4 forward filter rule 6 description 'ICMP unreachable/frag-needed - required for PMTUD'
set firewall ipv4 forward filter rule 6 protocol icmp
set firewall ipv4 forward filter rule 6 icmp type 3
set firewall ipv4 forward filter rule 6 icmp code 4

The message PMTUD depends on is ICMP type 3, code 4 — destination unreachable, fragmentation needed. VyOS names some ICMP types with icmp type-name, but that node has no entry for this one; it lives under destination-unreachable as a code. Matching the type and code numerically is the precise form. If you would rather permit every destination-unreachable, icmp type-name destination-unreachable does that, at the cost of also permitting the ones an attacker can use to tear down flows.

For TCP specifically you can stop depending on that message:

set interfaces ethernet eth1 ip adjust-mss clamp-mss-to-pmtu

Clamping fixes TCP and does nothing for UDP, so the firewall rule is the first fix and the clamp is the second.

The evidence order, in one picture

flowchart TD
  S["Ticket: the leak is broken"] --> D1["Define: which packet, which direction"]
  D1 --> E1{"Route in the sender's kernel table?"}
  E1 -- "No" --> F1["Failure 1 or 2: vrf leaf, or next-hop unresolvable in the named table"]
  E1 -- "Yes" --> E2{"Route for the reply in the return table?"}
  E2 -- "No" --> F3["Failure 3: one-way leak — add the return leak"]
  E2 -- "Yes" --> E3{"Carried by BGP import?"}
  E3 -- "Yes" --> F4["Failure 4: source RIB empty, or vrf import route-map denying"]
  E3 -- "No" --> E4{"Firewall counters moving?"}
  E4 -- "Drop counter moves" --> F5["Failure 5: read that rule"]
  E4 -- "Nothing moves" --> F6["Not in this chain — tcpdump the wire"]
  E4 -- "Accept counters move" --> E5{"Large packets fail?"}
  E5 -- "Yes" --> F7["Failure 6: PMTUD across the boundary"]
  E5 -- "No" --> F8["Service-side, not leak-side"]

Each decision node is answered by one command, and every branch ends at a named subsystem. An operator who runs it in order arrives at the same answer as an operator who guesses well, in a predictable amount of time and with the evidence already collected.

How it fails

• Starting at the RIB. show ip route vrf mgmt shows the route, the operator concludes routing is fine, and the actual defect is that the route was never installed. The kernel table would have said so first.
• Debugging a session that does not exist. Hours spent on BGP state for an import vrf relationship, which has no session.
• Treating a missing return leak as a firewall problem. The ticket goes to the security team, who correctly report that nothing is being dropped, and the loop repeats.
• Reasoning about firewall matches instead of reading counters. The counter answers in one packet what the argument does not settle in twenty minutes.
• Testing from the router. ping 198.51.100.53 from the CLI uses the default table, not the tenant’s. It can succeed while every host in the VRF fails, and it can fail while they are fine. Use vrf mgmt on the command, or test from a host.
• Changing two things. The leak and the firewall are edited in one commit, the flow starts working, and nobody knows which change did it — so the runbook records both, and the next incident starts from a worse position.

Rollback

• Any change made inside configure is reversible: rollback 1 then commit returns to the previous commit. Confirm the return with the same kernel-table command you used to diagnose, not with the configuration.
• If a firewall rule was added during the investigation, remove it before anything else. A rule added under pressure is the change most likely to have side effects on flows nobody was watching.
• If a leak was added and did not help, delete it rather than leaving it. delete vrf name mgmt protocols static route 198.51.100.53/32, then commit. A speculative leak left behind is the undocumented leak the anti-patterns lesson warns about.
• If import vrf was configured, remove the import before removing any VRF it referenced — VyOS refuses to delete a VRF another context imports from, and the error arrives at commit time when you are already mid-change.

Production discipline

Cross-course references

vyos-xv-03-vrf-routing-protocols establishes the two leak mechanisms — the vrf leaf and import vrf — and is where the “install here, resolve there” model is developed. vyos-xvi-02-leaking-config and vyos-xvi-03-leaking-firewall are the configurations this lesson takes apart, and vyos-xvi-04-leaking-ipv6 carries the same failure list into IPv6, where the return-path failure is more common because a dual-stack pair needs four statements. The Linux course’s XXII-Linux-NetTroubleshoot covers conntrack and PMTUD as host behaviour.

Quiz