Route Leak Between VRFs

Reported symptoms

The finding arrives from outside the network team, which is the first thing that makes it hard to act on.

• Tenant B’s security team reports scanning activity from 10.20.0.41. That address belongs to tenant A. It should not be routable anywhere inside tenant B, and the tenant-B team say so in the ticket, in the tone of people who expect to be told they have misread a log.
• A tenant-B database logs successful connections from a source outside its allow-list. The DBA assumes a NAT artefact and asks the network team to confirm which translation is producing it.
• The quarterly isolation attestation fails. The auditor put a host in tenant A, pinged a tenant-B address, and captured the replies. They have the pcap.
• The network team cannot reproduce it from the router. show ip route vrf tenant-a contains no route to tenant B. The first written response to the audit finding is that the auditor tested from the wrong VLAN.

And the context that makes it worse rather than better:

• Nothing changed recently. The last VRF-related commit is four months old; the one before it is eight months old. Both were reviewed, both were approved, and neither mentions the other tenant.
• The shared-services leak works correctly. DNS, NTP and syslog resolve from both tenants, which is what the leak exists for, so the leak is not under suspicion.
• The NMS has been polling tenant devices happily since the four-month-old change, and nobody connects that success to this failure.

The estate

One VyOS 1.5 LTS chassis carries the whole multi-tenant edge:

Three shared services live in global and every VRF needs them: DNS at 198.51.100.53, NTP at 198.51.100.123, syslog at 198.51.100.200. They are leaked into each consumer VRF by a BGP instance per VRF, peering with the global instance over internal links — 10.255.0.4/30 on eth4 to tenant-a, 10.255.0.8/30 on eth5 to tenant-b — with a LEAK-IN route-map on the consumer side that permits only those three prefixes.

The design document is explicit that the tenants must not reach each other, and it is the basis of the contractual isolation statement the auditor is testing.

Evidence provided

vyos@rtr-mt-1:~$ show ip route vrf tenant-a

C>* 10.20.0.0/24 is directly connected, eth2
S>* 0.0.0.0/0 [1/0] via 198.51.100.1, 08:14:22
B>* 198.51.100.53/32 [200/0] via 10.255.0.5, 04:02:11
B>* 198.51.100.123/32 [200/0] via 10.255.0.5, 04:02:11
B>* 198.51.100.200/32 [200/0] via 10.255.0.5, 04:02:11

vyos@rtr-mt-1:~$ show ip route vrf tenant-b

C>* 10.30.0.0/24 is directly connected, eth3
S>* 0.0.0.0/0 [1/0] via 198.51.100.1, 08:14:22
B>* 198.51.100.53/32 [200/0] via 10.255.0.9, 04:02:11
B>* 198.51.100.123/32 [200/0] via 10.255.0.9, 04:02:11
B>* 198.51.100.200/32 [200/0] via 10.255.0.9, 04:02:11

vyos@rtr-mt-1:~$ show ip route

C>* 198.51.100.0/24 is directly connected, eth0
C>* 10.255.0.4/30 is directly connected, eth4
C>* 10.255.0.8/30 is directly connected, eth5
B>* 10.20.0.0/24 [200/0] via 10.255.0.6, 04:02:11
B>* 10.30.0.0/24 [200/0] via 10.255.0.10, 04:02:11

vyos@rtr-mt-1:~$ show configuration commands | match 'protocols static route 0.0.0.0/0'

set vrf name tenant-a protocols static route 0.0.0.0/0 next-hop '198.51.100.1'
set vrf name tenant-b protocols static route 0.0.0.0/0 next-hop '198.51.100.1'

vyos@rtr-mt-1:~$ show configuration commands | match redistribute

set vrf name global protocols bgp address-family ipv4-unicast redistribute connected route-map 'SHARED-SERVICES-ONLY'
set vrf name tenant-a protocols bgp address-family ipv4-unicast redistribute connected
set vrf name tenant-b protocols bgp address-family ipv4-unicast redistribute connected

vyos@rtr-mt-1:~$ monitor traffic interface eth3

IP 10.20.0.41.51422 > 10.30.0.7.5432: Flags [S], seq 2244916, win 64240
IP 10.30.0.7.5432 > 10.20.0.41.51422: Flags [S.], seq 881204, ack 2244917
IP 10.20.0.41.51422 > 10.30.0.7.5432: Flags [.], ack 1, win 502

vyos@rtr-mt-1:~$ show system commit

0   2026-04-19 10:52:31 by ops2 via cli   allow NMS to poll tenant devices
1   2026-04-02 08:15:03 by ops1 via cli   syslog: add secondary collector
2   2025-12-11 15:27:44 by ops1 via cli   tenant internet egress via shared upstream

Work the evidence before reading on

The trap in this scenario is that the obvious check exonerates the router. Both tenant tables are clean. If you accept that as the answer you write the reply the network team already wrote, and the auditor’s pcap is still true.

1. The auditor has packets. The tenant table has no route. Both are correct, so the packet is not being forwarded by a route in that table. Which lookup, in which table, is doing the work?
2. Read the tenant default route as an instruction rather than as a destination. What does “via 198.51.100.1” actually cause, given that 198.51.100.1 is not in the tenant-A table either?
3. Now read the global table with that in mind, and specifically the two B>* entries. Ask where they came from and what they make possible for a packet that arrives in the global table without a specific match of its own.
4. Compare the three redistribute connected lines against each other. One is filtered. Ask which direction the estate’s double-filter rule was written for, and which direction did not exist when it was written.
5. Put the two commits on a timeline and ask what was true about the global table in December, and what became true in April.

Before continuing: neither tenant table contains a route to the other tenant, and the traffic is flowing anyway. Name the table that is carrying it, and name the two changes that had to both be present for it to work.

Root cause

1. A leaked default route is a second lookup, not a destination

The tenant default 0.0.0.0/0 next-hop 198.51.100.1 reads like “send outbound traffic to the upstream”. That is not what it does. 198.51.100.1 is not in the tenant-A table; the route is a cross-table instruction, exactly like the shared-service leaks in the design, and it says: any destination this table cannot match is handed to the global table, where a second lookup decides what happens next.

Which means the reachability this route grants is not a property of the route at all. It is a property of the global table, evaluated fresh on every packet, forever. In December the global table held the upstream segment and the shared services, so the change delivered exactly what its ticket said and tested clean. Nothing about the change guaranteed that would remain true.

This is the wildcard-leak anti-pattern that the estate’s own configuration standard names: a default route from one VRF into another leaks whatever the other table happens to contain, and the operator who writes it is thinking about egress while the configuration is describing something much larger.

2. The double-filter rule was written for one direction

The design is explicit that a BGP leak carries a route-map on the export side and a route-map on the import side, so that a mistake on one side is caught by the other. The estate implements that correctly for the direction it was designed for: global redistributes connected routes through SHARED-SERVICES-ONLY, and each consumer filters again with LEAK-IN on the way in.

The April change added traffic in the opposite direction — tenant instances redistributing their connected routes so the NMS could poll tenant devices — and there was no rule to violate, because the rule had been written when that direction did not exist. Nothing filters what leaves a tenant instance, and nothing filters what the global instance imports from one. So 10.20.0.0/24 and 10.30.0.0/24 entered the global table.

Read on its own, that change is fine and it did what it promised. The NMS reaches tenant devices, and has ever since.

3. The composition nobody reviewed

Put the two together and follow a packet from 10.20.0.41 to 10.30.0.7:

1. It arrives on eth2 and is looked up in table 1002. No specific match, so the default fires and the packet is handed to the global table.
2. It is looked up again in table 254, where B>* 10.30.0.0/24 via 10.255.0.10 matches, and the packet is delivered into tenant B.
3. The reply from 10.30.0.7 takes the mirror path: no specific match in table 1003, default into global, specific match for 10.20.0.0/24, delivered into tenant A.

Full bidirectional reachability between two tenants, established by two commits four months apart, neither of which contains the string tenant-b in the same diff as tenant-a. And the route that carries it is in neither tenant table, which is why every check the network team ran came back clean.

Resolution

1. Decide what this is before deciding what to do. It is a tenant-isolation breach with a four-month exposure window and a contractual dimension, and it is simultaneously a change that will interrupt two tenants if you get it wrong. Those two facts pull in opposite directions and the decision needs an owner senior enough to make it.
2. Do not reach for the fastest lever. Deleting the tenant default routes closes the leak in seconds and takes tenant internet egress with it - a service-affecting change to two paying tenants, made during an incident, without a window. It is a defensible choice, but only if someone chooses it deliberately.
3. Do not reach for the other fast lever either. Filtering what the global instance imports from the tenant sessions removes the specific routes and takes NMS polling with it, and leaves the wildcard default in place to reproduce this the next time the global table learns anything tenant-adjacent.
4. The fix is both halves of the design, restored. Replace each tenant wildcard default with the filtered form: a default carrying a route-map that permits only the shared-service prefixes, so the tenant table receives those three prefixes and nothing else that the global table may acquire later.
5. Then close the direction the design never covered: apply an import route-map on the global side of each tenant BGP session, so the double-filter rule holds outbound from a tenant as well as inbound to it. If the NMS genuinely needs tenant reachability, that is a designed path with its own filter, not the absence of one.
6. Read the whole diff with compare before committing, and commit with commit-confirm. The failure mode of this remediation is a tenant losing DNS and NTP rather than gaining a neighbour, and the timer is what gives them back.
7. If the window is not available tonight, contain rather than improvise. A VRF-aware firewall rule dropping tenant-to-tenant forwarding is a legitimate compensating control. Record it as containment with a named owner and an explicit end date - the routing is still wrong underneath it, and a compensating control with no expiry becomes the architecture.
8. In parallel, and independently of the configuration work, answer the question the incident actually turns on: did anything cross, and what. That is a flow-record review over the four-month window. The router as it stands now cannot answer it, and the answer is what the tenants and the auditor are owed.

The filtered leak, in the form the design specifies:

set policy prefix-list SHARED-SERVICES rule 10 action permit
set policy prefix-list SHARED-SERVICES rule 10 prefix 198.51.100.53/32
set policy prefix-list SHARED-SERVICES rule 20 action permit
set policy prefix-list SHARED-SERVICES rule 20 prefix 198.51.100.123/32
set policy prefix-list SHARED-SERVICES rule 30 action permit
set policy prefix-list SHARED-SERVICES rule 30 prefix 198.51.100.200/32

set policy route-map LEAK-SHARED-INTO-TENANT-A rule 10 action permit
set policy route-map LEAK-SHARED-INTO-TENANT-A rule 10 match ip address prefix-list SHARED-SERVICES

set vrf name tenant-a protocols static route 0.0.0.0/0 next-hop 198.51.100.1 route-map LEAK-SHARED-INTO-TENANT-A

Verification

1. Prove the negative from a host. From a test host in tenant A, a tenant-B address must be unreachable. Run the same test against a destination that worked before the change, so that failure to reach tenant B is evidence and not an artefact of a broken test host.
2. The leak still does its job. DNS, NTP and syslog are reachable from both tenants, and show ip route vrf tenant-a still lists the three shared-service prefixes. A remediation that restores isolation by breaking name resolution for two tenants will be reverted before morning.
3. The tenant tables no longer carry an unfiltered default, and the global table no longer carries tenant prefixes. Read both with show ip route and its per-VRF form.
4. Check the FRR RIB and the kernel FIB separately. A route-map denial and a kernel rejection look identical from one side, and the estate needs to know which one it is looking at.
5. State plainly whether NMS polling of tenant devices still works. If the fix broke it, raise it now as a finding with a designed replacement, rather than letting someone restore the unfiltered redistribution on Monday to make a dashboard go green.
6. Re-run the auditor test with the auditor. The attestation closes on their evidence, from their host, not on a route table shown to them by the team whose configuration is under review.

Prevention

• A leak is a pair, not a line. It is defined by what the source table cannot match and by what the destination table can, so every change to either table is a change to the leak. That is why this defect appears in no diff and why diff review cannot catch it.
• Regenerate the reachability matrix from the running configuration. Which tenant can reach which prefix set, derived rather than maintained. The entire failure is that two engineers each held a correct half of that matrix and neither held the whole.
• Make a wildcard leak fail review. A default route from one VRF into another delivers whatever the other table happens to contain, now and in every future state of that table. The configuration standard already names it as an anti-pattern; the gap is that nothing enforces it.
• Apply the double-filter rule to directions that did not exist yet. The tenant sessions were filtered on the way in and not on the way out, because outbound was not part of the design when the rule was written. Rules about “every leak” have to be re-applied when a new leak appears, and a redistribution added for monitoring is a new leak.
• Test isolation as a positive control. A scheduled job that attempts tenant-to-tenant traffic and alerts when it succeeds. Isolation that is only ever asserted is a claim, and the auditor should not be the first party to test it.
• Alert on a VRF table gaining a prefix it did not have. The global table acquiring two tenant subnets in April was the moment this became a breach. It was a routine, approved, successful change, and it was completely silent.