Bad Redistribution Creates Routing Loop

Reported symptoms

Two data centres, one OSPF domain across the interconnect, one iBGP mesh in AS 64512. bdr-a1 and bdr-b1 are the border routers, one in each site, and each of them bridges the IGP and BGP for its own reason.

Nothing was declared an incident. Five separate observations accumulated over three weeks and were logged in four different queues.

• Capacity planning flagged the DC-A to DC-B interconnect: utilisation up roughly a fifth since late July, made almost entirely of small packets, and none of the applications that use the link grew.
• The address-management team cannot reclaim 172.20.9.0/24. Their reclaim gate refuses to release a block that is still routed, and this one is still routed estate-wide. The VLAN behind it was deleted nine days ago.
• Anybody who traceroutes into that prefix gets the same two addresses alternating for thirty lines and then nothing.
• CPU on both border routers has drifted upward over the same three weeks. top names ospfd and bgpd, and the SPF counters climb considerably faster than the network is changing.
• A partner sees occasional thirty-second stalls reaching one analytics prefix, on a service neither side has touched.

Two facts kept the investigation away from the routing. Both border routers passed an independent configuration review last month: every redistribution statement on both of them carries a route-map, and both carry the tag-400 feedback control the estate’s design document mandates. And the only relevant change — the redistribution added on bdr-b1 three weeks ago — was reviewed, filtered, tagged, committed under commit-confirm, and verified at the export, where it showed exactly the four prefixes on the ticket.

Evidence provided

vyos@bdr-a1:~$ show ip route 172.20.9.0/24

Routing entry for 172.20.9.0/24
Known via "bgp", distance 200, metric 0, best
Last update 09d02h ago
  192.0.2.12 (recursive)
*   198.51.100.10, via eth2

vyos@bdr-b1:~$ show ip route 172.20.9.0/24

Routing entry for 172.20.9.0/24
Known via "ospf", distance 110, metric 20, tag 0, best
Last update 09d02h ago
*   198.51.100.9, via eth2

vyos@bdr-a1:~$ show ip ospf database external self-originate

       AS External Link States

LS age: 412
LS Type: AS-external-LSA
Link State ID: 172.20.9.0 (External Network Number)
Advertising Router: 192.0.2.11
Network Mask: /24
      Metric Type: 2 (Larger than any link state path)
      Metric: 20
      External Route Tag: 0

Link State ID: 172.20.12.0 (External Network Number)
Link State ID: 172.20.13.0 (External Network Number)
Link State ID: 172.20.14.0 (External Network Number)

vyos@bdr-b1:~$ show ip bgp neighbors 192.0.2.30 advertised-routes

   Network          Next Hop            Metric LocPrf Weight Path
*> 172.20.9.0/24    192.0.2.12               0    100  32768 ?
*> 172.20.12.0/24   192.0.2.12               0    100  32768 ?
*> 172.20.13.0/24   192.0.2.12               0    100  32768 ?
*> 172.20.14.0/24   192.0.2.12               0    100  32768 ?

Total number of prefixes 4

vyos@bdr-a1:~$ show ip bgp 172.20.9.0/24

BGP routing table entry for 172.20.9.0/24
Paths: (1 available, best #1, table default)
Local
  192.0.2.12 (metric 20) from 192.0.2.30 (192.0.2.30)
    Origin incomplete, metric 0, localpref 100, valid, internal, best
    Originator: 192.0.2.12, Cluster list: 192.0.2.30

$ traceroute -n 172.20.9.14

traceroute to 172.20.9.14 (172.20.9.14), 30 hops max, 60 byte packets
1  172.20.4.1   0.402 ms
2  198.51.100.9   0.911 ms
3  198.51.100.10  1.744 ms
4  198.51.100.9   2.588 ms
5  198.51.100.10  3.401 ms
6  198.51.100.9   4.219 ms
7  198.51.100.10  5.037 ms

vyos@bdr-a1:~$ show configuration commands | match 'redistribute|route-map BGP-INTO-OSPF'

set policy route-map BGP-INTO-OSPF rule 10 action 'deny'
set policy route-map BGP-INTO-OSPF rule 10 match tag '400'
set policy route-map BGP-INTO-OSPF rule 20 action 'permit'
set protocols ospf redistribute bgp metric 20 metric-type 2 route-map 'BGP-INTO-OSPF'

Work the evidence before reading on

Each router is telling the truth about a prefix that no longer exists.

1. Put the two show ip route outputs side by side. bdr-a1 points at bdr-b1; bdr-b1 points at bdr-a1. Where did either of them get the route, and is there a third router anywhere that has a real one?
2. bdr-b1 stamps tag 400 on the way out. Read the tag field in bdr-b1’s own route entry, and read the External Route Tag in the Type-5 LSA that bdr-a1 originated. What value is actually arriving, and what would match tag 400 have to see to fire?
3. Rule 20 of BGP-INTO-OSPF permits and has no match clause. If rule 10 does not fire, what does this route-map do?
4. bdr-b1’s redistribution statement is redistribute ospf. bdr-a1 has put an OSPF external route into the domain. Is that route eligible for bdr-b1’s redistribution, and if it is, what is now keeping the prefix alive?
5. This ran for three weeks with no visible symptom and then produced a forwarding loop. What changed nine days ago, and why did that particular event turn a quiet fault into a loud one?

Then the question that resolves it: which router originates 172.20.9.0/24, and what happens to your answer when you follow it round once more?

Root cause

1. Two redistributions face each other, and no router shows the pair

bdr-b1 redistributes OSPF into BGP. That is the change from three weeks ago: four analytics prefixes, an explicit prefix-list, a route-map, an export verified against the ticket. It is a good change.

bdr-a1 redistributes BGP into OSPF. That is two years old and was made for a completely unrelated reason — a partner peering at DC-A whose prefixes needed to be visible inside the IGP. It is also a reasonable change.

Together they form a circle, and neither router’s configuration contains any trace of the other. The engineer who added bdr-b1’s statement reviewed bdr-b1. The review last month reviewed each router against a checklist that each router passed. The fault is a property of the pair, and the pair is not displayed anywhere.

2. The feedback control uses a mechanism BGP does not carry

This is the part worth remembering long after the topology is forgotten.

The estate’s design document specifies tag-based feedback prevention: stamp tag 400 on the way out, deny tag 400 on the way in. bdr-b1’s REDIST-OUT sets it. bdr-a1’s BGP-INTO-OSPF denies it. Both configurations contain the number, both engineers could point at their half, and the reviewer confirmed both halves exist.

A route tag is a 32-bit value that OSPF carries in the External Route Tag field of an AS-external LSA, and that zebra keeps on a local route entry. BGP has no equivalent. There is no route-tag path attribute; an UPDATE carries the attributes BGP defines and nothing else, so a tag set while redistributing into BGP never leaves the router that set it.

The evidence says so plainly once you look for it. bdr-b1’s own route entry for the prefix shows tag 0. The External Route Tag in bdr-a1’s Type-5 is 0. The received BGP path at bdr-a1 has no tag at all. Rule 10 has never matched anything and never will.

The attribute BGP does carry, and the one the control should have used, is the community.

3. A deny-list with a trailing permit fails open

Rule 10 denies. Rule 20 permits, with no match clause.

That shape only stops what it has been told about. Every prefix arriving over BGP that is not tagged 400 — which, as established, is every prefix — is permitted into OSPF. The route-map is not filtering; it is a permit-any with a decoration on the front.

The correct shape is the inverse: enumerate what may pass, deny at the end. A route-map written that way cannot silently acquire new members when somebody adds a redistribution three sites away.

4. redistribute ospf includes OSPF externals, and that closes the loop

bdr-a1 originates a Type-5 external for each analytics prefix into the same OSPF domain the prefixes came from. Every router in the domain now holds two entries for each: the real intra-area route in DC-B, and bdr-a1’s external.

For three weeks nothing broke, because OSPF prefers intra-area over inter-area and inter-area over external, so the real route kept winning everywhere it existed. The cost was a doubled database entry per prefix and an SPF tax, which is the CPU drift.

Then 172.20.9.0/24 was decommissioned and the real route disappeared. At that moment:

• bdr-b1 still had an OSPF route for it — bdr-a1’s external — because redistribute ospf does not distinguish internal from external, so bdr-b1 kept advertising it into BGP.
• bdr-a1 still had the BGP route, so it kept originating the Type-5.

The prefix now has no source anywhere and sustains itself perfectly. bdr-a1 forwards towards bdr-b1 because its best path is iBGP; bdr-b1 forwards towards bdr-a1 because its best path is the external. Every packet crosses the interconnect until its TTL runs out, which is the utilisation, the small packets and the traceroute.

Resolution

1. Decide whether this is a window change, and be honest about the answer. The interconnect has headroom and the CPU is drifting rather than saturating, so this is not a 3am change — but the loop is live, the address block is stuck, and every future decommission behind either redistribution will do this again. It goes in the next window, with an owner, not on a backlog.
2. Enumerate what is currently reaching OSPF through bdr-a1. show ip ospf database external self-originate lists everything the trailing permit has been letting in for two years. Some of it is the partner prefixes the route-map was written for; some of it is not, and something may have grown to depend on it. Tightening the route-map without this list turns one incident into two.
3. Convert BGP-INTO-OSPF from a deny-list into an allow-list: explicit permit rules matching a prefix-list of the partner prefixes it was always meant to carry, and a deny at the end. Build the prefix-list and the route-map change in one candidate so there is no commit in which the map is half-converted.
4. Replace the inert tag contract with a community contract in the same window. bdr-b1 stamps it on the way out with set policy route-map REDIST-OUT rule 10 set community add 64512:400; bdr-a1 denies it on the way in by matching a community-list in the first rule of BGP-INTO-OSPF. Change both ends, record the value in the design document with the mechanism named, and tell both owners.
5. Apply under commit-confirm on the router you are changing, and read show ip ospf database external self-originate before you confirm. The failure mode here is withdrawing a prefix somebody needs, and the confirm timer is the cheapest way to un-make that.
6. Watch the decommissioned prefix leave rather than assuming it left. BGP has to withdraw it and the Type-5 has to age out; there is a window in which it is still in somebody's table and that is expected, not a failed fix.
7. Refuse the three symptom fixes. Raising the OSPF external metric, widening the administrative-distance gap so the internal route wins more decisively, or filtering 172.20.9.0/24 by name each make today quieter and leave the loop in place for the next decommission or link failure to find.
8. Check whether any other pair in the estate is in the same shape. The register does not exist yet, so this is a one-off sweep: show configuration commands | match redistribute on every router that runs two protocols, collected and read as one document.
9. Hand the address-management team a date, not a promise. Their gate is the external verification for this incident and they have been blocked for nine days.

Verification

1. 172.20.9.0/24 is absent from show ip ospf database external self-originate on bdr-a1, from show ip bgp on both routers, and from show ip route on a router in each data centre. Three places, because the prefix was being held up by two of them at once.
2. The address-management reclaim gate passes and the block is released. That is the only check in this incident that is independent of the routers that produced it.
3. show ip ospf database external self-originate on bdr-a1 contains the enumerated partner prefixes and nothing that originated inside OSPF. A prefix on that list that is also an intra-area route somewhere in the domain is the loop, still there.
4. show ip bgp neighbors PEER advertised-routes on bdr-b1 matches the written prefix list, and a prefix that should not be there is confirmed absent by name. Proving what you do not advertise is the half of the test an unfiltered redistribution passes.
5. The new control can fire. In the lab, stamp the community on a test prefix and confirm it does not appear as a Type-5 at the far end. A control nobody has watched deny anything is what this incident was made of.
6. Interconnect utilisation returns to its pre-July baseline and the small-packet component disappears. This is observable from the monitoring system rather than from either router.
7. SPF counters and prefix counts are stable across two observations a convergence interval apart. A number still moving is not a result.
8. show configuration commands | match redistribute on both routers, read side by side by one person who can state what the pair does. If nobody can say that in a sentence, the register is the outstanding work.

Prevention

• Keep one register of every redistribution statement in the estate — router, direction, route-map, prefix set — and review it as a set. Mutual redistribution is a property of a pair, and both halves of this fault passed a per-router review that was asking exactly the right questions of exactly one router.
• Use the attribute the medium carries: a tag inside an IGP, a community across BGP. Put the mechanism in the design document and not only the value, so that “tag 400” cannot survive as a number everyone repeats and nobody can trace to an UPDATE.
• Write route-maps as allow-lists with a trailing deny. A deny-list with a trailing permit fails open, acquires new members silently, and is indistinguishable from the correct thing in a review.
• Remember that redistribute ospf includes externals. Anything another router injects into the IGP is eligible to come straight back out, so bound the redistribution with the prefix-list rather than with an assumption about where the routes came from.
• Demonstrate feedback controls on a schedule instead of reading them. Marking a test prefix and confirming the far end refuses it takes minutes and is the only thing that separates a control from a comment.
• Treat a prefix that is still routed after its source is removed as a routing defect. It is the earliest, clearest and cheapest signal this class of fault produces, and here it sat in an address-management queue for nine days.