Upgrade Breaks Routing Daemon Behaviour

Reported symptoms

edge2 is one of two internet edge routers. It holds eBGP sessions to two transit providers — A at 198.51.100.1 and B at 203.0.113.1 — and an iBGP session to edge1. On Sunday night it was upgraded to a new image with add system image, set system image default-boot and a reboot, in a change window that ran from 22:00 to 23:10.

The post-upgrade validation passed. The configuration committed cleanly, both eBGP sessions came up Established, the OSPF adjacencies reached Full, and test traffic worked. The window was closed on time.

By Monday morning there were three tickets, and none of them was about the upgrade:

• Provider B’s link has carried nothing in either direction since 22:14 on Sunday. The NOC raised it with the provider, who replied that their side is up and that they are receiving zero prefixes from us — from their view, we withdrew our entire estate.
• Provider A is carrying all of the transit on its own and is congesting at peak. That is being worked as a capacity problem, with a request to the provider for a link upgrade.
• Nothing has alerted. The BGP monitoring watches session state, and neither session has left Established for a moment.

On the router, the first checks all come back clean. Both sessions are up. The configuration is byte-identical to Friday’s — the diff between the pre-upgrade and post-upgrade show configuration commands captures contains nothing in the BGP tree at all. And the iBGP session to edge1, on the same box, on the same image, works perfectly.

Evidence provided

vyos@edge2:~$ show ip bgp summary

IPv4 Unicast Summary (VRF default):
BGP router identifier 10.255.0.2, local AS number 64512

Neighbor          V     AS   MsgRcvd   MsgSent   Up/Down  State/PfxRcd
198.51.100.1      4  64500   1841022      1193  11:04:18        951204
203.0.113.1       4  64510       412       409  11:04:12             0
10.255.0.1        4  64512      2214      2209  11:03:57          1180

Total number of neighbors 3

vyos@edge2:~$ cat /config/pre-upgrade-bgp.txt

Neighbor          V     AS   MsgRcvd   MsgSent   Up/Down  State/PfxRcd
198.51.100.1      4  64500  22841901    418822  27w1d          950877
203.0.113.1       4  64510  19022145    311904  19w3d          950812
10.255.0.1        4  64512    884101    883217  27w1d            1179

vyos@edge2:~$ show ip bgp neighbors 203.0.113.1 advertised-routes

Total number of prefixes 0

vyos@edge2:~$ show configuration commands | match 'protocols bgp neighbor 198.51.100.1'

set protocols bgp neighbor 198.51.100.1 remote-as '64500'
set protocols bgp neighbor 198.51.100.1 address-family ipv4-unicast route-map import 'RM-TRANSIT-A-IN'
set protocols bgp neighbor 198.51.100.1 address-family ipv4-unicast route-map export 'RM-ORIGINATE-ONLY'

vyos@edge2:~$ show configuration commands | match 'protocols bgp neighbor 203.0.113.1'

set protocols bgp neighbor 203.0.113.1 remote-as '64510'
set protocols bgp neighbor 203.0.113.1 address-family ipv4-unicast

vyos@edge2:~$ diff /config/pre-upgrade-cli.txt /config/post-upgrade-cli.txt

vyos@edge2:~$ diff /config/pre-upgrade-frr.txt /config/post-upgrade-frr.txt

--- /config/pre-upgrade-frr.txt
+++ /config/post-upgrade-frr.txt
@@ router bgp 64512 @@
-  no bgp ebgp-requires-policy
+  bgp ebgp-requires-policy

Work the evidence before reading on

An unchanged configuration produced a changed router, and both halves of that sentence are provably true.

1. The CLI diff is empty and the FRR diff is not. Who writes each of those two files, when is each of them produced, and which one did the upgrade replace?
2. Provider A and provider B are the same kind of session on the same router with the same image. List what differs between them, using the two configuration extracts, and note that the difference is two lines that are present on one and absent on the other.
3. The session to provider B is Established and has exchanged nothing in either direction. Which parts of BGP have to work for a session to reach Established, and is deciding what to advertise one of them?
4. advertised-routes is zero as well as received-routes. A fault that silenced one direction would be a filter; what kind of rule silences both at once, and what does it use as its trigger?
5. Sunday’s validation checked that the sessions were up. What would it have had to compare against to catch this, and does that artefact exist on this router?

Then the question that resolves it: nothing in the configuration says what should happen to an eBGP session that has no policy attached. So who decided, and where does their decision live?

Root cause

1. The operator owns half the routing configuration

On VyOS you do not write FRR configuration. You write the VyOS tree, and the image renders it into FRR’s own syntax at commit time using templates that ship with that image. There are therefore two configuration files behind every routing decision this box makes, and only one of them is in your repository.

That is why the two diffs disagree. The CLI capture is unchanged because the migration had nothing to rewrite in the BGP tree. The rendered FRR capture gained a line under router bgp that the previous image did not emit, and that line is the whole incident.

2. RFC 8212, and what “no policy” used to mean

FRR implements RFC 8212: an eBGP session with no inbound and no outbound policy attached may establish and exchange nothing at all. VyOS exposes the switch as set protocols bgp parameters ebgp-requires-policy.

Until Sunday, the rendered configuration on this box did not have the enforcement in force, and a neighbour with no route-map behaved as though it had a permit-everything policy in both directions. Since Sunday it does have it, and the same neighbour behaves as the RFC says it should.

Provider A is unaffected for a reason worth stating plainly: it was built properly. It has an inbound route-map that filters what a transit provider is allowed to hand you and sets the local preference the design intends, and an outbound route-map that advertises your own prefixes and nothing else. Satisfying the enforcement is a side effect of that, not the reason for it.

Provider B was turned up in a hurry nineteen weeks ago, worked immediately with no policy, and was never revisited. The upgrade did not break it. The upgrade stopped concealing that it had been built without the two objects a transit session needs.

3. Every check that was run was true, and none of them was about this

The commit was clean because the commit engine validates the configuration against the schema. It has no opinion about whether the router still does what it did on Friday, and it cannot have one.

The session was Established because policy enforcement is not a session failure. TCP came up, the OPEN messages were exchanged, capabilities were negotiated, keepalives flow. The peer is entirely healthy and has nothing to say.

The alert did not fire because it watches session state. The one number that changed — the prefix count — was the number nothing was measuring, and it sits in the same line of the same command that the on-call engineer read at 23:05 and again at 09:20.

4. The blast radius arrived as somebody else’s ticket

With no prefixes from B, every route is learned from A, so all outbound traffic follows A. With nothing advertised to B, the return path collapses onto A as well. The estate did not lose connectivity; it lost half its capacity and all of its redundancy, and both losses presented as third-party problems — a provider who has stopped sending traffic, and a provider whose link is congested.

That is the characteristic shape of this failure class. It does not produce an outage on the router that caused it. It produces two plausible tickets pointing outward.

Resolution

1. Decide the objective first: this is a redundancy and capacity outage rather than a connectivity one, so the choice is between a fast configuration change on a live router and a reboot back onto the previous image. Name both costs out loud before choosing.
2. Prefer attaching policy. Write an inbound route-map that filters what a transit provider may hand you and sets the local preference the design intends, and an outbound route-map that advertises your own prefixes and nothing else, then attach both to the provider B neighbour address-family.
3. Refuse a pair of permit-any route-maps. They satisfy the enforcement, restore the prefix counts, and leave you one mistake away from re-advertising a full table from one provider to the other with your own CPU as the only limit.
4. Refuse turning the enforcement off. ebgp-requires-policy disabled is a one-line fix that restores Sunday behaviour on every eBGP session on the box, including the ones nobody has audited, and removes the guard that would catch the next session built in a hurry.
5. Apply under commit-confirm so a mistake rolls itself back without a console visit, and expect the CPU and convergence cost of a full table arriving on a router that is currently carrying all of the estate transit.
6. Keep the image rollback as a named option rather than a fallback nobody discussed: set system image default-boot with the previous image and a reboot restores the old default in one step, at the cost of a second outage window, of reverting everything else the upgrade brought, and of meeting this again on the next attempt.
7. Once B is exchanging prefixes, re-check the traffic ratio between the two transits and close the provider A capacity ticket if the congestion was purely a consequence of this.
8. Sweep the rest of the fleet: every router upgraded in that window, and on each one every eBGP neighbour with no route-map in one or both directions. That list is a grep of the configuration, not an investigation.

Verification

1. show ip bgp summary shows provider B back within a sane band of the count in /config/pre-upgrade-bgp.txt. The prefix column is the test; the state word was Established throughout the outage.
2. show ip bgp neighbors 203.0.113.1 advertised-routes lists exactly the prefixes you intend to originate and nothing else. An enforcement that is satisfied and a policy that is correct are different claims, and this is the one that decides whether you can leak the internet between two providers.
3. Provider B confirms the prefix count they receive from you. That is the only evidence in this incident not produced by the router that produced the fault.
4. Traffic has actually returned, read from interface counters and the ratio between the two transits rather than from the routing table. Routes present and traffic flowing are separate facts.
5. vtysh -c "show running-config" still shows the policy enforcement in force, confirming you fixed the session rather than disabled the guard.
6. The check can fail. On a lab router, remove one direction of policy and confirm the prefix count falls to zero and the new alert fires, so the threshold has been watched tripping rather than assumed.
7. The monitoring and automation are run by hand against the upgraded image and their output read. An exporter that stopped parsing a changed output format is the other half of this upgrade blast radius and produces silence rather than an error.

Prevention

• Build every eBGP session with an inbound and an outbound policy, whatever the current default permits. An option you do not set is a value somebody else owns, and they are allowed to change it in a release that is entirely correct and properly documented.
• Make upgrade validation a diff, not a checklist. Capture the configuration, the rendered FRR configuration, route counts, per-session prefix counts and adjacency counts before the reboot; take the same captures after; compare the numbers. “Sessions Established, adjacencies Full, pings fine” describes a router nobody has checked.
• Alert on prefix count per session with a floor, not on session state alone. A session Established at zero prefixes is a distinct condition and it was this entire incident.
• Read both versions off the box and put them in the change record — show version for the image and vtysh -c 'show version' for the routing engine. When a behaviour argument turns on which routing engine you are running, the second command is what settles it.
• Canary the upgrade on a router that is not carrying half your transit, and run the monitoring and automation against it before the fleet, because output formats move between releases and a scraper that stopped parsing fails quietly.
• Keep the pre-upgrade configuration file off the box. A configuration saved after an upgrade is a new-schema file that the previous image has no migration to read, so it is not the rollback artefact it is assumed to be.
• Treat the asymmetry as a finding in its own right. One session was built properly and one was built in a hurry; the upgrade did not create that difference, it published it.