Route churn — FRR zebra CPU, route-flap, iBGP convergence, OSPF SPF

Route churn is the steady-state pressure on a router’s control plane. A router in a stable topology sees a few route updates per minute; a router in a flapping topology sees thousands. The CPU cost of route churn is borne by FRR’s zebra daemon, which maintains the RIB (Routing Information Base) and pushes updates to the kernel FIB (Forwarding Information Base) via netlink. A sustained churn rate of 10,000 updates per second can saturate zebra; a churn rate of 100,000 updates per second saturates the kernel’s netlink socket and the BGP/OSPF daemons that are also trying to update.

This lesson is the production reference for route churn on VyOS 1.5 LTS: the four sources of churn, the evidence that distinguishes them, and the canonical fixes (BGP dampening, route-map filtering, OSPF area design, summarisation) that bound the blast radius.

The four churn sources

flowchart TD
  S["Route churn"]
  S --> BGP["BGP update burst<br/>(peer sends 100k routes)"]
  S --> IGP["iBGP convergence<br/>(full-table refresh)"]
  S --> OSPF["OSPF SPF thrash<br/>(LSA flood)"]
  S --> FLAP["Route flap<br/>(link or peer bouncing)"]
  BGP --> Z["zebra CPU 100%"]
  IGP --> Z
  OSPF --> O["ospfd CPU 100%"]
  FLAP --> Z
  FLAP --> O

Each source has a distinct signature:

BGP update burst. A peer sends a large route update (initial table load, configuration change on the peer side, route-leak recovery). zebra processes the update, installs the routes in the RIB, and pushes to the kernel FIB. The cost is proportional to the number of routes; a 100,000-route burst takes seconds; a 1,000,000-route burst takes minutes.

iBGP convergence. The local router is a route reflector; an iBGP peer reset triggers a full-table refresh. The local router sends its full table to the peer; the peer sends its full table back; the local router processes both. The cost is proportional to the table size.

OSPF SPF thrash. A flapping OSPF link triggers a new LSA; the LSA is flooded; every router in the area runs SPF to recompute the SPT (Shortest Path Tree). The cost is proportional to the number of LSAs and the area size; a large area with many routers can saturate ospfd.

Route flap. A link or a peer oscillates between Up and Down. Each transition generates a route update; the cumulative cost can saturate zebra even if no single update is large.

Reading the evidence

The operator’s first three commands:

show process cpu history
show ip route summary
show ip bgp summary

show process cpu history reveals whether zebra is consuming CPU. show ip route summary reveals the route-table size and the protocol distribution. show ip bgp summary reveals the BGP peer state and message counts.

For BGP-specific churn:

show ip bgp <prefix>
show ip bgp flap-statistics
show ip bgp dampening

show ip bgp <prefix> shows the route’s history (if it has flapped, the history is recorded). show ip bgp flap-statistics shows the cumulative flap count per prefix. show ip bgp dampening shows the dampening state per prefix.

For OSPF-specific churn:

show ip ospf database
show ip ospf interface
show ip ospf neighbor

show ip ospf database shows the LSA count and age; a high count with low ages indicates active SPF. show ip ospf interface shows the cost and hello/dead timers; a fast hello/dead interval can amplify churn. show ip ospf neighbor shows the neighbour state and the number of state changes.

flowchart LR
  S["Symptom<br/>(zebra CPU high, route instability)"]
  S --> Q1{Which daemon is hot?}
  Q1 -->|zebra + bgpd| BGP["BGP churn"]
  Q1 -->|ospfd| OSPF["OSPF SPF"]
  Q1 -->|all three| FLAP["Route flap"]
  BGP --> BQ{Peer-driven or local?}
  BQ -->|peer-driven| B1["BGP dampening<br/>max-prefix"]
  BQ -->|local-driven| B2["Local config change"]
  OSPF --> OQ{Link flap or design?}
  OQ -->|link flap| O1["Stabilise the link"]
  OQ -->|design| O2["Stub areas<br/>summarisation"]
  FLAP --> FQ{Source?}
  FQ -->|uplink| F1["ISP coordination"]
  FQ -->|downlink| F2["Device replacement"]

The canonical fix: BGP dampening

BGP route flap dampening (RFC 2439) suppresses routes that flap repeatedly. Each flap adds a penalty; the penalty decays exponentially; when the penalty exceeds a threshold, the route is suppressed (not advertised). The route reappears when the penalty decays below a reuse threshold.

VyOS configuration:

configure
set protocols bgp <asn> dampening 15 750 1000 15
commit

The arguments are reuse-threshold suppress-threshold max-penalty-time decay-time. The values 15 750 1000 15 mean: reuse at penalty 750, suppress at penalty 1000, max-penalty-time 15 minutes, decay half-life 15 minutes.

The operator validates:

show ip bgp dampening
# Routes with active penalties are listed

The canonical fix: maximum-prefix

maximum-prefix bounds the number of prefixes the local router will accept from a peer. If the peer sends more than the configured number, the session is reset (or a warning is logged if warning-only is configured).

VyOS configuration:

configure
set protocols bgp <asn> neighbor <peer-ip> address-family ipv4-unicast maximum-prefix 500000 warning-only
commit

The operator configures maximum-prefix to the expected peer prefix count plus a 20% margin. If the peer sends more, the session resets and the operator is alerted (via log, SNMP, or telemetry).

The discipline: maximum-prefix is the seatbelt for BGP. Every BGP peer in production should have maximum-prefix configured; a missing maximum-prefix is a configuration defect that allows a peer misconfiguration to consume the local router’s RIB.

The canonical fix: OSPF area design

OSPF SPF thrash is bounded by area design. The discipline:

• Small areas. Each router in an area runs SPF for every LSA change. A 50-router area has 50 routers running SPF on every link flap. A 5-router area has 5. The fix is to break the network into smaller areas.
• Stub areas. A stub area does not receive external (redistributed) routes; the ABR injects a default route. The result: the stub area’s routers do not run SPF for external changes. The fix is to designate customer-edge areas as stub.
• Summarisation. An ABR summarises the area’s prefixes into a single LSA. The backbone sees only the summary; a flap inside the area does not propagate to the backbone. The fix is area <id> range <summary>.

VyOS configuration:

configure
set protocols ospf area 1 range 10.0.0.0/16
set protocols ospf area 1 stub
commit

The operator validates:

show ip ospf database
# The backbone's LSDB should show the summary LSA, not the internal specifics

The canonical fix: route-map filtering

A route-map applied to a BGP peer filters which routes the peer can advertise. The discipline: every BGP peer should have an inbound route-map that bounds what it can send.

VyOS configuration:

configure
set policy prefix-list ALLOWED-FROM-ISP-A rule 10 permit 10.0.0.0/8
set policy route-map FROM-ISP-A rule 10 action permit
set policy route-map FROM-ISP-A rule 10 match ip address prefix-list ALLOWED-FROM-ISP-A
set protocols bgp <asn> neighbor <peer-ip> address-family ipv4-unicast route-map FROM-ISP-A in
commit

The route-map rejects any prefix not in the allowed list. A peer misconfiguration that sends a private prefix, a bogon, or an unexpected prefix is rejected before it enters the RIB.

Production failure modes

The route-churn failure modes the operator encounters:

• Peer sends full table unexpectedly. A peer was supposed to send customer prefixes but sends the full table. zebra saturates; the router flaps. Fix: inbound route-map with prefix-list; maximum-prefix warning-only; investigate why the peer is sending more than expected.
• Flapping uplink. The uplink to an ISP bounces; every transition generates a route update. zebra saturates. Fix: ISP coordination; BFD (Part XXXII) for fast failure detection without churn; physical link stabilisation.
• OSPF area too large. A single area with 200 routers; every flap triggers SPF on every router. ospfd saturates. Fix: split into smaller areas; stub areas; summarisation.
• Route-flap propagation. A downstream link flaps; the flap propagates up through OSPF; the upstream BGP sessions reflect the churn. Fix: dampening on the BGP side; stub area design on the OSPF side.
• iBGP convergence storm. A route reflector reset triggers a full-table refresh; every iBGP peer sends its table to every other peer; the network saturates. Fix: BGP add-path for incremental updates; route-refresh bounding; graceful restart.
• Redistribution amplifying churn. A static route flap (interface bouncing) is redistributed into OSPF; OSPF propagates the flap; BGP propagates the flap. Fix: passive interface default; passive-interface for the flapping interface; redistribution route-maps.

Rollback

Route-churn fixes are typically small but the impact is large (a too-aggressive dampening can suppress legitimate routes). The rollback discipline:

• BGP dampening — delete protocols bgp <asn> dampening and commit. The dampening stops immediately; previously suppressed routes reappear.
• maximum-prefix — delete protocols bgp <asn> neighbor <peer-ip> address-family ipv4-unicast maximum-prefix and commit. The bound is removed; the peer can send any number of routes.
• OSPF area changes — delete protocols ospf area <id> or delete protocols ospf area <id> range and commit. The area is removed or the summary is removed; the LSAs return.
• Route-map changes — delete protocols bgp <asn> neighbor <peer-ip> address-family ipv4-unicast route-map and commit. The filter is removed; the peer can send any prefix.

For all changes, use commit-confirm 5:

configure
# ... make the change ...
commit-confirm 5
# If the change causes routing instability, the auto-rollback restores
# the previous configuration.

Production discipline

Cross-course references

• Part L-01 (L-VyOS-Performance / CPU saturation) covers the diagnostic method that identifies zebra CPU saturation.
• Part XXXII (XXXII-VyOS-BFD) covers BFD for fast failure detection that reduces route-flap amplification.
• Part XXIV (XXIV-VyOS-BGP) and Part XXVIII (XXVIII-VyOS-BGP-PrefixFiltering) cover BGP peer configuration and the prefix-list primitives that this lesson assumes.
• Part XIX (XIX-VyOS-OSPF) and Part XX (XX-VyOS-OSPF-Areas) cover OSPF area design.
• Part XXXIV (XXXIV-VyOS-Redistribution) covers redistribution route-maps that bound what is redistributed between protocols.
• The Observability course covers the telemetry side (route-churn dashboards in Grafana, alerts on flap rates).
• The Ansible course’s XLII-Ansible-BeyondLinux covers the automation hand-off (BGP peer configuration validation in Ansible playbooks).

Quiz