iBGP full-mesh problem — why IBGP requires a full mesh, what the route reflector solves, and the loop-prevention trio (originator-id, cluster-list, cluster-id)

The iBGP full-mesh problem is the canonical scaling bottleneck of an internal BGP deployment. The rule is simple: a route learned from one iBGP peer must not be re-advertised to another iBGP peer. The rule exists to prevent loops inside the AS — without it, a route bouncing between two iBGP peers would never settle. The side effect is that every iBGP speaker must be a peer of every other iBGP speaker. For an AS with 10 routers, that is 45 sessions. For 100, it is 4 950. The full mesh does not scale.

The route reflector is the structural fix. One router (the RR) is allowed to break the split-horizon rule and re-advertise iBGP routes to other iBGP peers. The RR is a relay. The other routers in the AS are clients of the RR, and they form a partial mesh (every client peers with the RR; clients do not peer with each other). The split-horizon rule is preserved for regular iBGP peers; the RR is the exception.

The split-horizon rule and its cost

The split-horizon rule exists because BGP’s normal loop detection stops working inside an AS. Between ASes, a router discards any route whose AS_PATH already contains its own ASN — that is the whole of path-vector loop prevention. Inside an AS the AS_PATH is present but never modified, because nobody prepends on an iBGP advertisement. Every internal copy of a route therefore carries an identical AS_PATH, and a route circulating between three internal routers would look, to each of them, exactly like a fresh one. The split-horizon rule substitutes a topological restriction for the missing check: a route learned from one iBGP peer must not be re-advertised to another.

flowchart LR
  subgraph R1["Router A"]
    A1["eBGP route X"]
  end
  subgraph R2["Router B"]
    B1["iBGP route X from A"]
  end
  subgraph R3["Router C"]
    C1["iBGP route X from A"]
  end
  A1 -->|"iBGP to B"| B1
  A1 -->|"iBGP to C"| C1
  B1 -->|"refused by split-horizon"| C1

The split-horizon rule says: B learned X from A (iBGP), B cannot advertise X to C (iBGP). The route is dropped on the egress from B to C. The only way for C to learn X is to peer directly with A.

For N routers, the full mesh has N*(N-1)/2 sessions. The cost grows quadratically:

The TCP session state, the BGP table state, and the configuration management overhead all grow as O(N^2). The full mesh is the limiting factor for any AS with more than a few dozen iBGP speakers.

The route reflector as a relay

The route reflector design picks one router (or a few, for redundancy) and grants it the privilege of re-advertising iBGP routes. The RR is the single hub in a hub-and-spoke topology. Every client peers with the RR; clients do not peer with each other.

flowchart LR
  subgraph RR["Route Reflector"]
    R1["RR (the relay)"]
  end
  subgraph CLIENTS["Clients"]
    C1["Client A"]
    C2["Client B"]
    C3["Client C"]
    C4["Client D"]
  end
  R1 -->|"iBGP"| C1
  R1 -->|"iBGP"| C2
  R1 -->|"iBGP"| C3
  R1 -->|"iBGP"| C4
  C1 -.-|"no peer"| C2
  C1 -.-|"no peer"| C3
  C1 -.-|"no peer"| C4

For N routers with one RR as the hub, the session count is N-1 (every client peers with the RR). The cost grows linearly. The trade-off is the RR itself becomes a single point of failure. The production pattern is to deploy two RRs in the same cluster for redundancy (covered in vyos-xxx-03-cluster-id).

The reflection rule is the mirror of the split-horizon rule. The RR is allowed to:

• Receive iBGP routes from clients and re-advertise them to other clients (breaking the split-horizon rule).
• Receive iBGP routes from non-clients (regular iBGP peers) and re-advertise them to clients.
• Receive iBGP routes from clients and re-advertise them to non-clients.

The non-client cannot advertise a route to the RR and expect the RR to re-advertise it to another non-client. The split-horizon rule still applies to non-clients. The RR is the exception, not the replacement.

The reflection attributes, and the one that is not an attribute

RFC 4456 makes reflection safe with two new optional non-transitive path attributes, plus one piece of local configuration that feeds one of them. The distinction matters when you are reading a capture: two of these three things travel on the wire and one does not.

Calling cluster-id an “attribute” is a common shorthand and a misleading one: there is no CLUSTER_ID attribute in an UPDATE, and there is nothing to look for in a capture. What the far end sees is its effect — an extra entry on CLUSTER_LIST.

The three work together:

flowchart TD
  ORIG["Router A originates route X<br/>(router-id 10.255.0.1)"]
  RR1["RR-1 (cluster-id 1.2.3.4)<br/>stamps ORIGINATOR_ID=10.255.0.1<br/>stamps CLUSTER_LIST=[1.2.3.4]"]
  RR2["RR-2 (cluster-id 1.2.3.4)<br/>receives route with CLUSTER_LIST=[1.2.3.4]<br/>DROPS (its own cluster-id is in the list)"]
  ORIG --> RR1
  RR1 --> RR2
  RR1 --> CL["Client B receives route<br/>ORIGINATOR_ID=10.255.0.1<br/>CLUSTER_LIST=[1.2.3.4]"]
  CL --> CHK["B checks: am I the originator?<br/>10.255.0.1 == my RID? NO (B's RID is 10.255.0.2)<br/>B accepts the route"]

The originator-id check is the inner loop prevention. The cluster-list check is the outer loop prevention. The originator-id ensures the route is not reflected back to the router that originated it. The cluster-list ensures the route is not reflected back to an RR that has already processed it.

How the originator-id works

The originator-id is the router-id of the router that originally injected the route into iBGP. The RR stamps the attribute on the first reflection. The attribute is propagated through the AS. The receiver checks the attribute: if the receiver’s router-id matches the originator-id, the route is rejected.

The originator-id is the loop-prevention mechanism for the inner loop (between RRs and their own clients). The RR reflects a route to a client; the client is the originator of the route; the route is rejected.

Be precise about where that rejection happens, because it is a common source of confusion. It is a validity check applied when the UPDATE is received — the route never becomes a candidate at all. It is not a best-path tiebreaker. ORIGINATOR_ID does appear separately in the best-path comparison, where RFC 4456 says to use it in place of the peer’s BGP Identifier when it is present, but that is a different use of the same value and it only applies to routes that survived the validity check first.

flowchart TD
  A["Router A (RID 10.255.0.1)"]
  RRC["Client of RR-1"]
  RR1["RR-1 (RID 10.255.0.5)"]
  A -->|"iBGP from A"| RR1
  RR1 -->|"reflects to client A"| A
  A -->|"checks: am I originator?"| Q{"RID == ORIGINATOR_ID?"}
  Q -->|"yes (10.255.0.1 == 10.255.0.1)"| DROP["A rejects the route"]
  Q -->|"no"| ACCEPT["A accepts the route"]

The check is what makes the route reflector safe to use. Without it, the RR would happily reflect a route back to the originator, and the originator would happily accept it (because the split-horizon rule does not apply to reflected routes).

How the cluster-list works

The cluster-list is the chain of cluster-ids the route has traversed. The RR stamps its own cluster-id on every reflected route. The receiver checks the cluster-list: if the receiver’s cluster-id is already in the list, the route is rejected.

The cluster-list is the loop-prevention mechanism for the outer loop (between RRs in the same cluster). The cluster-list is grown at every RR; the check is performed at every RR.

flowchart TD
  C1["RR-1 (cluster-id 1.2.3.4)"]
  C2["RR-2 (cluster-id 1.2.3.4)"]
  C1 -->|"reflects with CLUSTER_LIST=[1.2.3.4]"| C2
  C2 -->|"checks: is my cluster-id in CLUSTER_LIST?"| Q{"1.2.3.4 == 1.2.3.4?"}
  Q -->|"yes"| DROP["RR-2 rejects the route"]
  Q -->|"no"| ACCEPT["RR-2 accepts the route"]

The cluster-id is the deduplication key. Two RRs in the same cluster share the same cluster-id; a route reflected by one is seen by the other as already-processed, and the other drops it. The mechanism is the loop-breaker between peer RRs.

The default cluster-id is the router-id of the RR. The operator can override it with set protocols bgp parameters cluster-id, and the convention is to give both RRs in a cluster the same value.

Configuration on VyOS 1.5 LTS

The configuration makes a neighbour a route-reflector client:

set protocols bgp system-as '65001'
set protocols bgp parameters router-id '10.255.0.5'

set protocols bgp neighbor 10.0.0.1 remote-as '65001'
set protocols bgp neighbor 10.0.0.1 update-source 'dum0'
set protocols bgp neighbor 10.0.0.1 address-family ipv4-unicast route-reflector-client

The single flag route-reflector-client is the difference between a regular iBGP neighbour and a client. The RR is the router on which this set command is run; the client is the neighbour named in it. Note that nothing marks a router as “a route reflector” globally — being an RR is nothing more than having at least one neighbour flagged this way.

The placement of that flag is the thing to get right on 1.4 and 1.5. It sits under the address family, not directly under the neighbour, which means it applies to exactly one family:

# IPv4 unicast reflection for this peer
set protocols bgp neighbor 10.0.0.1 address-family ipv4-unicast route-reflector-client

# IPv6 unicast is a separate decision and a separate line
set protocols bgp neighbor 2001:db8::1 address-family ipv6-unicast route-reflector-client

A dual-stack deployment that sets the flag under ipv4-unicast and forgets ipv6-unicast commits without complaint, reflects IPv4 perfectly, and silently drops every IPv6 route on the floor between clients. That is one of the two most common route-reflector misconfigurations, and the CLI gives no hint of it.

The cluster-id is a router-wide parameter rather than a per-neighbour one, so it lives under parameters:

set protocols bgp parameters cluster-id '1.2.3.4'

The value is a 32-bit number written in dotted-quad notation, the same way a router-id is. It has nothing to do with any IP address on the router — 1.2.3.4 is simply how you write the number — and the default, if you set nothing, is the RR’s own router-id.

For a two-RR cluster, the two RRs share the same cluster-id:

# On RR-1
set protocols bgp parameters cluster-id '1.2.3.4'

# On RR-2
set protocols bgp parameters cluster-id '1.2.3.4'

Both RRs now stamp the same value. A route reflected by RR-1 carries CLUSTER_LIST=[1.2.3.4]; RR-2 sees its own cluster-id already in the list and discards the route rather than reflecting it onward. That discard is the point: the two RRs serve the same clients without re-reflecting each other’s work.

How the result is validated

show ip bgp
show ip bgp <prefix>
show ip bgp <prefix> json
show ip bgp summary
show ip bgp neighbors
show ip bgp neighbors <ip> | match route-reflector

The first command shows the BGP table. The second shows the per-route details including the originator-id and cluster-list attributes. The JSON form is the most explicit:

{
  "originatorId": "10.255.0.1",
  "clusterList": ["1.2.3.4"]
}

The fifth command shows the BGP neighbour summary. The sixth command shows the per-neighbour session state, including the route-reflector flag.

How it fails

Production failure modes:

• The route is reflected back to its originator. The originator-id is missing. The fix is to make sure the RR is stamping the attribute on every reflection. The check is show ip bgp <prefix> | match originator.
• The route is reflected between two RRs in the same cluster. The cluster-list is empty. The fix is to make sure the RR is prepending its cluster-id to the list. The check is show ip bgp <prefix> | match cluster.
• The RR is not configured as a route-reflector. The neighbour is a regular iBGP peer. The split-horizon rule still applies. The fix is to add route-reflector-client to the neighbour configuration.
• The cluster-id is wrong. The two RRs are in the same cluster but have different cluster-ids. The cluster-list is not deduplicating. The fix is to make the cluster-ids identical.

Rollback

RR configuration is regular VyOS configuration. The rollback path is:

• compare to see the diff before commit.
• commit-confirm <timeout> for any remote change.
• rollback N; commit; save to revert to the previous configuration.
• load /config/archive/<known-good-file>; commit; save to revert to a specific snapshot.

The operator who changes the RR configuration must also know that the change is disruptive. The clients will see the route-reflector flag change and may re-evaluate the BGP table. The rollback must be tested before the change is committed.

Production discipline

Cross-course references

The Linux course’s XIX-Linux-NetFoundations covers the FIB. The OPNsense course’s XXX-OPNsense-DynamicRouting covers the equivalent FRR-managed route reflector on the firewall side. The BGP lessons vyos-xxiii-02-ebgp-ibgp and vyos-xxiv-01-bgp-config cover the prerequisites; the lesson vyos-xxx-02-rr-config covers the configuration of reflection, vyos-xxx-03-cluster-id covers the cluster-id and redundancy, and vyos-xxx-06-rr-troubleshoot walks the debugging of reflection failures.

Quiz