Confederations — sub-AS, AS_CONFED segments, eBGP-within-confederation, and when to use RR vs confederation

The confederation is the alternative scaling mechanism to route reflection. Where the route reflector breaks the iBGP split-horizon rule with a single hub (and a cluster-id to deduplicate), the confederation splits the AS into sub-ASes with private ASNs and treats the sub-AS sessions as eBGP. The two patterns are different in spirit: the route reflector preserves the iBGP design and adds a single exception; the confederation replaces the iBGP design with a sub-AS eBGP mesh.

The choice between them is the central scaling decision for a large AS. The route reflector is simpler to configure and operate; the confederation is more structured and easier to reason about in a complex topology. Most production ASes use route reflection. A few use confederations.

The federation design

The confederation is a federation of sub-ASes. Each sub-AS is a private ASN (in the 64512..65534 range for 2-byte, or a within-AS 4-byte range for 4-byte). The sub-ASes form eBGP sessions between them, but eBGP-within-confederation is a special case: the sub-AS path is recorded inside the confederation using AS_CONFED_SEQUENCE segments rather than ordinary AS_SEQUENCE ones, and the session keeps the iBGP-style attribute handling that makes a single AS behave as a single AS.

flowchart TD
  subgraph CONFED["AS 65001 (public)"]
    subgraph SUB1["Sub-AS 65010 (private)"]
      R1["Router 1"]
      R2["Router 2"]
    end
    subgraph SUB2["Sub-AS 65020 (private)"]
      R3["Router 3"]
      R4["Router 4"]
    end
    subgraph SUB3["Sub-AS 65030 (private)"]
      R5["Router 5"]
      R6["Router 6"]
    end
  end
  OUTSIDE["Other AS"]
  R1 -->|"eBGP-within-confederation"| R3
  R3 -->|"eBGP-within-confederation"| R5
  R1 -->|"eBGP"| OUTSIDE
  R5 -->|"eBGP"| OUTSIDE

The sub-ASes are connected by eBGP-within-confederation. The routes are advertised inside the confederation with the sub-AS recorded in the path. The confederation identifier (the public ASN) is the ASN that the rest of the Internet sees; the sub-ASes are not visible outside the confederation.

AS_CONFED_SEQUENCE and AS_CONFED_SET

The most common misconception about confederations is that they add new path attributes. They do not. RFC 5065 adds two new segment types inside the existing AS_PATH attribute:

This distinction is not pedantry — it is the reason the mechanism works at all. Because the sub-AS path lives inside AS_PATH, every BGP speaker that already parses AS_PATH parses it. There is no new attribute to negotiate and no “unknown attribute” behaviour to reason about. And because it lives inside AS_PATH, the confederation border router can strip it by editing one attribute on egress rather than by suppressing a separate one.

AS_CONFED_SEQUENCE is the inside-confederation equivalent of AS_SEQUENCE: an ordered list of sub-ASNs, built up as the route traverses sub-ASes. AS_CONFED_SET is the inside-confederation equivalent of AS_SET, produced by aggregation.

Two consequences follow from the segment design, and both show up in operations:

• The confed segments are removed at the confederation border. RFC 5065 requires a speaker advertising to a peer outside the confederation to remove all AS_CONFED segments from AS_PATH and prepend the confederation identifier instead. The outside world sees 65001, never 65010.
• The confed segments do not count toward AS path length by default. A route that crossed three sub-ASes has the same AS-path length as one that crossed none, which is what makes the confederation behave like one AS at best-path step 4. set protocols bgp parameters bestpath as-path confed changes that, and it is a router-wide behaviour change.

flowchart LR
  R1["Router 1 (sub-AS 65010)"]
  R3["Router 3 (sub-AS 65020)"]
  R5["Router 5 (sub-AS 65030)"]
  OUT["Outside AS"]
  R1 -->|"AS_CONFED_SEQUENCE (65010)"| R3
  R3 -->|"AS_CONFED_SEQUENCE (65010 65020)"| R5
  R5 -->|"AS_PATH 65001 (public)"| OUT

In show ip bgp output the confederation portion of the path is printed inside parentheses — (65010 65020) 64512 i — which distinguishes it at a glance from an AS_SET, printed in braces. Seeing a bare 65010 where you expected (65010) inside the confederation, or any sub-ASN at all on a route received from outside, is the leak this lesson’s failure section is about.

The eBGP-within-confederation semantics

The session is configured as eBGP (the ASNs differ), but the attribute handling is iBGP-like:

• The sub-AS path is preserved inside the confederation (the AS_CONFED_SEQUENCE grows).
• LOCAL_PREF is propagated across sub-AS boundaries, which an ordinary eBGP session would not do. This is what lets one policy decision hold across the whole confederation.
• NEXT_HOP is preserved across sub-AS boundaries rather than being rewritten to the advertising router, as ordinary eBGP does.
• Communities are propagated across the confederation.
• MED is not compared between paths from different sub-ASes by default. RFC 5065 treats the confederation as one AS, but FRR keeps the comparison off until you ask for it with set protocols bgp parameters bestpath med confed. An operator who assumes the confederation makes MEDs comparable, and does not set the knob, has a policy that silently does nothing.

The 1.5 configuration tree

This is a family that moved in VyOS 1.4 and it moved in shape, not just in name. Three separate changes apply:

The session to a sub-AS peer is an ordinary neighbour stanza. The remote-as is the sub-AS, not the public ASN — nothing in the neighbour stanza marks it as a confederation peer:

set protocols bgp neighbor 10.255.0.3 remote-as '65020'

What makes that session eBGP-within-confederation is the pair of parameters confederation nodes. The identifier is the public ASN:

set protocols bgp parameters confederation identifier '65001'

The peers node lists the other sub-ASes in the same confederation, and it is a multi-value leaf: one value per set, not a quoted space-separated list.

set protocols bgp parameters confederation peers '65020'
set protocols bgp parameters confederation peers '65030'

The configuration walk

The configuration on Router 1 (sub-AS 65010):

configure
set protocols bgp system-as 65010
set protocols bgp parameters router-id '10.255.0.1'
set protocols bgp parameters confederation identifier '65001'
set protocols bgp parameters confederation peers '65020'
set protocols bgp parameters confederation peers '65030'

set protocols bgp neighbor 10.0.0.2 remote-as '65010'
set protocols bgp neighbor 10.255.0.3 remote-as '65020'
set protocols bgp neighbor 192.0.2.2 remote-as '64512'
commit
save

The router’s local AS is 65010 (the sub-AS). The confederation identifier is 65001 (the public ASN). The confederation peers are 65020 and 65030 (the other sub-ASes). The neighbour 10.0.0.2 is inside the same sub-AS, so it is plain iBGP. The neighbour 10.255.0.3 is in sub-AS 65020 and therefore eBGP-within-confederation. The neighbour 192.0.2.2 is in AS 64512 — real eBGP, and the point at which the confed segments are stripped and 65001 is prepended.

The configuration on Router 3 (sub-AS 65020):

set protocols bgp system-as 65020
set protocols bgp parameters router-id '10.255.0.3'
set protocols bgp parameters confederation identifier '65001'
set protocols bgp parameters confederation peers '65010'
set protocols bgp parameters confederation peers '65030'

set protocols bgp neighbor 10.0.0.4 remote-as '65020'
set protocols bgp neighbor 10.255.0.1 remote-as '65010'
set protocols bgp neighbor 10.255.0.5 remote-as '65030'

The confederation identifier is 65001, the same value on every router in the confederation. That is the invariant: the identifier is a property of the confederation, the system-as is a property of the router, and the peers list is a property of the router’s view of the confederation.

The configuration on Router 5 (sub-AS 65030):

set protocols bgp system-as 65030
set protocols bgp parameters router-id '10.255.0.5'
set protocols bgp parameters confederation identifier '65001'
set protocols bgp parameters confederation peers '65010'
set protocols bgp parameters confederation peers '65020'

set protocols bgp neighbor 10.0.0.6 remote-as '65030'
set protocols bgp neighbor 10.255.0.3 remote-as '65020'
set protocols bgp neighbor 198.51.100.1 remote-as '64513'

Router 5 is the second confederation border router: its session to 198.51.100.1 in AS 64513 is where confed segments are stripped on that side.

The choice between RR and confederation

The two patterns offer different trade-offs:

The route reflector is simpler to configure and operate. The confederation is more structured and easier to reason about in a complex topology.

The route reflector is the canonical choice for a small to medium AS (up to a few hundred routers). The confederation is the canonical choice for a large AS with multiple geographies or multiple operator teams.

The route reflector and the confederation can be combined: a confederation with one or more route reflectors inside each sub-AS. That is the canonical production design for a genuinely large AS, and it is also where the two mechanisms’ failure modes compound — a route can now be suppressed by a cluster-list check or a confed-segment loop check, and the two look nothing alike in the output.

How the result is validated

show ip bgp
show ip bgp 198.51.100.0/24
show ip bgp 198.51.100.0/24 json
show ip bgp regexp _65010_
show ip bgp summary
show configuration commands | match confederation

The first shows the BGP table. The second shows the per-route detail including the confederation portion of the path. The third is the JSON form, which is the one to parse in automation because it separates the segments instead of relying on brackets. The fourth finds every route whose path mentions a given sub-AS — run it on a border router against a sub-ASN and it should return nothing for routes learned from outside. The fifth shows the sessions. The sixth is the only reliable way to confirm which sub-ASes this router will treat as confederation-internal.

$ show ip bgp 198.51.100.0/24

BGP routing table entry for 198.51.100.0/24
Paths: (1 available, best #1, table default)
(65010 65020) 64512
  10.255.0.3 from 10.255.0.3 (10.255.0.3)
    Origin IGP, localpref 100, valid, confed-external, best (First path received)

Two markers to read: the parenthesised (65010 65020) is the AS_CONFED_SEQUENCE, and the peer type reads confed-external rather than external or internal. On the same prefix received by a peer outside the confederation, both disappear — the path there should read 65001 64512 with nothing in parentheses.

How it fails

Production failure modes:

• The confederation identifier is wrong or missing. With it set to the sub-AS, the border router prepends the private ASN and leaks it to the DFZ. With it missing entirely, the sub-AS sessions are ordinary eBGP: LOCAL_PREF stops crossing sub-AS boundaries and NEXT_HOP is rewritten at every hop, so policy that used to hold AS-wide quietly stops applying one sub-AS away from where it is configured.
• The confederation peers list is missing a sub-AS. The session to that sub-AS is treated as plain eBGP. The session still establishes and routes may still flow, which is what makes this hard: the symptom is not “no routes” but “routes with the wrong attributes”, and the router that is wrong is the one that is missing the entry, not the one sending.
• The peers list was written as a quoted list. See the callout above — one string where the tree wants repeated values.
• bestpath med confed assumed rather than set. MEDs from different sub-ASes are not compared by default, so a MED-based egress policy inside the confederation does nothing.
• The sub-AS is too small. A sub-AS with one router is a single point of failure and gains none of the structural benefit. At least two routers per sub-AS.
• Renumbering a sub-AS in production. Changing system-as resets every session on the router and requires editing the peers list on every other router in the confederation. There is no partial-migration path; plan it as a maintenance window, not a change.

Rollback

Confederation configuration is regular VyOS configuration, but it is not a low-risk change: altering system-as, the identifier, or the peers list tears down and rebuilds sessions.

• compare to see the diff before commit.
• commit-confirm 5 for any remote change: it commits, starts a five-minute timer, and reverts to the previously saved configuration unless confirm is entered inside the window. On a confederation change the sessions bounce twice — once on commit, once on revert — so size the window for reconvergence, not for typing.
• rollback 1 to return to the previous revision. VyOS prompts before applying it; read the prompt.
• load /config/archive/<known-good-file> followed by commit to return to a specific snapshot.

The operator who changes the confederation configuration must also know that the change is disruptive by construction. The sub-AS relationships are re-evaluated and the routes are re-advertised with different path attributes.

Production discipline

Cross-course references

The Linux course’s XIX-Linux-NetFoundations covers the FIB. The OPNsense course’s XXX-OPNsense-DynamicRouting covers the equivalent FRR-managed confederation on the firewall side. The BGP lessons vyos-xxx-01-rr-concept, vyos-xxx-02-rr-config, and vyos-xxx-03-cluster-id cover the route reflector pattern; the lesson vyos-xxx-05-multipath-relax covers load-balancing across equal-cost paths, and vyos-xxx-06-rr-troubleshoot walks the debugging of reflection failures. The best-path lesson vyos-xxvii-01-best-path-algorithm covers the bestpath as-path confed and bestpath med confed knobs referenced here.

Quiz