Extended communities — the 8-byte format, MPLS VPN RT/RD, and link-bandwidth

The third BGP community format is the extended community. Where the standard community is 32 bits and the large community is 96 bits, the extended community is 64 bits: a 1-byte type field and a 7-byte payload. The 1-byte type field is the breakthrough: it lets the protocol carry different kinds of metadata in the same attribute, and the type field tells the reader how to interpret the payload.

The two production-relevant sub-types are the Route Target (RT) used by MPLS L3VPN and the link-bandwidth community used for weighted unequal-cost load-balancing. The same Path Attribute (code 16) carries both; the type field distinguishes them. The Route Distinguisher (RD) is discussed alongside the RT throughout this lesson because L3VPN uses the two together — but as the next section shows, it is not an extended community at all.

The 8-byte wire format

The extended community is Path Attribute code 16, type Optional Transitive. The wire format is:

+----------------------------------------------------+
|  Flags: Optional = 1, Transitive = 1, Partial = 0  |  (1 octet)
+----------------------------------------------------+
|  Attribute Type Code = 16                          |  (1 octet)
+----------------------------------------------------+
|  Length (1 octet, in octets of the value)          |
+----------------------------------------------------+
|  Value: N × 8 octets, each one:                    |
|    Type (1 octet)                                  |
|    Sub-Type (1 octet)                              |
|    Payload (6 octets)                              |
+----------------------------------------------------+

Flags come first, then the type code, then the length — the ordering every BGP path attribute uses. A packet capture read in the other order will have you looking at the length byte and calling it the attribute type.

The first two octets are the Type/Sub-Type pair. The first octet (Type) determines the high-level category; the second octet (Sub-Type) refines it. The 6 payload octets are interpreted differently for each Type/Sub-Type combination.

For the sub-types the operator encounters in production:

The high bit of the Type octet is the transitivity flag: 0x00, 0x01 and 0x02 are transitive, and 0x40, 0x41, 0x42 are the non-transitive counterparts of the same three address forms. That is why link bandwidth — type 0x40 — stops at an AS boundary while a Route Target does not.

The display form is RT:65001:100 (Route Target), LB:65001:125000000 (Link Bandwidth), or SoO:65001:1 (Site of Origin). The prefixes are mnemonic; the wire format is the same 8 bytes for all of them.

Route Target and Route Distinguisher

The MPLS L3VPN architecture uses two 8-byte identifiers in tandem — one of them an extended community, one of them not:

• Route Distinguisher (RD) is a globally-unique identifier prepended to every VPN route to make the route’s NLRI unique. Two customers with the same prefix 10.0.0.0/24 in different VRFs become RD-A:10.0.0.0/24 and RD-B:10.0.0.0/24 — different NLRIs, no collision. The RD is only locally significant — the operator on the other side of the MPLS cloud does not need to see it.
• Route Target (RT) is an extended community that identifies which VRF the route belongs to. When a PE router receives a route carrying RT:65001:100, it installs that route into every local VRF whose import list contains 65001:100. The RT is the VRF membership signal, and because it is an attribute it can be matched and rewritten by policy.

flowchart LR
  CE1["CE-1 in VRF-A<br/>10.0.0.0/24"]
  CE2["CE-2 in VRF-B<br/>10.0.0.0/24"]
  CE3["CE-3 in VRF-C<br/>10.0.0.0/24"]
  CE1 --> PE1["PE-1<br/>RD 65001:1"]
  CE2 --> PE1
  CE3 --> PE2["PE-2<br/>RD 65001:1"]
  PE1 --> MP["MPLS backbone<br/>iBGP + RT community"]
  MP --> PE2
  PE1 -->|"RT 65001:100 import"| VRFA["VRF-A receives 10.0.0.0/24"]
  PE1 -->|"RT 65001:200 import"| VRFB["VRF-B receives 10.0.0.0/24"]
  PE2 -->|"RT 65001:300 import"| VRFC["VRF-C receives 10.0.0.0/24"]

The RT is the filter. The PE router receives a BGP UPDATE with RT:65001:100 and asks “which VRF has this RT in its import list?”. The route is installed into the matching VRF and nowhere else.

The RD is the uniqueness marker. The 8-byte RD is prepended to the IPv4 prefix to make the NLRI unique within the VPN table. FRR prints the VPN RIB grouped by RD — a Route Distinguisher: 65001:1 heading with that RD’s prefixes listed beneath it — which is the view to reach for when two VRFs carry the same prefix and you need to see which is which.

Configuring RT and RD on VyOS 1.5 LTS

There is no top-level vpn node under protocols bgp. On 1.4 and 1.5 the VRF owns its own BGP instance, and the VPN plumbing hangs off that instance’s address family:

set vrf name CUSTOMER-A table '1000'
set vrf name CUSTOMER-A description 'Customer A L3VPN'
set interfaces ethernet eth1 vrf 'CUSTOMER-A'

set vrf name CUSTOMER-A protocols bgp system-as '65001'
set vrf name CUSTOMER-A protocols bgp address-family ipv4-unicast rd vpn export '65001:100'
set vrf name CUSTOMER-A protocols bgp address-family ipv4-unicast route-target vpn both '65001:100'
set vrf name CUSTOMER-A protocols bgp address-family ipv4-unicast label vpn export 'auto'
set vrf name CUSTOMER-A protocols bgp address-family ipv4-unicast export vpn
set vrf name CUSTOMER-A protocols bgp address-family ipv4-unicast import vpn
set vrf name CUSTOMER-A protocols bgp address-family ipv4-unicast redistribute connected

set vrf name CUSTOMER-A protocols static route 10.0.0.0/24 next-hop 192.0.2.2

Read that as five separate decisions rather than one block:

• rd vpn export stamps the Route Distinguisher onto this VRF’s routes as they enter the VPN RIB. There is no matching rd vpn import, because the RD is not a filter — nothing selects on it.
• route-target vpn both sets the import and export RT to the same value, which is the common case. route-target vpn import and route-target vpn export are separate nodes when a hub-and-spoke design needs them to differ.
• label vpn export auto lets FRR allocate the MPLS label the remote PE will use to reach this VRF. Without a label there is nothing for the data plane to act on.
• export vpn and import vpn are the two directions of the gate between this VRF’s unicast RIB and the shared VPN RIB. They are the step operators most often leave out: the RD and RT are configured, show bgp ipv4 vpn is empty, and the reason is that nothing was ever exported into it.
• redistribute connected — or a network statement, or an IGP — is what puts routes in the VRF’s unicast RIB in the first place. The VPN machinery only moves what is already there.

The PE-to-PE session itself carries the VPN family, which is a separate address family from unicast and is configured in the default routing instance rather than inside the VRF:

set protocols bgp system-as '65001'
set protocols bgp neighbor 192.0.2.2 remote-as '65001'
set protocols bgp neighbor 192.0.2.2 update-source 'dum0'
set protocols bgp neighbor 192.0.2.2 address-family ipv4-vpn

If that last line is missing, the two PEs have a healthy BGP session that never exchanges a single VPN route — the family was never negotiated. show bgp ipv4 vpn summary is the command that tells you so.

For a 4-byte ASN the values look identical and the encoding differs:

set vrf name CUSTOMER-A protocols bgp system-as '200000'
set vrf name CUSTOMER-A protocols bgp address-family ipv4-unicast rd vpn export '200000:100'
set vrf name CUSTOMER-A protocols bgp address-family ipv4-unicast route-target vpn both '200000:100'

The 4-byte ASN is carried as Type 0x02 on the wire (RFC 5668) with a 2-byte local part, where a 2-byte ASN uses Type 0x00 with a 4-byte local part. The display form is AS:NN either way, so the notation hides the difference — which matters when you are choosing a numbering scheme, because the local part you have room for shrinks from 4 bytes to 2 as soon as the ASN grows.

The IPv4 form is the third option, and the one to use when the allocation scheme wants to be readable rather than compact:

set vrf name CUSTOMER-A protocols bgp address-family ipv4-unicast rd vpn export '192.0.2.1:100'

The address occupies the high-order 4 bytes and the local number the low-order 2. 192.0.2.1:100 on every PE, with 100 meaning “customer A”, is a scheme an on-call engineer can decode from the route alone.

The link-bandwidth extended community

The link-bandwidth community is defined by draft-ietf-idr-link-bandwidth, which has been an IETF draft for well over a decade and has never been published as an RFC. That is worth knowing before you design around it: implementations agree in practice, but there is no standards-track document to point a vendor at when they do not. The Type is 0x40 (non-transitive two-octet AS-specific), the Sub-Type is 0x04, and the 6-byte payload is a 2-byte AS number followed by the bandwidth as a 4-byte IEEE floating-point value in bytes per second.

The use case is per-flow unequal-cost load-balancing. A PE router with two egress links of different capacities (say 10 Gbps and 100 Gbps) wants to load-balance flows across the two links proportionally to the bandwidth. The standard eBGP/iBGP multipath logic picks equal-cost paths; the link-bandwidth extended community lets the operator override the “equal” assumption.

flowchart LR
  subgraph PE
    A["Route 10.0.0.0/24"]
  end
  subgraph "Two egress links"
    L1["10 Gbps link<br/>LB:10Gbps"]
    L2["100 Gbps link<br/>LB:100Gbps"]
  end
  A -->|"1/11 of flows"| L1
  A -->|"10/11 of flows"| L2

The PE marks the routes advertised over the 10 Gbps link with a link bandwidth of 10 Gbps and the routes over the 100 Gbps link with 100 Gbps. The receiving router installs both paths and splits flows roughly 1:10 across them.

The community is attached by a route-map on the advertising side:

set policy route-map SET-LB-10G rule 10 action 'permit'
set policy route-map SET-LB-10G rule 10 set extcommunity bandwidth '10000'

set protocols bgp system-as '65001'
set protocols bgp neighbor 192.0.2.2 remote-as '65002'
set protocols bgp neighbor 192.0.2.2 address-family ipv4-unicast route-map export 'SET-LB-10G'

The unit at the CLI is megabits per second, not the bytes per second that go on the wire — FRR takes a value in the range 1 to 25600 and does the conversion itself. 10000 is therefore 10 Gbps, and a value copied from the wire format (10000000000) is not just wrong, it is out of range and will be rejected. FRR also accepts cumulative and num-multipaths in place of a number, which derive the value rather than stating it.

On the receiving side there is no route-map to write. FRR uses the link bandwidth for weighted ECMP by default; what the receiver needs is multipath to be enabled at all, so that there is more than one path to weight:

set protocols bgp address-family ipv4-unicast maximum-paths ebgp '4'

The knob that changes the default is bgp bestpath bandwidth, and it governs the awkward case rather than the happy one: what to do when some of the candidate paths carry the community and some do not. Left alone, FRR declines to guess and falls back to plain equal-cost sharing across all of them — which is exactly the symptom that sends operators looking for a receiver-side route-map that was never the problem.

How the result is validated

Work outwards from the VRF, because each layer can be healthy while the next one is empty, and the first empty layer is the answer:

# 1. Does the VRF exist and hold the interfaces you think it does?
show vrf

# 2. Does the VRF's own unicast RIB have the routes at all?
#    Nothing can be exported that was never here.
show ip route vrf CUSTOMER-A

# 3. Did those routes reach the shared VPN RIB, and with what RD?
#    FRR groups the output under a "Route Distinguisher:" heading.
vtysh -c 'show bgp ipv4 vpn'

# 4. What is on one specific route — RT, RD, label, next hop?
vtysh -c 'show bgp ipv4 vpn rd 65001:100 10.0.0.0/24'

# 5. Is the VPN family even negotiated with the far PE?
vtysh -c 'show bgp ipv4 vpn summary'

# 6. What did VyOS actually render into FRR?
vtysh -c 'show running-config' | grep -A 20 'router bgp'

Step 3 is the one that most often ends the investigation. An empty VPN RIB with a correct-looking VRF configuration nearly always means export vpn is missing, or that step 2 was empty and there was nothing to export in the first place.

For raw wire-format inspection, sudo tcpdump -n -i any -vvv 'port 179' captures the UPDATE and the Path Attribute block. The extended community attribute is visible as code 16 in the attribute list — though for reading attributes, FRR’s own decode in step 4 is easier to trust than a hand-parsed hex dump.

How it fails

Production failure modes:

• The RT is set on the export but no VRF has the same RT on its import list. The route leaves the PE router with the RT, but no remote PE installs it into any VRF. The route is a black hole. The fix is to either add the RT to the importing VRF or remove it from the exporting route-map.
• The RD is changed on a VRF in production. The RD is locally significant but the operator changes it during a maintenance window. The VRF is no longer reachable from the rest of the network because every existing route’s NLRI has the old RD. The fix is to keep the RD stable across the lifetime of the VRF and only change it during a coordinated migration.
• The RD and RT are configured but export vpn is not. The VRF looks fully set up and show bgp ipv4 vpn is empty. Nothing was ever moved from the VRF’s unicast RIB into the VPN RIB, so there was nothing to stamp an RD onto. The fix is the missing export vpn (and import vpn for the other direction).
• The link bandwidth is only on some of the candidate paths. The receiver falls back to plain equal-cost sharing across all of them rather than guessing a weight for the ones that lack the community. The fix is to make the community consistent across every path for the prefix, or to tell FRR explicitly what to do with the gap through bgp bestpath bandwidth. Note that this is not fixed by a receiver-side route-map, and not by bgp bestpath as-path multipath-relax either — multipath-relax (see vyos-xxx-05-multipath-relax) governs whether paths from different ASes are eligible for multipath at all, which is a separate precondition.
• The link bandwidth was expected to survive an AS hop. It is a non-transitive community: the neighbouring AS uses it and does not pass it on. A design that depends on the value two AS hops away has no mechanism behind it.
• The operator confuses RT and RD. RT is the import/export filter and is an attribute; RD is the NLRI uniqueness marker and is not. A route-map that sets RT:65001:100 on a plain unicast route attaches a real extended community that no VRF import list will ever consult, so nothing happens and nothing errors. The fix is to configure RTs through the VRF’s route-target vpn nodes, where they are attached to routes that are actually in the VPN RIB.

Rollback

Extended-community configuration is regular VyOS configuration. The rollback path is:

• compare to see the diff before commit.
• commit-confirm <timeout> for any remote change.
• rollback N; commit; save to revert to the previous configuration.
• load /config/archive/<known-good-file>; commit; save to revert to a specific snapshot.

The operator who changes the RT or RD on a VRF in production must know that the change is disruptive. The VRF will lose all routes for the duration of the change; the rollback must be tested before the change is committed.

Production discipline

Cross-course references

The Linux course’s XIX-Linux-NetFoundations covers the FIB. The OPNsense course’s XXX-OPNsense-DynamicRouting covers the equivalent FRR-managed extended community handling on the firewall side. The BGP lessons vyos-xv-01-vrf-concept (VRF basics) and vyos-xv-03-vrf-routing-protocols (VRF routing) cover the prerequisites; the lesson vyos-xxix-03-large-communities covers the other 12-byte community format, and vyos-xxix-06-community-troubleshoot covers the propagation-failure debugging for all three formats.

Quiz