BGP authentication and GTSM

BGP authentication protects the TCP session that carries routing updates. It does not make a route valid, and it does not replace prefix filters. On VyOS 1.5 LTS, the documented BGP password option creates TCP-MD5 protection. TCP-MD5 is standardized by RFC 2385 and uses a 16-byte MD5 signature option on TCP segments.

Modern designs also discuss TCP-AO, HMAC-SHA, key IDs, and key chains under RFC 5925 and RFC 5926. The operator must distinguish protocol direction from VyOS configuration exposure: a current 1.5 LTS tree may document TCP-MD5 and GTSM while a particular FRR or platform build exposes a different TCP-AO integration. Do not invent a BGP command that the installed build does not support. This lesson explains both the interoperable VyOS command and the security concepts needed to select a stronger mechanism deliberately.

TCP-MD5 and RFC 2385

RFC 2385 defines the TCP MD5 Signature Option. The signature covers the TCP pseudo-header, the TCP header, the segment data, and a shared key. The receiver drops a segment when its calculated digest does not match. A forged TCP reset is therefore much harder to inject into a session that has the key.

The original option has no algorithm negotiation. Both endpoints must use the same configured secret, and the protection applies to the TCP segments associated with that socket. The key is not a BGP path attribute and is never advertised in an UPDATE.

sequenceDiagram
  participant L as Local bgpd
  participant T as TCP path
  participant R as Remote bgpd
  L->>T: TCP SYN with RFC 2385 digest
  T->>R: signed SYN
  R-->>T: signed SYN-ACK
  T-->>L: signed SYN-ACK
  L->>R: signed OPEN
  R-->>L: signed KEEPALIVE and UPDATE
  Note over L,R: Missing digest means drop, not a BGP route decision

MD5 protects transport integrity, not confidentiality. Packet payload is not encrypted by this option. It also has known algorithm weaknesses and no modern key-management framework. It remains useful for legacy interoperability and for a deliberate short-term control where a provider cannot yet support a stronger method.

The VyOS password option

Configure the same secret at both ends of a neighbor:

configure
set protocols bgp system-as 65000
set protocols bgp neighbor 192.0.2.2 remote-as 65100
set protocols bgp neighbor 192.0.2.2 password CHANGE-ME-OUT-OF-BAND
commit
save

The secret must be coordinated through an approved secret store. Do not paste it into an issue, a shell history that is retained, or a ticket that travels with the saved router configuration. A saved configuration containing a BGP password is a credential artifact; protect it with the same controls as an API key.

Verify the peer and the BGP logs:

show bgp neighbor 192.0.2.2
show bgp summary
show log protocol bgp

A session with a matching key can still fail if the source, port, or TCP path is wrong. A mismatch commonly leaves the session in Active, repeatedly resets after TCP completes, or produces a packet capture with RST packets and no OPEN. The symptom is transport, not path selection.

SHA and TCP-AO direction

RFC 5925 defines the TCP Authentication Option, or TCP-AO, as the replacement direction for TCP-MD5. TCP-AO uses a different TCP option kind and supports keyed message authentication with algorithms such as HMAC-SHA-1 and HMAC-SHA-256, as defined by the relevant TCP-AO cryptographic profile. It also provides key IDs and replay-related sequence-number machinery, making controlled key changes and stronger cryptographic profiles possible.

The protocol concepts are important even when the stock VyOS 1.5 LTS BGP tree only exposes password:

• TCP-MD5 is the live neighbor password mechanism documented for this tree. It is MD5 with a shared secret.
• TCP-AO is a separate TCP option and security association model. A peer cannot silently switch an established connection from MD5 to TCP-AO; endpoints must agree on the mechanism and key IDs.
• SHA is a family of algorithms and is not a synonym for password. A platform that claims TCP-AO support must name the MAC, KDF, key ID, and direction of support.
• Key chain is a management pattern for overlapping send and receive keys. It is not a replacement for the peer stanza.

For a lab or supported FRR build, a key-chain concept may look like the following, but the binding command depends on the installed version:

set key chain BGP-AO-KEYS key 1 key CHANGE-ME-OUT-OF-BAND
set key chain BGP-AO-KEYS key 1 cryptographic-algorithm hmac-sha-256
set key chain BGP-AO-KEYS key 1 send-lifetime start 2026-08-15T00:00:00+00:00

The example creates a SHA-256-keyed key-chain object; it does not by itself configure a VyOS BGP neighbor to consume that object. On a build that exposes TCP-AO, use the peer-specific TCP-AO syntax from the installed FRR documentation and validate it in a lab. Never assume that a key chain named here is referenced by neighbor password.

Key rotation

A single shared key is a coordination hazard. If one router changes while the other does not, the TCP connection fails during the rotation. A production plan uses an overlap window or a platform- specific TCP-AO key-ID change procedure.

For the documented TCP-MD5 option, the safe sequence is:

1. Generate a new secret in the approved secret store.
2. Add the new secret to one side without removing the old secret.
3. Confirm the peer has the same new secret and the session remains stable.
4. Move the other side to the new secret.
5. Remove the old secret only after both sides have accepted it and the rollback window has closed.
6. Record the key identifier, timestamps, and validation evidence.

Do not put the old and new secret in the same unencrypted ticket. If a rotation fails, restore the previous secret on the side that was changed, confirm TCP is stable, and then schedule a new coordinated change. A session outage is preferable to a silently unprotected session.

GTSM and TTL security

The Generalized TTL Security Mechanism, or GTSM, protects a peer from spoofed packets arriving from farther away than expected. The current VyOS tree exposes:

configure
set protocols bgp neighbor 192.0.2.2 ttl-security hops 1
commit
save

When GTSM is enabled, the local speaker sends with TTL 255 and accepts incoming packets only from the configured number of hops. The option is mutually exclusive with ebgp-multihop. GTSM is a reachability constraint, not authentication by itself. Pair it with a matching source and return policy, and use TCP-MD5 or TCP-AO when the threat model requires secret-keyed segment authentication.

A direct eBGP peer normally has TTL 1. A multihop design raises the TTL and needs a deliberate allowed hop count. Choosing GTSM and multihop together creates an impossible or contradictory contract and will fail commit or validation on a supported build.

Validation and failure modes

Use the packet boundary to choose the first test:

ss -tnp | grep ':179'
tcpdump -ni any host 192.0.2.2 and tcp port 179

Then use the protocol boundary:

show bgp summary
show bgp neighbor 192.0.2.2
show log protocol bgp

Common failures are:

• No TCP handshake. The source, route, port, or firewall is wrong; authentication has not been reached.
• TCP completes then RST repeats. The secrets or the transport authentication option differ.
• OPEN is sent but the peer resets. The transport may be valid while AS, capability, or family negotiation fails.
• TTL security rejects a legitimate path. The source is not at the configured hop distance, or the design was changed from multihop to GTSM without returning the route to the expected topology.
• Key rotation is one-sided. The active configuration and the saved secret record disagree.
• The command is unsupported. A key-chain object was created, but the running FRR build has no TCP-AO peer binding. Remove the object or use the documented transport method.

Rollback

For a bad TCP-MD5 secret:

configure
delete protocols bgp neighbor 192.0.2.2 password
commit
save

For a GTSM change:

configure
delete protocols bgp neighbor 192.0.2.2 ttl-security hops
commit
save

For a failed TCP-AO experiment, restore the previous peer-specific mechanism and its known-good key, then remove the unsupported or unreferenced key-chain object. Verify that the session is not left with a partial security state.

Cross-course references

XXIV-VyOS-BGPSession covers the neighbor and multihop contracts that security depends on. LIII-VyOS-Security broadens control-plane hardening, while XXVIII-VyOS-BGPPrefixFilters protects route content. XXXI-VyOS-BGPTroubleshoot uses the Active-to-OpenSent state model for authentication incidents. The OPNsense course demonstrates the same FRR and TCP concerns on a BSD platform, and the Linux course covers the TCP socket and packet filter underneath the session.

Quiz