Routing protocol authentication — what VyOS actually exposes, and what it does not

Routing-protocol authentication is the operator’s first defence against an attacker who has reached a transit link, and against the much more common accident where a cable, a VLAN or a cloud subnet brings up a neighbour nobody intended. A session with authentication configured rejects packets that do not carry the matching digest; a session without it accepts whatever the wire delivers.

Each protocol exposes a different mechanism. OSPF carries the digest in the OSPF header itself. BGP carries it in a TCP option, below BGP entirely. IS-IS carries it per PDU, at three different scopes. RIP keys it per interface. The mechanisms are not interchangeable, and on VyOS they are not equally capable — which is the part of this subject most often taught wrong, because the RFCs describe a far richer toolkit than the CLI in front of you actually offers.

This lesson covers what VyOS 1.5 LTS / FRR 10.x really exposes, the rotation you can genuinely perform with it, the gaps you have to plan around instead of configuring your way out of, and the failure modes where a mismatch produces an outage with no error message anywhere.

What routing-protocol authentication protects against

The threat model is not the Internet at large. It is a party that can already transmit on a link you are running a routing protocol over:

• An attacker on the same L2 segment injecting bogus routes.
• Someone who has reached a router’s management plane and wants to inject routes from inside the estate.
• A misconfiguration — a patch panel change, a trunk carrying a VLAN it should not, a cloud subnet shared with a tenant you have never met — that brings up an unintended adjacency. This is the case that actually happens, and authentication is the control that turns it from an incident into a log line.

Authentication does not encrypt anything. An attacker who can sniff the link still reads every prefix you advertise. What it prevents is injection by a party that can transmit but does not hold the key.

flowchart LR
  A[Attacker on link] -->|inject OSPF Hello| R1[R1 eth0]
  R1 -->|verify digest| D{Digest match?}
  D -->|yes| ACC[Accept]
  D -->|no| DROP[Drop silently]
  R2[R2 eth0] -->|authentic OSPF Hello| D

Note the word silently on the drop path. That is the protocol behaving correctly, and it is also why every failure mode in this lesson presents as “the neighbour is just gone” rather than as an error.

The surface VyOS 1.5 actually gives you

Before any configuration, the honest inventory. These are the authentication nodes in the VyOS 1.5 CLI, and there are no others:

Three things are not in that table, and no amount of searching the tree will produce them:

• HMAC-SHA for OSPFv2. RFC 5709 defines it. The VyOS OSPF authentication node offers md5, plaintext-password and null, and nothing else.
• TCP-AO for BGP. RFC 5925 defines it, with key IDs and the machinery for changing keys on a live session. The VyOS BGP neighbour has exactly one authentication node and its help string reads “BGP MD5 password”.
• Key chains. There is no set key chain tree, no send-lifetime, no accept-lifetime, and nothing under protocols that references a chain by name.

OSPF authentication

OSPFv2 carries authentication in the Auth Type and Auth Data fields of every OSPF packet. RFC 2328 defines three types: null (0), simple password (1), and cryptographic (2), which in practice means keyed MD5. VyOS exposes all three.

# Per interface — this is where the key lives.
set protocols ospf interface eth0 authentication md5 key-id 1 md5-key 'S3cret-Key-16ch'

# Or, for interop with something that cannot do MD5:
set protocols ospf interface eth1 authentication plaintext-password 'weak8chr'

# Or explicitly none, overriding an area-wide setting:
set protocols ospf interface eth2 authentication null

Two constraints in that block are real and catch people: md5-key is limited to 16 characters and plaintext-password to 8. A 32-character secret out of your password manager is rejected at commit. That is not VyOS being awkward — it is the size of the fields the protocol defines — but it does mean the key for an OSPF adjacency cannot be generated by the same policy as the rest of your secrets, and the exception needs recording.

The area level sets the type for every interface in the area, without carrying a key:

set protocols ospf area 0.0.0.0 authentication md5

That is the equivalent of FRR’s area 0.0.0.0 authentication message-digest: it says packets in this area must be cryptographically authenticated. The keys still come from the interface nodes. An area configured for md5 with an interface that has no md5 key-id is a router that requires a digest it cannot produce, and the adjacency does not form.

Interface configuration takes precedence over the area setting, which is what makes authentication null on a single interface a usable escape hatch — for a link to a device that genuinely cannot authenticate, without dropping the requirement across the whole area.

Rolling an OSPF key without dropping the adjacency

VyOS has no key chain and no lifetimes, so the rotation is an ordered manual procedure rather than a scheduled one. The ordering is the whole trick, and it works because of the Key ID.

sequenceDiagram
  participant R1 as R1
  participant R2 as R2
  Note over R1,R2: Both hold key-id 1 only
  R1->>R2: OSPF packet, Key ID 1
  R2->>R1: OSPF packet, Key ID 1
  Note over R1,R2: Step 1 — add key-id 2 on BOTH routers
  R1->>R2: OSPF packet, Key ID 1 or 2
  R2->>R1: OSPF packet, Key ID 1 or 2
  Note over R1,R2: Either is accepted, because both keys are configured on both
  Note over R1,R2: Step 2 — remove key-id 1 from BOTH routers
  R1->>R2: OSPF packet, Key ID 2
  R2->>R1: OSPF packet, Key ID 2

The procedure:

1. Add the new key on both routers, keeping the old one. Both ends now hold key-id 1 and key-id 2, so whichever ID either end stamps its packets with, the other validates it.

   set protocols ospf interface eth0 authentication md5 key-id 2 md5-key 'N3w-Key-16chars'
   commit-confirm 5

2. Verify the adjacency survived — show ip ospf neighbor must still show Full on both ends — then confirm and save.

3. Remove the old key on both routers. Only after step 2 has passed on both.

   delete protocols ospf interface eth0 authentication md5 key-id 1
   commit-confirm 5

4. Verify again, then confirm and save.

The failure mode is doing step 3 before step 1 has completed on both ends. The moment one router holds only key 2 and its neighbour holds only key 1, both sides drop each other’s packets, the adjacency times out after the dead interval, and the recovery is to get the same key back onto both — under whatever routing conditions the outage has now created.

BGP authentication, and the gap

BGP authentication is not part of BGP. It is a TCP option on the session’s TCP connection, which is why its failure mode is so distinctive.

RFC 2385, TCP-MD5, is what VyOS gives you:

set protocols bgp system-as 64512
set protocols bgp neighbor 10.0.0.1 remote-as 64513
set protocols bgp neighbor 10.0.0.1 password 'CHANGE-ME-OUT-OF-BAND'

Note the 1.4/1.5 shape of the surrounding commands: system-as rather than the ASN in the node name, and neighbor as a sibling of it rather than a child of the ASN.

RFC 5925, TCP-AO, is what the security standard you have probably been handed asks for. It uses a different TCP option kind, supports HMAC-SHA-1-96 and HMAC-SHA-256 instead of MD5, and — the operationally important part — carries key IDs and master key tuples so that a key can be changed on an established session. VyOS 1.5 does not expose it. There is no node for it under the BGP neighbour, and there is no key chain for it to reference.

RIP and IS-IS

Both are less common in a modern estate than OSPF and BGP, and both have real nodes in the VyOS tree.

RIP keys per interface, with the same key-id model as OSPF:

set protocols rip interface eth0 authentication md5 1 password 'RIP-Key-16chars'

The 1 is the key ID. plaintext-password is the alternative and is limited to 16 characters. RIPv1 has no authentication at all, which is one of several reasons not to run it.

IS-IS authenticates at three scopes, and choosing the wrong one is the usual mistake:

# Per circuit — protects the hellos on this link
set protocols isis interface eth0 password md5 'ISIS-Circuit-Key'

# Per area — protects L1 LSPs within the area
set protocols isis area-password md5 'ISIS-Area-Key'

# Per routing domain — protects L2 LSPs between areas
set protocols isis domain-password md5 'ISIS-Domain-Key'

Each takes either md5 or plaintext-password as its type, with the key as the value. A circuit password alone protects the adjacency but not the LSPs that flood through it; an area password alone protects the LSPs but lets an unauthenticated neighbour form an adjacency. In a protected deployment you set all three, and you set them consistently across every router in the scope — an area password on some routers and not others partitions the area.

Failure modes

An authentication mismatch is a silent outage

The operator configures MD5 on R1 and not on R2. R1 stamps its Hellos with a digest, R2 rejects them because it expects Auth Type 0. R2 sends Hellos with no digest, R1 rejects them because it expects Auth Type 2. Both sides drop the other’s packets exactly as designed.

The adjacency goes to Down on both ends with nothing on either terminal. show ip ospf neighbor is simply empty.

show ip ospf neighbor
show ip ospf interface eth0

The interface output shows the interface as configured and up, with no neighbours, which reads exactly like a cabling fault. A mismatched key under a matching type produces the identical symptom, as does a mismatched key ID.

The diagnostic that separates authentication from cabling in one step is a capture: if OSPF packets are arriving on the interface and no adjacency forms, the packets are being rejected rather than lost.

The key was removed before the far end was ready

The rotation done out of order, described above. Both peers reject each other during the gap. This is a self-inflicted version of the previous failure and it is the reason the ordering is worth memorising rather than improvising.

A plaintext password in a configuration export

plaintext-password is exactly what it says: the password travels in every OSPF packet in the clear, and appears in show configuration commands and in every configuration backup. Anyone with configuration read access, or a copy of /config/config.boot, has the key.

Use md5 and treat even that key as a credential — MD5 keys are also readable from a saved configuration. The digest protects the wire, not the config file.

An area demands a type no interface can produce

set protocols ospf area 0.0.0.0 authentication md5 with an interface that has no md5 key-id configured. The commit succeeds because the two nodes are independently valid, and the adjacency never forms. Set the area type and the interface key in the same change, and verify before you leave.

Clock drift is not your problem here — and that is worth knowing

A key chain would make NTP a prerequisite, because lifetimes are compared against the local clock and a router whose clock has drifted would silently fall back to no key. VyOS has no key chains, so OSPF and RIP authentication on this platform have no time dependency at all.

Keep NTP correct anyway — logs correlated across routers during an outage are worth more than the effort — but do not carry across a troubleshooting step that belongs to a mechanism this platform does not have. show date is a reasonable thing to check; a key that is “not yet active” is not a state that can exist here.

Rollback

Every change in this lesson is reversible, and the safe path is a timer rather than a revision:

# Applied under commit-confirm and the adjacency dropped:
# do nothing. The window expires and the router reverts itself.

# Remove OSPF interface authentication entirely:
delete protocols ospf interface eth0 authentication
commit

# Remove one key from a rotation, leaving the other:
delete protocols ospf interface eth0 authentication md5 key-id 1
commit

# Remove the area-wide requirement:
delete protocols ospf area 0.0.0.0 authentication
commit

# Remove BGP TCP-MD5 — expect the session to reset:
delete protocols bgp neighbor 10.0.0.1 password
commit

Removing authentication restores the adjacency and leaves the session unauthenticated. That is a legitimate emergency action and an illegitimate resting state: record it, and give the re-enablement an owner and a date before you close the incident.

Prefer commit-confirm over rollback N for anything in this lesson. The VyOS documentation states that applying a rollback revision reboots the router, which on a box carrying transit is a larger event than the one you were trying to undo.

Production discipline

Cross-course references

• XIX-VyOS-OSPFConfig (vyos-xix-05-ospf-authentication) covers the OSPF authentication configuration in detail.
• XXIV-VyOS-BGPSession (vyos-xxiv-04-bgp-authentication) covers TCP-MD5 on a BGP neighbour and the TCP-AO direction.
• XXII-VyOS-OSPFTroubleshoot (vyos-xxii-04-authentication) covers the troubleshooting sequence for an authentication mismatch.
• XXXI-VyOS-BGPTroubleshoot (vyos-xxxi-01-session-states) covers why a TCP-MD5 mismatch produces no Notification and what the FSM looks like while it is failing.
• The Linux course’s XX-Linux-NetConfig covers the kernel side of the TCP MD5 socket option that BGP authentication relies on.

Quiz