OSPF authentication — plaintext, MD5, SHA-256, key chain, virtual links

OSPF carries authentication in the header of every OSPF packet. Without authentication, an attacker who can inject packets into an OSPF-speaking segment can poison the LSDB — advertise any prefix as an OSPF route — and redirect traffic. Authentication prevents this: the OSPF speakers validate that the packet was signed by a router that knows the shared key.

This lesson covers the four authentication types VyOS 1.5 LTS / FRR 10.x support (plaintext, MD5, SHA-256), the key-chain mechanism for key rotation, and the special case of virtual-link authentication.

The authentication model

OSPF’s authentication lives in the OSPF header (RFC 2328 section 8.1). Every OSPF packet — Hello, DD, LSR, LSU, LSAck — carries an authentication field. The receiver validates the field; on failure, the packet is dropped silently.

flowchart TB
  subgraph "OSPF packet"
    HDR["OSPF header<br/>version | type | packet length | router-id | area-id | checksum | auth-type | auth-data"]
    BODY["OSPF body<br/>(Hello / DD / LSR / LSU / LSAck)"]
  end
  HDR --> BODY
  AUTH["auth-type: 0 (none) | 1 (plaintext) | 2 (MD5) | 3 (SHA-256 via key chain)"]
  AUTH --> HDR

Plaintext authentication

Plaintext authentication sends the shared key in clear text in every OSPF packet. The cryptographic protection is zero; any attacker with a packet capture can read the key. Plaintext exists for interop with legacy peers that do not support MD5 or SHA-256; new deployments should never use it.

set protocols ospf interface eth0 authentication plaintext-password SHAREDSECRET
commit
save

The configuration accepts the password directly. FRR stores the hash in its running configuration; the rendered VyOS config shows the plaintext.

The matching requirement: both ends must use the same password. A typo is the most common production failure — the operator types SHAREDSECRE on R1 and SHAREDSECRET on R2, and the adjacency does not form.

MD5 authentication

MD5 uses a key-id (so multiple keys can coexist during rotation) and a shared MD5 key. The header carries an MD5 digest; the receiver validates that the digest matches the expected value for the packet.

set protocols ospf interface eth0 authentication md5 key-id 1 md5-key SHAREDSECRET
commit
save

The key-id (1 in the example) lets two routers have multiple MD5 keys in flight during a rotation. Both ends must agree on the key-id; a mismatch is the second most common production MD5 failure.

MD5 is acceptable in production for legacy interop, but it is on the FIPS-restricted list for federal environments and most new deployments should prefer SHA-256.

SHA-256 authentication via key chain

SHA-256 OSPF authentication in VyOS 1.5 LTS / FRR 10.x is configured via a key chain. The key chain is a separate configuration tree (set key chain ...) that the OSPF interface block references by name.

set key chain OSPF-KEYS key 1 key SECRET-1
set key chain OSPF-KEYS key 1 cryptographic-algorithm hmac-sha-256
set key chain OSPF-KEYS key 1 send-lifetime start 2026-01-01T00:00:00+00:00
set key chain OSPF-KEYS key 2 key SECRET-2
set key chain OSPF-KEYS key 2 cryptographic-algorithm hmac-sha-256
set key chain OSPF-KEYS key 2 send-lifetime start 2026-07-01T00:00:00+00:00
set protocols ospf interface eth0 authentication key-chain OSPF-KEYS
commit
save

The key chain has two keys:

• Key 1 is active from 2026-01-01.
• Key 2 becomes active from 2026-07-01.

The router uses whichever key is currently in its accept lifetime for incoming packets and whichever key is in send lifetime for outgoing packets. This lets the operator rotate keys without downtime: the new key is added to the chain before the rotation time; both ends accept both keys during the overlap window; the old key is removed after the rotation.

sequenceDiagram
  participant Op as Operator
  participant R1 as R1
  participant R2 as R2
  Op->>R1: set key chain OSPF-KEYS key 1 ...
  Op->>R1: set key chain OSPF-KEYS key 2 send-lifetime start 2026-07-01
  Op->>R1: commit
  Op->>R2: same configuration
  Op->>R2: commit
  Note over R1,R2: 2026-01-01 to 2026-06-30 — both use key 1
  Note over R1,R2: 2026-07-01 — both switch to key 2 simultaneously
  Op->>R1: delete key chain OSPF-KEYS key 1
  Op->>R1: commit
  Op->>R2: delete key chain OSPF-KEYS key 1
  Op->>R2: commit
  Note over R1,R2: post-rotation — only key 2 in the chain

The cryptographic-algorithm hmac-sha-256 is the hash function. RFC 7474 specifies HMAC-SHA-256; other algorithms (MD5, SHA-1) are also available in the key chain but should not be used for new deployments.

Per-area authentication

Per-area authentication overrides per-interface authentication for every interface in the area. The configuration block was introduced in lesson xix-03; the authentication primitives are the same:

set protocols ospf area 10 authentication md5 key-id 1 md5-key SHAREDSECRET
set protocols ospf area 10 authentication key-chain OSPF-KEYS
commit
save

Per-area authentication is the right choice when an entire area must share a single authentication policy. The matching requirement: every router with an interface in the area must agree on the authentication type and the key.

Virtual-link authentication

A virtual link is the OSPF mechanism for repairing a partitioned backbone. If area 0 becomes partitioned (because a link between two ABRs failed and a non-backbone area is the only remaining path between the two halves), a virtual link through the non-backbone area restores the partition. The virtual link is a logical point-to-point adjacency between the two ABRs.

flowchart LR
  subgraph "Before"
    ABR1["ABR-1<br/>area 0 (left)"]
    ABR2["ABR-2<br/>area 0 (right)"]
    X["partitioned<br/>(no link)"]
    ABR1 -.- X
    X -.- ABR2
  end
  subgraph "After"
    ABR1A["ABR-1"] -- "virtual-link through area 10" --> ABR2A["ABR-2"]
    subgraph "Area 10 (transit)"
      TL["transit area"]
    end
    ABR1A --> TL
    ABR2A --> TL
  end

The virtual link is authenticated like any other adjacency. The configuration block:

set protocols ospf area 10 virtual-link 10.255.0.2 authentication md5 key-id 1 md5-key SHAREDSECRET
commit
save

The 10.255.0.2 is the router-id of the remote ABR (the virtual-link endpoint). The authentication must match the remote ABR’s configuration.

How the result is validated

show ip ospf interface eth0            # per-interface auth
show ip ospf neighbor detail           # neighbour with auth details
show ip ospf                           # area-level auth
tcpdump -ni eth0 proto ospf            # capture and read the auth field

A working baseline shows:

• The interface lists the authentication type in show ip ospf interface (e.g. Authentication: MD5, key-id 1).
• The neighbour reaches Full and stays Full.
• A packet capture shows non-zero content in the authentication field; the field is not zero (which would indicate “no authentication”).

For a virtual link:

show ip ospf neighbor                   # shows the virtual-link neighbour
show ip ospf interface                  # shows the virtual-link as a tunnel interface

The virtual-link neighbour appears in show ip ospf neighbor with the remote router-id and the transit area.

How it fails

The production failure modes the engineer must recognise:

• Plaintext vs MD5 mismatch. R1 has plaintext, R2 has MD5. No adjacency. The Hello packets carry different auth-type fields.
• MD5 key-id mismatch. R1 has key-id 1, R2 has key-id 2. No adjacency. Both sides compute the MD5 with different keys and the digests disagree.
• MD5 password mismatch. Same key-id, different password. No adjacency. The MD5 digests disagree.
• Key chain with no matching key. R1’s chain has key 1 active, R2’s chain has key 2 active. The auth-type matches but the keys do not; no adjacency.
• Key chain send-lifetime gap. R1 sends key 1 from 2026-01-01, R2 sends key 1 from 2026-02-01. During January, R1 sends key 1 and R2 expects key 2 (or has no active send); no adjacency.
• Virtual-link authentication mismatched with transit area authentication. The virtual-link’s authentication type disagrees with the per-area authentication on the transit area. The virtual-link adjacency does not form; the backbone remains partitioned.
• Forgetting to commit key-chain changes. A common production trap: the operator updates the key chain but does not commit on both ends. One end has the new key; the other has the old; the adjacency flaps.

Rollback

The recovery from a bad authentication configuration:

• Per-interface plaintext or MD5. delete protocols ospf interface eth0 authentication and commit; save. The adjacency falls back to no authentication (or to the per-area authentication).
• Per-interface key chain. delete protocols ospf interface eth0 authentication key-chain and commit; save. The interface falls back to no authentication or per-area authentication.
• Per-area authentication. delete protocols ospf area 10 authentication and commit; save. The area falls back to per-interface authentication or no authentication.
• Virtual-link authentication. delete protocols ospf area 10 virtual-link 10.255.0.2 authentication and commit; save. The virtual link falls back to no authentication; the area’s per-area authentication applies if configured.

For any of these, rollback N inside configure reverts to a known-good revision.

Cross-course references

The VyOS lessons vyos-xviii-02-neighbours-and-adjacency covers the Hello packet and the adjacency state machine this lesson builds on. The lesson vyos-xix-02-ospf-interface-config covers the per-interface configuration block where authentication lives. The lesson vyos-xix-03-ospf-area-config covers the per-area authentication override. The VyOS lessons on key chains (Part XXIV) cover the key-chain primitive in detail. The Linux course’s XII-Linux-NetSecurity covers the host-side cryptographic primitives the OSPF authentication builds on.

Quiz