OSPF authentication — simple-text, MD5, SHA, key-id mismatch, key chain timing

OSPF authentication protects the routing protocol from spoofed or tampered packets. The protocol supports several authentication types: simple-text (insecure), MD5 with key-id, SHA-1, SHA-256, SHA-384, and SHA-512. The dominant authentication failure is key-id mismatch, where the two routers use different key-ids and the authentication check fails.

This lesson covers the authentication types, the key-id and key chain semantics, the diagnostic for authentication failures, and the production playbook for cryptographic-key rotation.

Authentication types

OSPF authentication types and their security properties:

The AuType field in the OSPF header indicates the authentication type. The receiving router checks the authentication against the locally configured key-id and key.

flowchart LR
  subgraph "OSPF Hello packet"
    H1["OSPF header<br/>AuType = 2 (MD5)"]
    H2["Authentication trailer<br/>Key ID = 1<br/>MD5 digest = ..."]
  end
  subgraph "Receiving router"
    R1["Local key-ID = 1<br/>Local key = SECRET"]
    R2["Compute HMAC-MD5<br/>Compare to received digest"]
  end
  H1 --> R2
  H2 --> R2
  R1 --> R2
  R2 -- "match: accept" --> R2
  R2 -- "mismatch: drop" --> R2

Simple-text authentication

The simplest authentication type. The key is sent in plaintext on the wire. Insecure; most operators disable it.

The VyOS configuration:

set protocols ospf area 0 authentication plaintext-key PLAINTEXTKEY
commit
save

The operator typically does not use this in production because the key is visible to anyone with a packet capture.

MD5 authentication

MD5 with a key-id is the legacy cryptographic authentication. The configuration:

set protocols ospf area 0 authentication md5 key-id 1 key SECRET md5pass
commit
save

The key-id allows multiple keys to be configured simultaneously; the router picks the active key by ID. The receiving router verifies the MD5 digest using the same key-id and key.

The MD5 hash is computed over the OSPF packet body (excluding the authentication trailer). The result is placed in the authentication trailer along with the key-id.

SHA authentication

SHA-1, SHA-256, SHA-384, and SHA-512 are the cryptographic authentication types defined in RFC 5709. SHA-256 is the production standard.

The configuration:

# SHA-1
set protocols ospf area 0 authentication sha1 key-id 1 key SECRETSHA1
# Or specify the hash directly
set protocols ospf area 0 authentication sha1 key-id 1 key SECRET sha1pass

# SHA-256
set protocols ospf area 0 authentication sha256 key-id 1 key SECRETSHA256

# SHA-512
set protocols ospf area 0 authentication sha512 key-id 1 key SECRETSHA512
commit
save

The key-id is the same as for MD5: it allows multiple keys to be configured simultaneously.

Key-id mismatch — the dominant failure cause

The dominant OSPF authentication failure is key-id mismatch. The sending router sends the OSPF packet with one key-id; the receiving router checks the key-id against its local configuration; if the key-ids do not match (or the key-id is not configured locally), the authentication fails.

sequenceDiagram
  participant R1 as R1 (key-id 1, key SECRET)
  participant R2 as R2 (key-id 2, key SECRET)
  R1->>R2: OSPF Hello<br/>AuType=MD5, KeyID=1, Digest=...
  Note over R2: KeyID 1 not configured locally<br/>Authentication fails<br/>Hello dropped
  R2->>R1: OSPF Hello<br/>AuType=MD5, KeyID=2, Digest=...
  Note over R1: KeyID 2 not configured locally<br/>Authentication fails<br/>Hello dropped
  Note over R1,R2: Adjacency fails

The diagnostic:

show ip ospf neighbor
# (empty)

debug ospf packet hello
# log shows: "Authentication failed for packet from 10.0.0.2"
# log shows: "Key ID mismatch"

# Check both sides
show configuration commands | match "authentication"
# R1: key-id 1
# R2: key-id 2

The fix is to align the key-id on both sides:

# On R2
delete protocols ospf area 0 authentication md5 key-id 2
set protocols ospf area 0 authentication md5 key-id 1 key SECRET md5pass
commit
save

Key chains for key rotation

Key chains allow the operator to configure multiple keys with acceptance and send lifetimes. The router uses the active key for sending; it accepts any key whose acceptance lifetime includes the current time.

The VyOS configuration:

set protocols ospf area 0 authentication key-chain KEYCHAIN
set protocols ospf area 0 authentication key-chain KEYCHAIN key 1
set protocols ospf area 0 authentication key-chain KEYCHAIN key 1 key SECRET
set protocols ospf area 0 authentication key-chain KEYCHAIN key 1 algorithm sha256

set protocols ospf area 0 authentication key-chain KEYCHAIN key 1 lifetime accept-start 2026-01-01T00:00:00
set protocols ospf area 0 authentication key-chain KEYCHAIN key 1 lifetime accept-end 2026-12-31T23:59:59
set protocols ospf area 0 authentication key-chain KEYCHAIN key 1 lifetime send-start 2026-01-01T00:00:00
set protocols ospf area 0 authentication key-chain KEYCHAIN key 1 lifetime send-end 2026-12-31T23:59:59

# Key 2 with overlapping lifetime for rotation
set protocols ospf area 0 authentication key-chain KEYCHAIN key 2
set protocols ospf area 0 authentication key-chain KEYCHAIN key 2 key NEW_SECRET
set protocols ospf area 0 authentication key-chain KEYCHAIN key 2 algorithm sha256

set protocols ospf area 0 authentication key-chain KEYCHAIN key 2 lifetime accept-start 2026-12-01T00:00:00
set protocols ospf area 0 authentication key-chain KEYCHAIN key 2 lifetime accept-end 2027-12-31T23:59:59
set protocols ospf area 0 authentication key-chain KEYCHAIN key 2 lifetime send-start 2026-12-01T00:00:00
set protocols ospf area 0 authentication key-chain KEYCHAIN key 2 lifetime send-end 2027-12-31T23:59:59
commit
save

The key chain has two keys with overlapping lifetimes. From 2026-01-01 to 2026-12-01, key 1 is active for both send and accept. From 2026-12-01 to 2026-12-31, key 1 is still accepted but key 2 is sent. From 2027-01-01 onwards, only key 2 is used.

gantt
    title Key Chain Lifetime
    dateFormat  YYYY-MM-DD
    section Key 1
    Send active        :a1, 2026-01-01, 2026-12-01
    Accept active      :a2, 2026-01-01, 2026-12-31
    section Key 2
    Send active        :b1, 2026-12-01, 2027-12-31
    Accept active      :b2, 2026-12-01, 2027-12-31

The overlapping window (2026-12-01 to 2026-12-31) allows the operator to deploy key 2 on one router while key 1 is still active on the other. The adjacency continues to form during the overlap because both keys are accepted.

Per-interface authentication

Authentication can be configured per-area (under area 0 authentication) or per-interface (under interface eth0 authentication). The per-interface configuration overrides the per-area configuration for that interface.

The configuration:

# Per-area (default for all interfaces in the area)
set protocols ospf area 0 authentication md5 key-id 1 key SECRET md5pass

# Per-interface (overrides the per-area)
set protocols ospf interface eth0 authentication md5 key-id 2 key DIFFERENT_SECRET md5pass
commit
save

The per-interface authentication is useful when the area has multiple interfaces with different security postures (e.g. one internal, one external).

Diagnostic sequence

The diagnostic sequence for an authentication failure:

1. Confirm the authentication type. show configuration commands | match "authentication" shows the configured authentication.
2. Compare both sides. Compare the authentication type, key-id, algorithm, and key on both sides.
3. Check the key chain lifetimes. If using a key chain, confirm the current time is within the acceptance window.
4. Verify the time synchronisation. OSPF authentication uses the local time for the key chain lifetime check. If the clocks are out of sync, the keys may be evaluated differently on each side.
5. Enable debug. debug ospf packet hello shows the authentication failure log.
6. Apply the fix. Align the authentication configuration on both sides.

Production failure modes

• Key-id mismatch. The dominant cause. The fix is to align the key-id.
• Algorithm mismatch. R1 uses MD5, R2 uses SHA-256. The fix is to align the algorithm.
• Key mismatch. Same key-id and algorithm, but different keys. The fix is to align the keys.
• Key chain lifetime expired. The current time is outside the acceptance window. The fix is to extend the lifetime or to deploy a new key.
• Clock drift between routers. The key chain lifetime check uses the local time; clock drift causes inconsistent acceptance. The fix is to synchronise the clocks (NTP).
• Plaintext key visible in show configuration. The operator sees the key in plaintext in the configuration. The fix is to use a hashed key (md5pass).

Rollback

The rollback for an authentication change:

# Capture the running configuration
show configuration commands | save /tmp/ospf-auth-$(date +%s).txt

# Compare
compare

# Commit with a short confirm window
commit-confirm 5

# Rollback if needed
rollback 1
commit

For a key-id change:

# Revert to the previous key-id
delete protocols ospf area 0 authentication md5 key-id 2
set protocols ospf area 0 authentication md5 key-id 1 key SECRET md5pass
commit
save

Production discipline

Cross-course references

The OSPF configuration lesson vyos-xix-05-ospf-authentication covers the configuration primitives. The OSPF troubleshooting lessons vyos-xxii-01-neighbour-stuck (neighbour-stuck states), vyos-xxii-03-area-mismatch (area mismatch), and vyos-xxii-05-duplicate-router-id (duplicate router-id) cover related failure modes. The OSPFv3 authentication lesson vyos-xxi-02-ospfv3-config (OSPFv3 authentication via RFC 7166) covers the OSPFv3 equivalent. The IPv6 firewall lesson vyos-xi-06-ipv6-troubleshoot covers the IPv6 firewall interaction with OSPFv3 authentication.

Quiz