OSPF over VPN — the multicast problem, NBMA with static neighbours, and area design

OSPF discovers its neighbours by multicast. A tunnel interface on Linux is not a multicast-capable device. That single sentence is the whole of this lesson’s difficulty, and almost every piece of advice you will find about OSPF over VPN is an attempt to work around it — some of it correct, a lot of it not.

This lesson establishes what is actually true on VyOS 1.5: why the tunnel is not multicast-capable, what changing the OSPF network type does and does not change, the configuration that works without depending on multicast at all, and where the honest limits of that answer are.

Why the tunnel is not multicast-capable

A Linux network device advertises its capabilities in its flags. IFF_MULTICAST is the one that says “this device can carry multicast”. A WireGuard interface is registered without it:

/* drivers/net/wireguard/device.c */
dev->flags = IFF_POINTOPOINT | IFF_NOARP;

An IPsec VTI on VyOS 1.5 is no better. VyOS creates vtiN as an xfrm interface (ip link add vtiN type xfrm if_id N), and that driver sets:

/* net/xfrm/xfrm_interface_core.c */
dev->flags = IFF_NOARP;

Neither driver ever sets IFF_MULTICAST. You can see it on the running router:

vyos@R1:~$ ip link show wg0
4: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 ...

No MULTICAST in that flag list. Contrast an Ethernet interface, which shows BROADCAST,MULTICAST.

What the OSPF network type actually changes

OSPF has four network types, and VyOS 1.5 exposes exactly those four keywords under protocols ospf interface <name> network. The important question for a tunnel is not “which one is right for a point-to-point link” but “which one stops OSPF addressing its hellos to 224.0.0.5”:

Read the second row again, because it is the claim this lesson exists to correct. Point-to-point does not make OSPF unicast its hellos. It changes the adjacency model — no designated router, a different LSA representation of the link — and it leaves the destination address exactly where it was. In FRR’s ospf_hello_send(), everything that is not NBMA (or point-to-multipoint explicitly placed in non-broadcast mode, which VyOS does not expose) falls into a single else branch that sends to OSPF_ALLSPFROUTERS.

non-broadcast is the only one of the four that iterates over configured neighbours and sends a separate unicast hello to each. That is what makes it the network type for a tunnel.

Configure OSPF over a tunnel on VyOS 1.5

Two things about the configuration surface, both of which older material gets wrong.

OSPF interface settings live under protocols ospf interface <name>. There is no ip ospf node under interfaces on 1.5 — the interfaces <type> <name> ip subtree holds adjust-mss, ARP tuning and source-validation, and nothing routing-protocol shaped.

Static neighbours live under protocols ospf neighbor <ipv4>, at the top level of the OSPF tree, not under the interface. This is the node NBMA needs.

configure

# R1: the tunnel. See Part XLI for the WireGuard detail.
run generate pki wireguard key-pair install interface wg0
set interfaces wireguard wg0 address '10.10.10.1/30'
set interfaces wireguard wg0 port '51820'
set interfaces wireguard wg0 peer SITE-B public-key 'cVn4T2sM8xQ6yB1hJ0dR7kL3pW9zA5eG2uY8iO4nX1c='
set interfaces wireguard wg0 peer SITE-B address '203.0.113.2'
set interfaces wireguard wg0 peer SITE-B port '51820'
set interfaces wireguard wg0 peer SITE-B allowed-ips '10.10.10.0/30'
set interfaces wireguard wg0 peer SITE-B allowed-ips '10.20.0.0/16'

# R1: OSPF on the tunnel, with no reliance on multicast
set protocols ospf interface wg0 area '0'
set protocols ospf interface wg0 network 'non-broadcast'
set protocols ospf interface wg0 cost '10'
set protocols ospf interface wg0 hello-interval '10'
set protocols ospf interface wg0 dead-interval '40'

# R1: the far end, as a static neighbour on the tunnel subnet
set protocols ospf neighbor '10.10.10.2'

compare
commit
save

R2 is the mirror image. Note that allowed-ips differs on each side because it describes what lives behind the other router, and that the static neighbour is the far end’s tunnel address:

configure

run generate pki wireguard key-pair install interface wg0
set interfaces wireguard wg0 address '10.10.10.2/30'
set interfaces wireguard wg0 port '51820'
set interfaces wireguard wg0 peer SITE-A public-key 'rSuuvkAuSoyqvogoA5OJjYBbJKKaYS13pWs4K69PTOE='
set interfaces wireguard wg0 peer SITE-A address '198.51.100.1'
set interfaces wireguard wg0 peer SITE-A port '51820'
set interfaces wireguard wg0 peer SITE-A allowed-ips '10.10.10.0/30'
set interfaces wireguard wg0 peer SITE-A allowed-ips '10.0.0.0/24'
set interfaces wireguard wg0 peer SITE-A allowed-ips '192.168.1.0/24'

set protocols ospf interface wg0 area '0'
set protocols ospf interface wg0 network 'non-broadcast'
set protocols ospf interface wg0 cost '10'

set protocols ospf neighbor '10.10.10.1'

compare
commit
save

Three things this configuration depends on, all of which are easy to get wrong:

1. The tunnel /30 must be inside allowed-ips on both ends. OSPF’s unicast hellos are ordinary packets to 10.10.10.x and are subject to cryptokey routing like everything else.
2. hello-interval and dead-interval must match on both routers. OSPF refuses an adjacency where they differ, and NBMA gives you no better error than any other network type.
3. NBMA elects a designated router, so priority matters. It defaults to 1 on both ends here, which is fine for two routers; on a hub-and-spoke design set the spokes to priority 0 so the hub always wins.

Demand-circuit is not available on VyOS 1.5

OSPF demand circuits (RFC 1793) suppress the periodic hellos and LSA refreshes on a link once the adjacency is up, which is exactly what you would want on a metered or very slow tunnel.

VyOS 1.5 does not expose it. There is no demand-circuit node anywhere in the OSPF interface definition; the interface leaves are area, authentication, bandwidth, cost, dead-interval, hello-interval, hello-multiplier, mtu-ignore, network, passive, priority, retransmit-interval and transmit-delay. Configuring it is not a thing you can do, correctly or otherwise.

That leaves the real problem — OSPF chatter on a constrained link — to be addressed with what the release does have:

# Lengthen the hello cycle. Both ends must agree, and dead-interval
# must be raised with it.
set protocols ospf interface wg0 hello-interval '30'
set protocols ospf interface wg0 dead-interval '120'

# Keep the LAN in OSPF without running the protocol on it
set protocols ospf interface eth1 area '1'
set protocols ospf interface eth1 passive

# Cut the LSA volume a spoke has to carry, rather than the hello rate
set protocols ospf area 1 area-type stub no-summary

Be honest with yourself about the arithmetic before spending effort here. An OSPF hello is roughly 60 to 100 bytes on the wire. At the default 10-second interval that is about 8 bytes per second per adjacency — around 0.007% of a 1 Mbps link. Hellos are not what makes OSPF expensive on a slow link; LSA flooding during a topology change is, and the lever for that is the area design, not the hello timer.

Area design for hub-and-spoke

flowchart LR
  HUB["Hub<br/>ABR, backbone in Area 0"]
  S1["Spoke A<br/>Area 1"]
  S2["Spoke B<br/>Area 2"]
  S3["Spoke C<br/>Area 3"]
  HUB -- "wg1, Area 1" --> S1
  HUB -- "wg2, Area 2" --> S2
  HUB -- "wg3, Area 3" --> S3

The design choice is how much topology each spoke has to carry:

• Everything in Area 0. Simplest to configure, and every spoke holds the full link-state database. Every flap anywhere is flooded everywhere. Fine for a handful of sites.
• One area per spoke, hub as ABR. The hub summarises. A flap inside a spoke does not reach the others as a router LSA. This is the usual production shape.
• Stub or NSSA spokes. area-type stub no-summary gives the spoke a default route and nothing else, which is what a branch office actually needs. NSSA where the spoke has external routes of its own to inject.

The hub must be in Area 0 — it is the backbone, and every other area attaches to it through the hub acting as ABR. A design where the spokes are in Area 0 and the hub is not does not have a backbone at all.

Give each spoke its own WireGuard interface rather than several peers on one. That is a WireGuard constraint, not an OSPF one: peers on one interface must have disjoint allowed-ips, and per-spoke interfaces also give you a per-spoke OSPF cost and a per-spoke area.

Validation

# Is there an adjacency, and in what state
show ip ospf neighbor

# What the router believes about the tunnel interface: network type,
# cost, timers, neighbour count
show ip ospf interface wg0

# The static neighbours NBMA is using
show configuration commands | match 'protocols ospf neighbor'

# What was learned, and whether it was installed
show ip ospf database
show ip route ospf

# Is anything OSPF-shaped crossing the tunnel at all, and where is it
# addressed
monitor traffic interface wg0 filter 'proto ospf'

# The tunnel underneath it
show interfaces wireguard wg0 summary

ping 10.20.0.5

A clean validation has four parts, and the order matters because each one rules out the layer below:

1. show interfaces wireguard wg0 summary shows a recent latest handshake — the tunnel carries traffic.
2. ping 10.10.10.2 succeeds — the tunnel subnet is routable and inside allowed-ips.
3. show ip ospf interface wg0 reports Network Type NBMA and a neighbour count of 1.
4. show ip ospf neighbor shows the peer in Full/DR or Full/BDR, and show ip route ospf has the far side’s prefixes.

The capture is the tie-breaker when those disagree. Hellos addressed to 10.10.10.2 mean NBMA is in effect; hellos addressed to 224.0.0.5 mean the network type is not what you think it is, whatever the configuration says.

Production failure modes

Network type left at broadcast or set to point-to-point

The commonest failure and the one this lesson exists for. OSPF addresses its hellos to 224.0.0.5, the tunnel is not a multicast device, and no adjacency forms. Setting point-to-point does not change the destination address and therefore does not help.

Diagnostic: show ip ospf interface wg0 reports Network Type BROADCAST or POINTOPOINT; show ip ospf neighbor is empty; a capture on wg0 shows hellos addressed to 224.0.0.5, or shows nothing leaving at all.

Fix: set protocols ospf interface wg0 network non-broadcast on both routers, plus a protocols ospf neighbor entry on each pointing at the other’s tunnel address.

Network type set to NBMA with no neighbours configured

NBMA does not discover anything. With no protocols ospf neighbor entry there is nobody to send a hello to, so the interface sits silent and looks identical to the previous failure.

Diagnostic: show ip ospf interface wg0 reports Network Type NBMA with a neighbour count of 0, and the capture on wg0 shows no OSPF packets at all — not even misaddressed ones.

Fix: add the far end’s tunnel address under protocols ospf neighbor. Remember it is a top-level OSPF node, not a child of protocols ospf interface.

The tunnel subnet is not in allowed-ips

The unicast hellos are generated and handed to wg0, and WireGuard discards them because no peer claims 10.10.10.2. Everything above looks correctly configured.

Diagnostic: ping 10.10.10.2 fails while show interfaces wireguard wg0 summary shows a healthy handshake. That combination is nearly diagnostic on its own.

Fix: add the tunnel /30 to that peer’s allowed-ips on both ends.

Hello or dead interval mismatch

R1 has hello-interval 10, R2 has 30. OSPF compares both values in the hello and refuses the adjacency.

Diagnostic: show ip ospf neighbor is empty, but the capture shows hellos arriving in both directions — the packets are getting through and being rejected, which distinguishes this from every failure above.

Fix: align both timers on both routers. Raising one without the other is the usual cause.

Two spokes share one WireGuard interface

Two peers on wg0 are given overlapping allowed-ips. The later assignment takes the prefix and the earlier peer silently stops receiving traffic for it, taking its OSPF adjacency with it. Nothing in the configuration looks wrong.

Diagnostic: show interfaces wireguard wg0 allowed-ips shows what the kernel actually holds, which is not what the configuration reads.

Fix: one interface per spoke, each on its own UDP port.

OSPF flaps with the tunnel

The tunnel drops and recovers; the adjacency follows it, and every cycle floods LSAs and reconverges the whole area.

Fix: find and fix the tunnel instability first — this is not an OSPF problem wearing an OSPF costume. Then limit the blast radius: a stub area for the spoke means a flap floods a much smaller database, and a static route with a high administrative distance gives the traffic somewhere to go while OSPF reconverges.

Rollback

# Enter configuration mode and write the running configuration to a
# file you can load back. `save` is a configuration-mode command that
# takes a path; operational mode has no `| save` pipe.
configure
save /config/pre-change-ospf-vpn-TICKET.conf

# Remove the OSPF configuration from the tunnel
delete protocols ospf interface wg0
delete protocols ospf neighbor 10.10.10.2

# Read the diff before committing anything
compare
commit-confirm 5
confirm

# Or restore a previous configuration
load /config/pre-change-ospf-vpn-TICKET.conf
commit
save

Removing OSPF from the tunnel withdraws every route it was providing. If this session arrives over one of those routes, commit-confirm is not optional.

Production discipline

Cross-course references

• Part XLIII-01 (XLIII-VyOS-VPNRouting / basics) covers the routing-over-tunnel fundamentals, including the recursive-routing trap.
• Part XLIII-02 (XLIII-VyOS-VPNRouting / BGP over VPN) is the alternative this lesson keeps pointing at.
• Part XLI-01 (XLI-VyOS-WireGuard / concept) covers the netdev flags and the unicast-only consequence.
• Part XLI-03 (XLI-VyOS-WireGuard / peers) covers allowed-ips and why peers on one interface must be disjoint.
• Part XVIII-04 (XVIII-VyOS-OSPF / areas) covers the area types referenced here.
• Part XIX-02 (XIX-VyOS-OSPF / interface config) covers protocols ospf interface in general.

Quiz