Multi-site fabric — combining VPN, dynamic routing and addressing into one network

A multi-site fabric is the production answer to the multi-site VPN problem: instead of configuring static routes for every site’s LAN subnet on every other site, run a dynamic routing protocol over the VPN tunnels. The sites learn each other’s subnets dynamically; failures are detected and rerouted automatically; adding a new site means one configuration change, not N-1. The cost is the operational complexity of running a routing protocol — and the discipline of getting the addressing plan right so the protocol can do its job.

This lesson covers the patterns for combining VPN with dynamic routing — OSPF for hub-and-spoke and small fabrics, BGP for larger fabrics with policy requirements — the addressing plan that supports summarisation, the failover behaviour, and the verification discipline.

When to use dynamic routing over a VPN

The decision to run a dynamic routing protocol over a VPN is a trade-off:

• Static routes. Simple. Predictable. The operator writes every route. The cost is N-1 configuration changes when adding a site to a full mesh, or summarisation logic at the hub.
• Dynamic routing. The sites learn each other’s subnets. Adding a site is one change. The cost is the operational complexity of running a routing protocol.

For 2-3 sites with stable addressing, static routes are sufficient. For 4+ sites, or for sites with subnets that change, dynamic routing pays for itself. The OPNsense FRR plugin supports OSPF, BGP, RIP, and BFD.

The addressing plan for a multi-site fabric

The addressing plan is what makes or breaks a dynamic routing deployment. Three rules:

1. Contiguous subnets for summarisation. Sites in the same region should have contiguous subnets so the routing protocol can summarise them into one advertisement. Branch sites on 10.1.0.0/24, 10.1.1.0/24, 10.1.2.0/24 can be summarised as 10.1.0.0/16 at the hub.
2. Tunnel subnets distinct from site subnets. Tunnel subnets (10.99.0.0/24) carry the routing protocol’s next-hop addressing; site subnets (10.0.0.0/16) carry the LAN traffic. The two must not overlap.
3. Predictable summary boundaries. A /16 summary at the hub means every site within the /16 is reachable through the hub. A /20 summary is finer-grained but more routes; a /8 summary is too coarse. The discipline: pick the summary that matches the failover behaviour the operator wants.

# Example addressing plan for a 10-site fabric with summarisation.
# Each site has a /24 LAN subnet.
# Sites are grouped into 2 regions of 5 sites each.
# Each region summarises to a /20 at the hub.

Region | Site | LAN subnet
-------|------|------------
North  | 1    | 10.1.0.0/24
North  | 2    | 10.1.1.0/24
North  | 3    | 10.1.2.0/24
North  | 4    | 10.1.3.0/24
North  | 5    | 10.1.4.0/24
South  | 6    | 10.2.0.0/24
South  | 7    | 10.2.1.0/24
South  | 8    | 10.2.2.0/24
South  | 9    | 10.2.3.0/24
South  | 10   | 10.2.4.0/24

# Hub summary for North: 10.1.0.0/20 (covers 10.1.0.0 - 10.1.15.255).
# Hub summary for South: 10.2.0.0/20.

OSPF over WireGuard

OSPF is the most common dynamic routing protocol for OPNsense multi-site fabrics. It runs over the WireGuard tunnel interfaces, discovers neighbours, builds a link-state database, and computes the shortest-path tree.

The configuration on each site:

# FRR configuration on a spoke site (ospfd.conf).

interface wg0
 ip ospf network point-to-point
 ip ospf cost 10
!
router ospf
 ospf router-id 10.99.0.2
 network 10.1.0.0/24 area 0
 network 10.99.0.0/24 area 0
!

The point-to-point network type is correct for WireGuard tunnels: each tunnel has exactly two endpoints, no DR/BDR election is needed. The cost is the OSPF metric; the operator tunes the cost to influence path selection.

BGP over WireGuard or IPsec

BGP is the right choice for multi-site fabrics with policy requirements. BGP is the protocol of the Internet; it scales to thousands of peers; it carries policy (local preference, MED, communities) that OSPF does not. For an OPNsense multi-site fabric, BGP is the choice when:

• The number of sites exceeds the practical limit for OSPF (around 50-100 for a single area; less with summarisation).
• The operator needs policy-based routing (send traffic from site A to site C via site B; route certain prefixes via different paths).
• The fabric connects to an upstream provider (the BGP sessions to the provider share the same operational discipline as the fabric’s internal BGP).

# FRR configuration on a spoke site (bgpd.conf).

router bgp 65010
 bgp router-id 10.99.0.2
 neighbor 10.99.0.1 remote-as 65000
 neighbor 10.99.0.1 soft-reconfiguration inbound
 !
 address-family ipv4 unicast
  network 10.1.0.0/24
  network 10.99.0.0/24
 exit-address-family
!

The hub runs BGP AS 65000 (a private ASN for the fabric); the spokes run unique ASNs from the 64512-65534 private range. Each spoke peers with the hub over the WireGuard tunnel; the hub reflects routes between spokes.

Failover behaviour

The dynamic routing protocol’s failover behaviour is what justifies the operational cost. With static routes, the operator configures a primary path; with dynamic routing, the protocol detects failures and reroutes.

The timeline:

1. Tunnel drop. The WireGuard or IPsec tunnel drops. The interface goes down.
2. Routing protocol detects. OSPF detects the interface down within its dead-interval (default 40 seconds). BGP detects within its hold-time (default 180 seconds). BFD detects within its configured interval (typically 50-200 milliseconds).
3. Route withdraw or alternative path. The routing protocol removes the affected routes from the routing table. If an alternative path exists, it is installed; otherwise the destination becomes unreachable.
4. Traffic reroutes. The kernel uses the new route; traffic follows the alternative path.

# Verify failover behaviour: check OSPF neighbour state and routing table.
vtysh
show ip ospf neighbor
show ip route

The discipline: tune the timers for the deployment. A fabric where failover must be sub-second uses BFD. A fabric where 30-second failover is acceptable uses OSPF’s default timers and skips BFD. A fabric where BGP is the protocol uses BGP’s timers and possibly BFD for sub-second detection.

Asymmetric routing and state

A multi-site fabric with dynamic routing can produce asymmetric routing — packets from site A to site B take one path, packets from site B to site A take a different path. For stateful firewalls, asymmetric routing is a state-table problem: the firewall on the longer path does not see the return packet.

The mitigations:

• Symmetric routing at the design level. Design the addressing and metric so the forward and reverse paths are the same. With OSPF and equal-cost paths, traffic may take different paths in different directions; the operator tunes metrics to force symmetry.
• Stateful failover with pfsync. In an HA deployment, pfsync synchronises state between firewalls; an asymmetric path that crosses a pair of HA firewalls is still handled correctly.
• Stateless routing on the fabric. A common production choice: the fabric’s interior is stateless (firewalls do not maintain state on fabric traffic); stateful inspection happens only at the ingress and egress. The discipline: do not run stateful firewalls on every fabric hop; run them only at the boundary.

# Confirm state is created for fabric traffic (or confirm it is not, deliberately).
pfctl -s state | grep -E 'wg0|ipsec0' | head -5

Verification

After deploying the fabric, verify:

1. From every site, traceroute to every other site’s LAN subnet — must traverse the fabric (or the hub).
2. From the hub, vtysh show ip ospf neighbor (or BGP equivalent) — every spoke must be a neighbour in the FULL state.
3. From any site, vtysh show ip route — must show the routes to every other site’s LAN subnet via the fabric.
4. Simulate a tunnel drop (down the WireGuard interface on one spoke) and confirm failover — must reroute within the configured timers.
5. From a packet capture on the tunnel, confirm the routing protocol’s packets are present (OSPF: ip proto 89; BGP: TCP 179).

A fabric that passes 1-2 but fails 3-5 has a neighbour adjacency but no routing. A fabric that passes 3-4 but fails 1 has routes but no path. The operator checks every link in the chain.