Reference topology — a complete dual-DC, dual-router VyOS production estate

A reference topology is not a marketing diagram. It is a working configuration that the operator can clone, deploy, and adapt to a real-world estate. The RunBook Academy dual-DC, dual-router topology is the canonical reference for a production VyOS 1.5 LTS estate: two sites, two routers per site, two ISPs, BGP multihoming, OSPF underlay, VRF segmentation, WireGuard and IPsec VPN, VRRP for first-hop redundancy, and the monitoring, backup, and DR layers that turn the topology into a production system.

This lesson is the first in Part LVII: the topology, the layers, the failure domains, and the validation discipline that makes the reference a usable starting point.

The high-level topology

flowchart TB
  subgraph DC1["Site 1 (DC-East)"]
    R1A["R1-A edge / 10.0.1.1"]
    R1B["R1-B edge / 10.0.1.2"]
    SW1["TOR switch"]
    SRV1["App servers"]
  end
  subgraph DC2["Site 2 (DC-West)"]
    R2A["R2-A edge / 10.0.2.1"]
    R2B["R2-B edge / 10.0.2.2"]
    SW2["TOR switch"]
    SRV2["App servers"]
  end
  subgraph NET["Network"]
    ISP1["ISP-A primary"]
    ISP2["ISP-B secondary"]
    PEER["Internet"]
  end
  R1A -.->|ISP-A| ISP1
  R1B -.->|ISP-B| ISP2
  R2A -.->|ISP-A| ISP1
  R2B -.->|ISP-B| ISP2
  R1A <-->|iBGP OSPF intra-DC| R1B
  R2A <-->|iBGP OSPF intra-DC| R2B
  R1A <-->|WireGuard IPsec inter-DC| R2A
  R1B <-->|WireGuard IPsec inter-DC| R2B
  R1A --> SW1
  R1B --> SW1
  R2A --> SW2
  R2B --> SW2
  SW1 --> SRV1
  SW2 --> SRV2
  ISP1 --> PEER
  ISP2 --> PEER

The topology has two sites (DC-East and DC-West), each with two edge routers (R1-A and R1-B at DC-East; R2-A and R2-B at DC-West). Each router has two ISP connections (one to ISP-A, one to ISP-B). The intra-DC routers are connected via iBGP and OSPF; the inter-DC routers are connected via WireGuard and IPsec VPN.

The Top-of-Rack (TOR) switch connects the routers to the application servers. The application servers are the canonical clients: they reach the Internet via the local site, the other site via the inter-DC VPN, and the local network via the local router.

The layers

The topology is organised into seven layers. Each layer has a specific purpose, a specific failure mode, and a specific validation discipline.

flowchart TD
  L1["1. Underlay<br/>(physical, link, IP)"]
  L2["2. IGP<br/>(OSPF)"]
  L3["3. BGP<br/>(eBGP to ISPs, iBGP inside)"]
  L4["4. VPN<br/>(WireGuard, IPsec)"]
  L5["5. Services<br/>(VRF, VRRP, NAT, DHCP)"]
  L6["6. Monitoring<br/>(SNMP, exporters, gNMI)"]
  L7["7. Backup & DR<br/>(config, image, offsite)"]
  L1 --> L2
  L2 --> L3
  L3 --> L4
  L4 --> L5
  L5 --> L6
  L6 --> L7

Layer 1 — Underlay. The physical layer: the routers, the switches, the ISP connections, the cross-connects. The canonical failure modes: link flap, SFP failure, switch port failure. The validation discipline: show interfaces, show ip route, monitor interfaces.

Layer 2 — IGP (OSPF). The interior gateway protocol: OSPF between the two intra-DC routers and the inter-DC VPN tunnels. The canonical failure modes: neighbour stuck, area mismatch, SPF storm. The validation discipline: show ip ospf neighbor, show ip ospf json, show ip ospf database.

Layer 3 — BGP (eBGP to ISPs, iBGP inside). The exterior gateway protocol: eBGP to the two ISPs, iBGP between the intra-DC routers. The canonical failure modes: session flap, route leak, prefix filtering. The validation discipline: show ip bgp summary, show ip bgp summary json, show ip bgp neighbor.

Layer 4 — VPN (WireGuard, IPsec). The overlay VPN: WireGuard for the inter-DC tunnels, IPsec for the site-to-site VPN. The canonical failure modes: tunnel flap, key expiry, MTU mismatch. The validation discipline: show interfaces wireguard wg0 summary, show vpn ipsec sa, show interfaces.

Both VPN commands are worth naming exactly, because the obvious shorter forms do not exist on 1.5. show wireguard was a 1.3 command and its op-mode definition is gone from the current tree; the peer view now lives behind the summary subcommand of a named interface, and the bare show interfaces wireguard wg0 gives you the link and its aggregate counters with no peer state in it. Likewise the IPsec command is show vpn ipsec sa, not show ipsec sa.

Layer 5 — Services (VRF, VRRP, NAT, DHCP). The service layer: VRF segmentation, VRRP for first-hop redundancy, NAT for Internet access, DHCP for the application servers. The canonical failure modes: VRRP mismatch, NAT port exhaustion, DHCP lease exhaustion. The validation discipline: show vrrp, show nat source translations, show dhcp server leases. The NAT and DHCP trees have no bare show nat or show dhcp leaf on 1.5 — each needs the direction or the role that follows it.

Layer 6 — Monitoring (SNMP, exporters, gNMI). The observability layer: SNMP for the legacy NMS, exporters for Prometheus, gNMI for streaming state. The canonical failure modes: exporter crash, SNMP timeout, gNMI subscription overrun. The validation discipline: show service snmp, curl http://<router>:9100/metrics, gnmic subscribe.

Layer 7 — Backup & DR (config, image, offsite). The disaster recovery layer: configuration backup, image backup, offsite replication, periodic DR drills. The canonical failure modes: configuration drift, image corruption, offsite disconnect. The validation discipline: show configuration, show system image, diff running saved.

The failure domains

The topology is designed to tolerate four failure domains:

flowchart LR
  subgraph FAILS["Failure domains"]
    F1["Site failure<br/>(DC-East or DC-West)"]
    F2["Router failure<br/>(R1-A or R1-B)"]
    F3["Link failure<br/>(single ISP link)"]
    F4["ISP failure<br/>(ISP-A or ISP-B)"]
  end
  subgraph TOL["Tolerance"]
    T1["Inter-DC VPN failover"]
    T2["iBGP failover + VRRP"]
    T3["BGP path failover"]
    T4["BGP path failover"]
  end
  F1 --> T1
  F2 --> T2
  F3 --> T3
  F4 --> T4

Site failure (DC-East or DC-West). A site failure is the most severe failure mode: the entire DC is down. The tolerance mechanism is the inter-DC VPN. The application servers at DC-East fail over to DC-West via the inter-DC VPN; the BGP advertisement changes to prefer the DC-West path. The failover time is 30 seconds (BGP convergence) plus the application timeout (typically 30-60 seconds). The operator must design the application to tolerate this failover time.

Router failure (R1-A or R1-B). A router failure is the most common failure mode. The tolerance mechanism is iBGP failover (the surviving router takes over the BGP sessions) and VRRP failover (the application servers fail over to the surviving router). The failover time is 3 seconds (VRRP) plus the BGP convergence (typically 30 seconds). The operator must configure the application servers to use the VRRP virtual IP, not the physical router IP.

Link failure (single ISP link). A link failure is the most common operational failure mode. The tolerance mechanism is BGP path failover (the router fails over to the surviving ISP via the other router). The failover time is 30 seconds (BGP convergence). The operator must configure BFD on the eBGP sessions to detect the failure faster (sub-second).

ISP failure (ISP-A or ISP-B). An ISP failure is the most severe external failure mode. The tolerance mechanism is BGP path failover (the router fails over to the surviving ISP via the other router). The failover time is 30 seconds (BGP convergence). The operator must configure multiple-AS eBGP to allow the router to peer with both ISPs simultaneously.

The validation discipline

Every change to the topology must be validated end-to-end before production deployment. The validation discipline:

1. Confirm the underlay is up. The operator runs show interfaces on every router and confirms every interface is up and the counters are incrementing.
2. Confirm the IGP is converged. The operator runs show ip ospf neighbor on every router and confirms every neighbour is in Full state.
3. Confirm the BGP is converged. The operator runs show ip bgp summary on every router and confirms every peer is in Established state.
4. Confirm the VPN is up. The operator runs show interfaces wireguard wg0 summary and show vpn ipsec sa on every router and confirms the tunnels are up. For WireGuard, “up” means the peer prints a recent latest handshake line and a transfer line with counts in both directions — the interface itself is up from the moment it is committed, whether or not anything at the far end has ever answered.
5. Confirm the services are up. The operator runs show vrrp and confirms the master/backup state is correct, and runs show nat source rules and confirms the NAT rules are present.
6. Confirm the monitoring is up. The operator curls the exporter’s /metrics endpoint and confirms the metrics are present.
7. Confirm the backup is current. The operator runs show configuration and confirms the active configuration matches the saved configuration, and the saved configuration has been replicated to the offsite backup.

Production failure modes

The top-level failure modes the operator encounters:

• Site failure. The entire DC is down. The failover to the surviving DC takes 30-60 seconds. Fix: ensure the application is designed to tolerate the failover time; ensure the inter-DC VPN has sufficient bandwidth for the failover traffic.
• Router failure. A single router is down. The failover to the surviving router takes 3-30 seconds. Fix: ensure the VRRP preemption is configured; ensure the application servers use the VRRP virtual IP.
• ISP failure. A single ISP is down. The failover to the surviving ISP takes 30 seconds. Fix: ensure BFD is configured on the eBGP sessions; ensure the BGP MED and local-pref are configured to prefer the primary ISP.
• VPN failure. The inter-DC VPN is down. The application servers at the affected DC fail over to the other DC. Fix: ensure the VPN has multiple paths (WireGuard + IPsec); ensure the VPN is monitored (alerts on tunnel down).
• Configuration drift. The active configuration does not match the saved configuration. Fix: configure the operator’s automation to detect and alert on configuration drift.
• Image drift. The router is running a different image than the rest of the fleet. Fix: configure the operator’s automation to detect and alert on image drift.

Rollback

Every change to the topology must have a tested rollback. The rollback discipline:

• Configuration changes — rollback N and commit. The operator must test the rollback in a lab before deploying to production.
• Image changes — the router boots into the new image; the operator must keep the old image available (show system image) and configure the operator’s automation to boot into the old image on a configuration failure.
• VPN changes — the VPN tunnel is down; the operator must have a backup path (a second VPN tunnel or a direct connection).
• BGP changes — the BGP session is down; the operator must have a backup path (a second ISP or a second router).

For every change, use commit-confirm:

configure
# ... make the change ...
commit-confirm 5
# If the change has unintended consequences, the auto-rollback
# fires after 5 minutes and the previous configuration is restored.

Production discipline

Bandwidth and capacity planning

The reference topology assumes a 10 Gbps data plane per site and a 1 Gbps inter-DC VPN. The operator must validate these assumptions against the estate’s actual bandwidth requirements.

flowchart LR
  subgraph BW["Bandwidth assumptions"]
    INT["Internal traffic<br/>(East-West)"]
    EXT["External traffic<br/>(North-South)"]
    DC["Inter-DC traffic<br/>(DC-East to DC-West)"]
  end
  subgraph CAP["Capacity"]
    R["10 Gbps router<br/>eth0, eth1, eth2"]
    V["1 Gbps VPN<br/>WireGuard"]
    I["100 Gbps internal<br/>switch fabric"]
  end
  INT --> R
  EXT --> R
  DC --> V
  R --> I

The canonical sizing for a medium estate: 10 Gbps per router, 1 Gbps per inter-DC VPN, 100 Gbps internal switch fabric. The operator must measure the actual bandwidth requirements before sizing the routers; the operator who deploys a 1 Gbps router for a 10 Gbps estate will discover the gap during the next incident.

The discipline: measure the actual bandwidth requirements (peak traffic, average traffic, burst traffic); size the routers and the inter-DC VPN for the peak traffic plus 50% headroom; monitor the bandwidth usage (the SNMP IF-MIB counters, the Prometheus exporter); alert on bandwidth usage > 80%.

Cross-course references

• Part XL (vyos-xl-01-vrrp-concept) covers the VRRP that the topology uses for first-hop redundancy.
• Part XXIV (vyos-xxiv-01-bgp-config) covers the BGP that the topology uses for ISP multihoming.
• Part XIX (vyos-xix-01-ospf-basics) covers the OSPF that the topology uses for the underlay.
• Part XLI (vyos-xli-01-wireguard-concept) covers the WireGuard that the topology uses for the inter-DC VPN.
• Part XLIX (vyos-xlix-01-interface-metrics) covers the monitoring layer that the topology uses for observability.
• Part LV (vyos-lv-01-saved-configuration) covers the backup layer that the topology uses for disaster recovery.

Quiz