Reference rationale — design decisions, trade-offs, and why the topology is shaped this way

A reference topology without a rationale is a marketing diagram. The operator who clones the topology without understanding the rationale will deploy a topology that does not fit the estate, will fail to recognise the trade-offs, and will not be able to adapt the topology to a changing requirement. The RunBook Academy dual-DC, dual-router topology is the canonical reference, but the rationale is the part that makes it a usable starting point.

This lesson is the design rationale for the reference topology: why every major decision was made, what trade-offs were accepted, and how to adapt the topology to a smaller or larger estate while preserving the design intent.

The decision matrix

The reference topology is the result of seven major design decisions. Each decision has a rationale, a trade-off, and an alternative.

flowchart TD
  D1["Decision 1<br/>Dual-DC vs single-DC"]
  D2["Decision 2<br/>Dual-router vs single-router"]
  D3["Decision 3<br/>Dual-ISP vs single-ISP"]
  D4["Decision 4<br/>WireGuard + IPsec vs single VPN"]
  D5["Decision 5<br/>VRRP for first-hop"]
  D6["Decision 6<br/>OSPF underlay vs eBGP-only"]
  D7["Decision 7<br/>Prometheus + Grafana + Loki"]
  D1 --> D2
  D2 --> D3
  D3 --> D4
  D4 --> D5
  D5 --> D6
  D6 --> D7

Decision 1 — Dual-DC vs single-DC

Decision: Dual-DC (two sites, DC-East and DC-West).

Rationale: A site failure is the most severe operational failure. A fire, a flood, a power failure, or a network failure can take down an entire DC. The dual-DC design protects against a site failure: the application servers at DC-East fail over to DC-West via the inter-DC VPN, and the BGP advertisement changes to prefer the DC-West path.

Trade-off: The cost is the operational complexity of managing two sites, the bandwidth cost of the inter-DC VPN, and the application complexity of designing for the failover time (30-60 seconds).

Alternative: A single-DC topology is simpler but does not tolerate a site failure. The operator who deploys a single-DC topology will discover the gap during the next incident. For a small estate (a single office, a single lab), the single-DC topology is acceptable; for a production estate with multiple sites, the dual-DC topology is the canonical choice.

Decision 2 — Dual-router vs single-router

Decision: Dual-router (two routers per site, R1-A and R1-B at DC-East).

Rationale: A router failure is the most common operational failure. A kernel panic, a hardware failure, a configuration error, or a power supply failure can take down a single router. The dual-router design protects against a router failure: the application servers fail over to the surviving router via VRRP, and the iBGP session allows the surviving router to inherit the BGP sessions to the ISPs.

Trade-off: The cost is the operational complexity of managing two routers per site (iBGP, VRRP, dual-control plane), the hardware cost of the second router, and the configuration complexity (the two routers must have similar but distinct configurations).

Alternative: A single-router topology is simpler but does not tolerate a router failure. The operator who deploys a single-router topology will discover the gap during the next incident. For a lab or a non-production environment, the single-router topology is acceptable; for a production environment, the dual-router topology is the canonical choice.

Decision 3 — Dual-ISP vs single-ISP

Decision: Dual-ISP (each router has a connection to ISP-A and ISP-B).

Rationale: An ISP failure is the most severe external failure. A fiber cut, a router failure at the ISP, or a DDoS attack can take down a single ISP. The dual-ISP design protects against an ISP failure: the BGP session to the surviving ISP is preferred, and the application servers continue to reach the Internet.

Trade-off: The cost is the operational complexity of managing two BGP sessions per router, the cost of the second ISP connection, and the configuration complexity (BGP MED, local-pref, prefix filtering, max-prefix).

Alternative: A single-ISP topology is simpler and cheaper but does not tolerate an ISP failure. The operator who deploys a single-ISP topology will discover the gap during the next ISP incident. For a small estate, the single-ISP topology may be acceptable; for a production estate, the dual-ISP topology is the canonical choice.

flowchart LR
  subgraph DUAL["Dual-ISP"]
    R1["Router"] --> A["ISP-A"]
    R1 --> B["ISP-B"]
  end
  subgraph SINGLE["Single-ISP"]
    R2["Router"] --> A2["ISP-A"]
  end

Decision 4 — WireGuard + IPsec vs single VPN

Decision: WireGuard + IPsec (WireGuard for the inter-DC tunnels, IPsec for the site-to-site VPN).

Rationale: A single VPN technology is a single point of failure. If WireGuard has a kernel bug, an interoperability issue, or a configuration error, the inter-DC tunnel is down. The dual-VPN design protects against a single VPN failure: WireGuard handles the high-bandwidth inter-DC traffic, IPsec handles the low-bandwidth site-to-site traffic, and the two are independent.

Trade-off: The cost is the operational complexity of managing two VPN technologies, the configuration complexity (WireGuard keys, IPsec IKEv2 proposals, ESP proposals, MTU), and the monitoring complexity (the operator must monitor both tunnels).

Alternative: A single-VPN topology (either WireGuard only or IPsec only) is simpler but has a single point of failure. The operator who deploys a single-VPN topology will discover the gap during the next VPN incident. For a small estate, the single-VPN topology is acceptable; for a production estate, the dual-VPN topology is the canonical choice.

Decision 5 — VRRP for first-hop

Decision: VRRP for first-hop redundancy (R1-A and R1-B share a virtual IP, the application servers use the virtual IP as their default gateway).

Rationale: The application servers must reach the Internet. The application servers must use a stable default gateway that survives a router failure. VRRP provides a stable virtual IP that moves between the two routers; the application servers do not need to be reconfigured when the master router changes.

Trade-off: The cost is the configuration complexity (priority, preemption, advertisement interval, tracked interfaces) and the monitoring complexity (the operator must monitor the VRRP state and the gratuitous ARP).

Alternative: A first-hop redundancy protocol like CARP or a load balancer like HAProxy is possible but does not fit the canonical VyOS reference. For a VyOS estate, VRRP is the canonical choice.

Decision 6 — OSPF underlay vs eBGP-only

Decision: OSPF underlay (OSPF between the two intra-DC routers and the inter-DC VPN tunnels).

Rationale: OSPF is the canonical IGP for a small to medium estate. It converges quickly, supports area design (backbone, stub, NSSA), and integrates with the FRRouting northbound interface. OSPF is the right choice when the IGP carries a small number of routes (the routes for the underlay, the VPN tunnels, and any redistribution from BGP).

Trade-off: The cost is the configuration complexity (OSPF areas, SPF tuning, redistribution) and the monitoring complexity (LSDB, SPF runs). For a small estate (a single site), OSPF is overkill; for a medium estate (multiple sites), OSPF is the right choice.

Alternative: eBGP-only is the canonical design for a large service-provider estate. The operator who deploys eBGP-only at a small to medium estate is over-engineering; the operator who deploys OSPF at a large service-provider estate is under-engineering.

Decision 7 — Prometheus + Grafana + Loki

Decision: Prometheus for metrics, Grafana for visualisation, Loki for logs (and optionally Tempo for traces).

Rationale: The observability stack must support metrics (Prometheus), logs (Loki), and traces (Tempo) for a production estate. The three are complementary: Prometheus records the metrics, Grafana visualises the metrics and the logs, Loki stores the logs and provides log-based queries, and Tempo records the traces.

Trade-off: The cost is the operational complexity of managing three (or four) systems, the storage cost of the time-series data, and the configuration complexity (Prometheus recording rules, Grafana dashboards, Loki label schema).

Alternative: A simpler stack (Prometheus + Grafana only) is acceptable for a small estate; a more comprehensive stack (Prometheus + Grafana + Loki + Tempo + Jaeger) is acceptable for a large estate.

Adapting the topology to a different estate

The reference topology is a starting point. The operator must adapt it to the specific estate (size, bandwidth, availability requirements). The adaptation is the application of the rationale to a different set of constraints.

flowchart TD
  A["Reference topology"]
  A --> B["Small estate (1 site, 1 router, 1 ISP)"]
  A --> C["Medium estate (1 site, 2 routers, 2 ISPs)"]
  A --> D["Large estate (2 sites, 4 routers, 2 ISPs)"]
  B --> B1["Remove dual-DC<br/>Remove dual-router<br/>Remove dual-ISP"]
  C --> C1["Remove dual-DC<br/>Keep dual-router<br/>Keep dual-ISP"]
  D --> D1["Keep all<br/>Add inter-DC VPN<br/>Add inter-DC iBGP"]

Small estate (1 site, 1 router, 1 ISP). Remove the dual-DC, dual-router, and dual-ISP components. The operator keeps the underlay (Layer 1), the IGP (OSPF, optional for a single-router estate), the services (VRRP is not needed for a single-router estate, but NAT and DHCP are still needed), the monitoring (SNMP + exporters), and the backup (config + image).

Medium estate (1 site, 2 routers, 2 ISPs). Remove the dual-DC component. Keep the dual-router and dual-ISP components. The operator adds VRRP for first-hop redundancy, iBGP for intra-DC failover, and the dual-ISP BGP configuration.

Large estate (2 sites, 4 routers, 2 ISPs). Keep all components from the reference topology. The operator adds the inter-DC VPN, the inter-DC iBGP, the inter-DC VRRP (or a separate VIP at each site), and the cross-site monitoring.

When the assumptions break down

The reference topology’s design assumptions break down in three canonical scenarios:

flowchart LR
  subgraph A["Reference assumptions"]
    A1["Dual-DC: site failure tolerated"]
    A2["Dual-router: router failure tolerated"]
    A3["Dual-ISP: ISP failure tolerated"]
    A4["VRRP: first-hop failover"]
    A5["OSPF underlay: small IGP"]
  end
  subgraph B["Assumption breakdown"]
    B1["Single site: dual-DC is overkill"]
    B2["Single router: dual-router is overkill"]
    B3["Single ISP: dual-ISP is not possible"]
    B4["Application uses physical IP: VRRP is irrelevant"]
    B5["Large IGP: OSPF area design needs review"]
  end
  A1 -.->|breakdown| B1
  A2 -.->|breakdown| B2
  A3 -.->|breakdown| B3
  A4 -.->|breakdown| B4
  A5 -.->|breakdown| B5

The discipline is to recognise the assumption breakdown before deploying the topology. The operator who deploys a dual-DC topology for a single-site estate is over-engineering; the operator who deploys a single-router topology for a production estate is under-engineering. The middle ground is to adapt the topology to the estate’s actual requirements while preserving the design rationale.

Production failure modes

The rationale failure modes the operator encounters:

• Topology is too large for the estate. The operator deploys the reference topology for a single-router estate and is over-engineered. Fix: adapt the topology to the estate size; remove the components that are not needed.
• Topology is too small for the estate. The operator deploys a single-router topology for a production estate and discovers the gap during the next incident. Fix: adapt the topology to the estate size; add the components that are needed.
• Rationale is not documented. The operator deploys the topology without understanding the rationale and cannot adapt it to a changing requirement. Fix: document the rationale for every design decision; review the rationale during every change.
• Trade-offs are not understood. The operator deploys the topology without understanding the trade-offs and cannot recognise when the trade-offs break down. Fix: document the trade-offs; review the trade-offs during every change.
• Alternatives are not considered. The operator deploys the topology without considering the alternatives and cannot recognise when an alternative is a better fit. Fix: document the alternatives; review the alternatives during every change.

Rollback

The rollback for a rationale change is the rollback of the topology. The discipline:

• Topology changes — rollback N and commit. The operator must test the rollback in a lab before deploying to production.
• Design rationale changes — the operator must document the change in the operator’s inventory; the change must be reviewed by a second operator before deployment.
• Trade-off changes — the operator must document the trade-off in the operator’s inventory; the trade-off must be reviewed by a second operator before deployment.

Production discipline

Reviewing the rationale during change

The rationale must be reviewed during every change. The discipline is to ask four questions for every change:

flowchart TD
  Q1["Q1: Which design decision is affected?"]
  Q2["Q2: What is the rationale for the decision?"]
  Q3["Q3: What is the trade-off being changed?"]
  Q4["Q4: What is the alternative if the rationale breaks down?"]
  Q1 --> Q2
  Q2 --> Q3
  Q3 --> Q4
  Q4 --> D["Decision: approve or reject"]

The operator who cannot answer all four questions does not understand the rationale for the change. The operator who can answer all four questions is making an informed decision. The discipline: every change must pass the four-question review; the review must be documented in the change pipeline; the review must be reviewed by a second operator before the change is deployed.

Cross-course references

• Part LVII-01 (vyos-lvii-01-reference-topology) covers the topology that this lesson explains.
• Part LVII-03 (vyos-lvii-03-reference-failure-domains) covers the failure domains that the rationale is designed to tolerate.
• The Observability course covers the consumer side: Prometheus, Grafana, Loki, Tempo.
• The Ansible course’s XLII-Ansible-BeyondLinux covers the automation hand-off (rolling out the topology to a fleet via a single playbook).
• The Linux course’s XXII-Linux-NetTroubleshoot covers the underlying file descriptor and TCP session primitives.

Quiz