Reference failure domains — blast radius, isolation, and failover for every layer

A failure domain is not a single failure. It is the boundary of impact: which routers are affected, which applications are affected, which users are affected. The operator who designs a topology without understanding the failure domains will deploy a topology that cascades (a failure in one layer causes a failure in another), will not be able to scope the impact of a failure, and will not be able to prioritise the recovery.

This lesson is the failure-domain discipline for the reference topology: every layer has a failure domain, an isolation mechanism, a failover mechanism, and a cascading failure mode. The discipline is to prevent a failure in one layer from cascading to another.

The failure-domain matrix

The reference topology is organised into seven layers. Each layer has a failure domain, an isolation mechanism, a failover mechanism, and a cascading failure mode.

flowchart TD
  L1["1. Underlay<br/>(failure: link, SFP, switch port)"]
  L2["2. IGP<br/>(failure: OSPF neighbour)"]
  L3["3. BGP<br/>(failure: eBGP session, iBGP session)"]
  L4["4. VPN<br/>(failure: WireGuard, IPsec tunnel)"]
  L5["5. Services<br/>(failure: VRRP, NAT, DHCP)"]
  L6["6. Monitoring<br/>(failure: exporter, SNMP, gNMI)"]
  L7["7. Backup & DR<br/>(failure: config, image, offsite)"]
  L1 --> L2
  L2 --> L3
  L3 --> L4
  L4 --> L5
  L5 --> L6
  L6 --> L7

Layer 1 — Underlay

Failure domain: Single link, single SFP, single switch port, single router.

Blast radius: A link failure affects the routers and the applications that depend on the link. A single-router failure affects all applications at that router’s site.

Isolation mechanism: Bond interfaces (LACP), dual-ISP connections, dual-router topology.

Failover mechanism: LACP failover (sub-second), BGP path failover (30 seconds), iBGP failover + VRRP (3 seconds + 30 seconds).

Cascading failure mode: A link failure can cause a BGP path failover, which can cause an IGP reconvergence, which can cause a transient packet loss. The discipline is to isolate the link failure at the underlay layer (LACP) before it cascades to the BGP layer.

Layer 2 — IGP (OSPF)

Failure domain: Single neighbour, single area, single router, single site.

Blast radius: A neighbour failure affects the routes that go through that neighbour. An area failure affects all routers in the area. A single-router failure affects all routes that go through that router.

Isolation mechanism: OSPF areas (backbone, stub, NSSA), OSPF interface authentication, OSPF passive interfaces.

Failover mechanism: OSPF neighbour failure detection (10 seconds by default), SPF recomputation (1-5 seconds), route installation (sub-second).

Cascading failure mode: A neighbour failure can cause a SPF storm (continuous SPF runs), which can cause CPU saturation, which can cause a BGP session failure. The discipline is to throttle the SPF (set protocols ospf timers throttle spf 200 1000 5000) to prevent the SPF storm from cascading to the BGP layer.

Layer 3 — BGP

Failure domain: Single peer, single AS, single router, single site.

Blast radius: A peer failure affects the routes that go through that peer. An AS failure affects all routes that go through that AS. A single-router failure affects all BGP sessions on that router.

Isolation mechanism: BGP prefix filtering, BGP AS-path filtering, BGP community filtering, BGP max-prefix.

Failover mechanism: BGP session failure detection (180 seconds by default, BFD: sub-second), BGP path failover (30 seconds), BGP route damping (15 minutes).

Cascading failure mode: A BGP update burst can cause CPU saturation, which can cause a router failure. The discipline is to bound the BGP update burst (neighbor maximum-prefix) to prevent the burst from cascading to the router.

flowchart LR
  subgraph CASCADE["Cascading failure: BGP → CPU → router"]
    B["BGP update burst<br/>(peer sends 900,000 routes)"]
    C["CPU saturation<br/>(bgpd consumes 100% CPU)"]
    R["Router failure<br/>(bgpd is killed by OOM)"]
  end
  B --> C
  C --> R

Layer 4 — VPN

Failure domain: Single tunnel, single peer, single router, single site.

Blast radius: A tunnel failure affects the traffic that goes through the tunnel. A peer failure affects all tunnels to that peer. A single-router failure affects all tunnels on that router.

Isolation mechanism: WireGuard + IPsec (dual technology), IKEv2 with PFS, MTU clamping.

Failover mechanism: WireGuard keepalive (10 seconds), IPsec DPD (Dead Peer Detection, 10 seconds), route failover (30 seconds).

Cascading failure mode: A tunnel failure can cause an IGP reconvergence, which can cause a BGP reconvergence, which can cause a transient packet loss. The discipline is to monitor the tunnel state and to fail over to the backup tunnel before the IGP reconverges.

Layer 5 — Services

Failure domain: Single service, single VRF, single router, single site.

Blast radius: A VRRP failure affects the application servers that use the VRRP virtual IP. A NAT port exhaustion affects all outbound traffic from the router. A DHCP lease exhaustion affects all DHCP clients at the site.

Isolation mechanism: VRRP (virtual IP failover), NAT pools (multiple IP addresses), DHCP scopes (multiple subnets).

Failover mechanism: VRRP failover (3 seconds), NAT port reuse (immediate), DHCP lease renewal (50% of lease time).

Cascading failure mode: A VRRP failure can cause a transient packet loss, which can cause the application to timeout, which can cause a cascade of application retries. The discipline is to design the application to tolerate the VRRP failover time (3 seconds).

Layer 6 — Monitoring

Failure domain: Single exporter, single SNMP agent, single gNMI subscription.

Blast radius: A monitoring failure affects the operator’s visibility into the router. The router itself is unaffected; the operator is blind.

Isolation mechanism: Multiple exporters (node_exporter + prometheus-vyos-exporter), multiple transports (SNMP + gNMI + Prometheus), multiple destinations (Prometheus + NMS).

Failover mechanism: None (the monitoring failure is silent; the operator does not know the monitoring is down until the next incident).

Cascading failure mode: A monitoring failure can mask a real failure. The operator believes the router is healthy (no alerts) when it is actually failing. The discipline is to monitor the monitoring (the Prometheus server’s /metrics endpoint, the SNMP agent’s response, the gNMI subscription’s stream).

Layer 7 — Backup & DR

Failure domain: Single configuration revision, single image, single offsite backup.

Blast radius: A configuration failure affects the router that runs the configuration. An image corruption affects the router that boots the image. An offsite disconnect affects the disaster recovery plan.

Isolation mechanism: Configuration revision history (rollback N), dual image (current + previous), offsite replication (multiple destinations).

Failover mechanism: Configuration rollback (rollback N and commit), image fallback (boot into the previous image), offsite fallback (reconnect to the backup site).

Cascading failure mode: A backup failure can cascade to a DR failure. The operator who does not have a tested backup will discover the gap during the next DR drill. The discipline: test the backup end-to-end before declaring the topology production-ready; run quarterly DR drills.

The isolation discipline

The isolation discipline is the set of design decisions that prevent a failure in one layer from cascading to another. The discipline has three dimensions:

flowchart TD
  ISO["Isolation discipline"]
  ISO --> PI["Process isolation<br/>(separate processes, cgroups)"]
  ISO --> NI["Network isolation<br/>(VRF, VLAN, firewall)"]
  ISO --> CI["Configuration isolation<br/>(separate configurations, version control)"]

Process isolation. The FRR daemons (zebra, bgpd, ospfd) run as separate processes; a crash in one daemon does not crash the others. The monitoring exporters run as separate processes; a memory leak in one exporter does not consume the router’s memory. The VPN processes (WireGuard, strongSwan) run as separate processes; a crash in one tunnel does not crash the other. The discipline: use separate processes for every layer; cap the resource usage via cgroup.

Network isolation. The underlay, the IGP, the BGP, the VPN, the services, and the monitoring use separate IP subnets and VRFs. A failure in one subnet does not cascade to another subnet. The discipline: use separate subnets and VRFs for every layer; use firewall rules to control the inter-subnet traffic.

Configuration isolation. The VyOS configuration tree is hierarchical; the operator can configure each layer independently. The discipline: use the configuration tree to isolate each layer; document the configuration in the operator’s inventory; use version control (Ansible, Git) to track the configuration changes.

Production failure modes

The failure-domain failure modes the operator encounters:

• Cascading failure from link to BGP. A link failure causes a BGP path failover (30 seconds), which causes an IGP reconvergence (5-10 seconds), which causes a transient packet loss. Fix: use LACP for the link; configure BFD for sub-second failover detection.
• Cascading failure from OSPF to CPU. A neighbour failure causes a SPF storm, which causes CPU saturation, which causes a router failure. Fix: throttle the SPF (set protocols ospf timers throttle spf 200 1000 5000); bound the LSDB size.
• Cascading failure from BGP to router. A BGP update burst causes CPU saturation, which causes the OOM killer to kill the FRR daemon. Fix: bound the BGP update burst (neighbor maximum-prefix); cap the FRR daemon’s memory usage via cgroup.
• Cascading failure from VPN to IGP. A tunnel failure causes an IGP reconvergence, which causes a BGP reconvergence. Fix: monitor the tunnel state; fail over to the backup tunnel before the IGP reconverges.
• Monitoring failure masking a real failure. The monitoring exporter crashes; the operator does not see the real failure. Fix: monitor the monitoring; alert on the absence of a metric.
• Backup failure cascading to DR failure. The backup is not tested; the DR drill reveals the gap. Fix: test the backup end-to-end; run quarterly DR drills.

Rollback

The rollback for a failure-domain change is the rollback of the topology. The discipline:

• Isolation changes — rollback N and commit. The operator must test the rollback in a lab before deploying to production.
• Failover configuration — rollback N and commit. The operator must test the failover in a lab before deploying to production.
• Cascading failure mitigation — the operator must identify the root cause before rolling back; the rollback is the last resort.

Production discipline

Common cascade chain patterns

The reference topology has several canonical cascade chain patterns that the operator must recognise:

flowchart LR
  subgraph CC1["Link → BGP → IGP cascade"]
    L1["Link flap"] --> B1["BGP session flap"]
    B1 --> I1["IGP reconvergence"]
    I1 --> P1["Transient packet loss"]
  end
  subgraph CC2["BGP → CPU → router cascade"]
    U1["BGP update burst"] --> C1["CPU saturation"]
    C1 --> O1["OOM killer fires"]
    O1 --> R1["Router failure"]
  end
  subgraph CC3["VPN → IGP → BGP cascade"]
    T1["VPN tunnel flap"] --> I2["IGP route withdrawal"]
    I2 --> B2["BGP next-hop change"]
    B2 --> P2["BGP reconvergence"]
  end
  subgraph CC4["Monitoring → operator cascade"]
    M1["Exporter crash"] --> O2["Operator blind"]
    O2 --> R2["Real failure invisible"]
    R2 --> I3["Incident response delayed"]
  end

Each cascade chain pattern has a trigger (the link flap, the BGP burst, the VPN tunnel flap, the exporter crash), a propagation path (BGP, IGP, CPU, monitoring), and an impact (packet loss, router failure, delayed incident response). The operator who recognises the cascade chain pattern can break the cascade at the earliest step; the operator who does not recognise the pattern will discover the cascade during the next incident.

The discipline: catalogue the cascade chain patterns in the operator’s inventory; document the trigger, the propagation path, and the impact for each pattern; document the earliest breaking point for each pattern; review the patterns during every change.

Cross-course references

• Part LVII-01 (vyos-lvii-01-reference-topology) covers the topology that this lesson scopes.
• Part LVII-02 (vyos-lvii-02-reference-rationale) covers the design rationale that this lesson explains.
• Part LII (vyos-lii-01-define-and-scope) covers the troubleshooting methodology that this lesson applies.
• Part XXXII (vyos-xxxii-01-bfd-concept) covers the BFD that this lesson recommends for sub-second failover detection.
• The Observability course covers the consumer side: Prometheus, Grafana, alerting on metric absence.

Quiz