Full router loss — RMA, replacement, restore, verify adjacency

A full router loss is the worst-case scenario for a VyOS deployment: the router is dead (hardware failure, fire, flood) and there is nothing to recover from the local disk. The recovery procedure involves RMA, replacement, restoration, and verification.

The procedure is the operator’s defence against the disaster that the configuration-as-code pipeline, the remote backup, and the Git repository were designed to survive. The recovery is the test of whether the backup strategy works.

This lesson covers the end-to-end procedure, the RMA process, the configuration restore, the adjacency verification, and the production failure modes where the recovery goes wrong.

The disaster recovery procedure

The end-to-end procedure has six stages:

1. Confirm the failure — verify that the router is truly dead (not just unreachable).
2. Initiate RMA — contact the vendor; obtain a replacement router.
3. Install the replacement — physical install, fresh VyOS installation, network connectivity.
4. Restore the configuration — load from the Git repository, verify, commit, save.
5. Verify adjacency — confirm that all routing- protocol sessions are established.
6. Verify traffic — confirm that production traffic flows correctly.

flowchart LR
  FAIL[Router failure] --> CONFIRM[Confirm failure]
  CONFIRM --> RMA[Initiate RMA]
  RMA --> INSTALL[Install replacement]
  INSTALL --> RESTORE[Restore from Git]
  RESTORE --> ADJ[Verify adjacency]
  ADJ --> TRAFFIC[Verify traffic]
  TRAFFIC --> DONE[Recovery complete]

The diagram shows the six-stage procedure. Each stage is a gate; the operator cannot proceed without verifying the previous stage succeeded.

Stage 1: confirm the failure

The router is unreachable. The operator must verify that the router is truly dead, not just unreachable due to a network outage or a configuration error:

• Power cycle the router — disconnect and reconnect power. If the router does not come back, it is dead.
• Check the console — connect via serial console or IPMI. If the console shows no output, the router is dead.
• Check the indicator lights — most routers have status LEDs that indicate hardware health. If the LEDs show no power, the router is dead.

The defensive idiom: every unreachable router is investigated via console before the operator declares it dead. A router that is unreachable due to a network outage is not dead; the operator must fix the network outage, not the router.

Stage 2: initiate RMA

The operator contacts the vendor and initiates the RMA (return merchandise authorisation) process:

• Identify the router — model, serial number, asset tag.
• Describe the failure — what happened, what the indicators show, what the operator has tried.
• Obtain a replacement — the vendor ships a replacement; the operator receives it within the RMA SLA (typically 1-5 business days).

The defensive idiom: the operator maintains a stock of spare routers for critical deployments. The spare router is pre-configured with the latest VyOS version and is ready to deploy. The RMA wait is avoided.

Stage 3: install the replacement

The operator installs the replacement router:

• Physical install — rack the router, connect power, connect network cables.
• VyOS installation — boot from the ISO, install to disk, configure basic system settings (hostname, IP addresses, default route).
• Network connectivity — verify that the router is reachable on the management network.

The installation follows the standard VyOS procedure covered in Part IV (vyos-iv-01-iso-install).

Stage 4: restore the configuration

The operator restores the configuration from the Git repository:

# Copy the configuration to the new router
scp backup@backup.internal:/srv/configs/edge-01-config.boot /tmp/edge-01-config.boot

# SSH to the new router and load the configuration
ssh vyos@192.0.2.1
configure
load /tmp/edge-01-config.boot
compare
commit
save
exit

The procedure is the same as the same-appliance restore: load, verify, commit, save. The configuration is from the Git repository (or the off-box file server); the operator verifies it matches the expected state before committing.

Stage 5: verify adjacency

After the configuration is applied, the operator verifies that all routing-protocol sessions are established:

# Verify OSPF adjacencies
show ip ospf neighbor

# Verify BGP sessions
show bgp summary

# Verify IS-IS adjacencies
show isis neighbor

# Verify routing table
show ip route

The output should show all expected adjacencies in the Full or Established state. A missing adjacency is a signal that the configuration is incomplete or that the peer is also affected by the same failure.

Note the BGP verb. On VyOS 1.5 the operational command is show bgp summary, and the address-family-qualified forms are show bgp ipv4 summary and show bgp ipv6 summary. The show ip bgp summary spelling belongs to the older CLI generation; a recovery runbook that still carries it is a runbook nobody has run on the version they are recovering onto, which is exactly the thing a disaster is a bad time to discover.

flowchart LR
  ADJ[Adjacency check] -->|all established| TRAFFIC[Traffic verification]
  ADJ -->|missing| DEBUG[Debug]
  DEBUG -->|firewall?| FW[Check firewall rules]
  DEBUG -->|interface?| IF[Check interface state]
  DEBUG -->|peer?| PEER[Check peer state]
  FW --> ADJ
  IF --> ADJ
  PEER --> ADJ

The diagram shows the adjacency verification loop: if an adjacency is missing, the operator investigates (firewall, interface, peer state) until the adjacency is established.

Stage 6: verify traffic

After all adjacencies are established, the operator verifies that production traffic flows correctly:

• Ping test — ping a known-good IP on the other side of the router.
• Traceroute — traceroute through the router to verify the path.
• BGP route check — verify that the expected routes are in the routing table.
• End-to-end test — run an application-level test (e.g. a synthetic transaction) to verify that the router is forwarding traffic.

The traffic verification is the operator’s final confirmation that the recovery succeeded. A router with all adjacencies established but no traffic forwarding is a router with a routing or firewall issue.

Failure modes

Wrong configuration restored

The operator restores the wrong configuration (e.g. the configuration from a different router, or the configuration from 6 months ago). The router comes up with the wrong BGP peers, the wrong firewall rules, the wrong static routes.

Diagnostic: the adjacency verification shows unexpected peers; the traffic verification shows incorrect paths.

Fix: load the correct configuration from the Git repository; verify with compare; commit; save.

Adjacency does not establish

The configuration is correct but the adjacency does not establish. The most common causes:

• Peer is also down — the peer router is also affected by the same failure (e.g. power outage in the data centre).
• Firewall blocks the protocol — the firewall rule for the routing protocol is not in the configuration.
• Interface is down — the interface is not configured or the cable is not connected.

Diagnostic: show ip ospf neighbor and show ip bgp summary show the session as Idle or Down.

Fix: investigate the cause (peer, firewall, interface); correct the configuration or the physical layer.

Traffic does not flow

All adjacencies are established but traffic does not flow. The most common causes:

• Routing table is incomplete — a route is missing from the routing table.
• Firewall rule blocks traffic — a firewall rule blocks the traffic.
• Asymmetric routing — the return path takes a different route than the forward path.

Diagnostic: show ip route shows the expected routes; the firewall log shows blocked packets; traceroute shows the path.

Fix: investigate the cause (routing, firewall, asymmetric routing); correct the configuration.

Rollback

A full router loss recovery is itself a recovery. There is no further rollback. The operator recovers from the Git repository; the Git repository is the source of truth.

The VyOS commit history on the new router starts empty — there is no previous revision to roll back to. The operator’s recovery mechanism is the Git repository, not the router’s commit history.

Production discipline

Cross-course references

• LV-VyOS-Backup (vyos-lv-01-saved-configuration through vyos-lv-04-restore-different-appliance) cover the saved configuration, remote backup, and restore procedures that the full router loss recovery depends on.
• LV-VyOS-Backup (vyos-lv-06-dr-validation, the next lesson) covers the disaster recovery drill that verifies the recovery procedure works.
• IV-VyOS-Install (vyos-iv-01-iso-install) covers the installation that precedes the configuration restore.

Quiz