DR validation — quarterly DR drill, full restore, scenario test

A disaster recovery procedure that has never been tested is not a procedure — it is a hypothesis. The operator who has never run a DR drill does not know whether the Git repository has a usable configuration, whether the spare router boots correctly, whether the recovery time is within the business SLA.

DR validation is the discipline that turns the hypothesis into a known result. The drill exercises the full recovery procedure; the operator measures the recovery time; the operator documents the issues; the operator fixes the issues before the next drill.

This lesson covers the quarterly DR drill, the full restore from the Git repository, the scenario test, and the production failure modes where the DR procedure has never been tested.

The quarterly DR drill

The DR drill is run on a quarterly schedule. The drill exercises the full recovery procedure for one router (or a subset of routers):

1. Select a router — choose a router that is representative of the production fleet.
2. Schedule the drill — schedule the drill during a maintenance window; communicate to stakeholders.
3. Simulate the failure — power off the selected router (or simulate by isolating it from the network).
4. Run the recovery procedure — follow the runbook; restore from the Git repository; verify adjacency; verify traffic.
5. Measure the recovery time — record the time from failure detection to full recovery.
6. Document the issues — record any issues encountered during the drill; assign action items.
7. Restore normal operation — verify that the router is fully operational; close the drill.

flowchart LR
  SELECT[Select router] --> SCHED[Schedule drill]
  SCHED --> SIM[Simulate failure]
  SIM --> RECOVER[Run recovery]
  RECOVER --> MEASURE[Measure RTO]
  MEASURE --> DOC[Document issues]
  DOC --> RESTORE[Restore operation]
  RESTORE --> NEXT[Schedule next drill]

The diagram shows the DR drill procedure. Each step is required; the operator cannot skip a step.

The full restore test

The DR drill includes a full restore test: the operator restores the router’s configuration from the Git repository onto a fresh installation. The test verifies that:

• The Git repository has the correct configuration.
• The fresh installation accepts the configuration without errors.
• All routing-protocol sessions establish.
• Production traffic flows correctly.

The full restore test is the operator’s defence against a Git repository that has drifted from the production state. A Git repository that has not been tested may have errors (typos, missing rules, stale references) that are only caught when the operator tries to restore from it.

The scenario test

The DR drill includes a scenario test: the operator simulates a specific disaster scenario and verifies that the recovery procedure handles it:

• Scenario 1: router hardware failure — the operator powers off the router and verifies that the spare router takes over.
• Scenario 2: data centre power outage — the operator simulates a data centre power outage and verifies that the routers come up when power is restored.
• Scenario 3: configuration corruption — the operator corrupts /config/config.boot and verifies that the router can be recovered from the Git repository.
• Scenario 4: security compromise — the operator simulates a compromise and verifies that the configuration can be restored from a known-good Git commit.

Each scenario tests a specific failure mode. The operator documents the recovery time for each scenario; the operator identifies the issues specific to each scenario.

Measuring the recovery time

The operator measures the recovery time (RTO — recovery time objective) for each drill. The RTO is the time from failure detection to full recovery.

The operator compares the measured RTO to the business SLA. If the measured RTO exceeds the SLA, the operator must identify the bottleneck and reduce the recovery time.

Common bottlenecks:

• RMA wait — the vendor’s RMA process is slow. The fix: maintain a spare router.
• Spare router install — the spare router is not pre-configured. The fix: pre-configure the spare router with the latest VyOS version.
• Configuration restore — the Git repository is not on the OOB network. The fix: copy the Git repository to the OOB network.

Failure modes

Git repository out of date

The DR drill reveals that the Git repository is missing the last 3 months of configuration changes. The production router has been managed manually outside the pipeline.

Diagnostic: the Git repository’s most recent commit is 3 months old; the production router has changes that are not in Git.

Fix: investigate why the manual changes were not committed to Git. Implement the configuration-as-code discipline (pipeline is the only path to the router).

Spare router not pre-configured

The DR drill reveals that the spare router is not pre-configured. The RTO is dominated by the install time.

Diagnostic: the spare router has no VyOS installation; the operator must install VyOS from scratch.

Fix: pre-configure the spare router with the latest VyOS version; the spare is ready to deploy.

Recovery procedure not documented

The DR drill reveals that the recovery procedure is not documented. The operator improvises the recovery; the recovery is slow and error-prone.

Diagnostic: the runbook is missing or out of date.

Fix: document the recovery procedure in a runbook; update the runbook after every drill.

Drill scheduled but not run

The DR drill is on the calendar but never actually run. The drill is “postponed” due to other priorities; the postponement extends indefinitely.

Diagnostic: the calendar shows the drill was scheduled but the drill report is missing.

Fix: assign a specific operator to the drill; the operator is accountable for running the drill on schedule.

Rollback

A DR drill is itself a recovery test. There is no rollback from a DR drill; the drill is the operator’s evidence that the recovery works.

The drill may surface issues that require fixes. The fixes are committed to the runbook; the next drill verifies the fixes.

Production discipline

Cross-course references

• LV-VyOS-Backup (vyos-lv-05-full-router-loss, the previous lesson) covers the full router loss recovery that the DR drill exercises.
• LIV-VyOS-Automation (vyos-liv-05-change-management) covers the change management discipline that ensures the Git repository is up to date.
• LII-VyOS-Troubleshooting (vyos-lii-01-define-and-scope) covers the troubleshooting methodology that the DR drill uses.

Quiz