Reference DR — backup strategy, RTO, RPO, and quarterly drills

A disaster recovery plan that has not been tested is a wish list. The operator who deploys a DR plan without testing it will discover the gap during the next disaster — when it is too late to fix. The RunBook Academy reference DR plan is the canonical pattern for a production VyOS 1.5 LTS estate: configuration backup, image backup, offsite replication, RTO/RPO targets, quarterly DR drills, and the recovery runbook for every canonical failure scenario.

This lesson is the reference DR plan: the backup strategy, the RTO/RPO targets, the recovery runbook, and the production discipline that ensures the plan works when it is needed.

The backup strategy

The backup strategy covers three layers: configuration, image, and offsite.

flowchart LR
  subgraph ROUTER["VyOS 1.5 router"]
    CFG["Configuration tree<br/>(running config)"]
    IMG["System image<br/>(VyOS ISO)"]
  end
  subgraph BACKUP["Local backup"]
    CFGB["Configuration backup<br/>(/var/lib/vyos/backup)"]
    IMGB["Image backup<br/>(/var/lib/vyos/image)"]
  end
  subgraph OFFSITE["Offsite backup"]
    S3["S3-compatible storage<br/>(encrypted, versioned)"]
    GIT["Git repository<br/>(configuration history)"]
  end
  CFG --> CFGB
  IMG --> IMGB
  CFGB --> S3
  CFGB --> GIT
  IMGB --> S3

Configuration backup. The VyOS configuration tree is the canonical source of truth. The operator must back up the configuration on every commit and store the backups in a versioned, encrypted offsite location.

# /usr/local/bin/vyos-backup.sh
#!/bin/bash
ROUTER=$(hostname)
DATE=$(date -u +%Y-%m-%dT%H:%M:%SZ)
BACKUP_DIR=/var/lib/vyos/backup
OFFSITE=s3://network-backups/vyos-configs

# Generate the backup
mkdir -p $BACKUP_DIR
show configuration commands > $BACKUP_DIR/${ROUTER}-${DATE}.conf

# Compress and encrypt
gpg --symmetric --cipher-algo AES256 \
    --passphrase-file /etc/vyos/backup-passphrase \
    --output $BACKUP_DIR/${ROUTER}-${DATE}.conf.gpg \
    $BACKUP_DIR/${ROUTER}-${DATE}.conf

# Upload to offsite
aws s3 cp $BACKUP_DIR/${ROUTER}-${DATE}.conf.gpg $OFFSITE/${ROUTER}/

# Push to Git repository
cd /var/lib/vyos/config-git
git add ${ROUTER}-${DATE}.conf
git commit -m "Backup ${ROUTER} at ${DATE}"
git push origin main

# Clean up old local backups (keep last 30 days)
find $BACKUP_DIR -name "*.conf" -mtime +30 -delete
find $BACKUP_DIR -name "*.conf.gpg" -mtime +30 -delete

The script runs on every commit (via a commit-confirm hook or a cron job), generates the backup, encrypts it, uploads it to the offsite, pushes it to the Git repository, and cleans up old local backups.

Image backup. The system image is the canonical source of the router’s operating system. The operator must back up the image on every release upgrade and store the backups offsite.

# /usr/local/bin/vyos-image-backup.sh
#!/bin/bash
ROUTER=$(hostname)
IMAGE=$(show version | grep Version | awk '{print $2}')
OFFSITE=s3://network-backups/vyos-images

# Export the image
cp /usr/share/vyos/.version $BACKUP_DIR/${ROUTER}-${IMAGE}.version
cp /var/lib/vyos/image/${IMAGE}.iso $BACKUP_DIR/${ROUTER}-${IMAGE}.iso

# Upload to offsite
aws s3 cp $BACKUP_DIR/${ROUTER}-${IMAGE}.iso $OFFSITE/${ROUTER}/

# Clean up old local backups (keep last 3 images)
ls -t $BACKUP_DIR/${ROUTER}-*.iso | tail -n +4 | xargs -r rm

The script exports the image and uploads it to the offsite. The operator keeps the last 3 images locally (the current image and two previous images) for quick rollback.

Offsite backup. The offsite backup is the canonical recovery location. The operator must use a separate physical location or cloud provider, with versioning and access controls.

# S3 bucket policy (Terraform)
resource "aws_s3_bucket" "network_backups" {
  bucket = "network-backups-prod"
  versioning {
    enabled = true
  }
  server_side_encryption_configuration {
    rule {
      apply_server_side_encryption_by_default {
        sse_algorithm = "AES256"
      }
    }
  }
  lifecycle_rule {
    enabled = true
    noncurrent_version_expiration {
      days = 365
    }
  }
}

The offsite backup is encrypted at rest (AES-256), versioned (so the operator can recover an earlier backup), and has a lifecycle rule (so old backups are automatically deleted after 365 days).

RTO and RPO targets

The RTO (Recovery Time Objective) and RPO (Recovery Point Objective) targets define the recovery expectations.

flowchart LR
  subgraph RTO["RTO (Recovery Time Objective)"]
    RTO1["RTO-1: Router failure<br/>3 minutes (VRRP failover)"]
    RTO2["RTO-2: Site failure<br/>30 minutes (inter-DC VPN)"]
    RTO3["RTO-3: Total loss<br/>4 hours (image + config)"]
  end
  subgraph RPO["RPO (Recovery Point Objective)"]
    RPO1["RPO-1: Commit-time backup<br/>5 minutes (cron interval)"]
    RPO2["RPO-2: Image backup<br/>0 minutes (every release)"]
    RPO3["RPO-3: Offsite replication<br/>1 hour (S3 sync interval)"]
  end

RTO-1: Router failure (3 minutes). The VRRP failover is the canonical recovery for a router failure. The surviving router takes over the VRRP virtual IP within 3 seconds; the application servers re-establish their connections within 30-60 seconds; the BGP sessions re-converge within 30 seconds. Total RTO: 3 minutes.

RTO-2: Site failure (30 minutes). The inter-DC VPN failover is the canonical recovery for a site failure. The application servers at the affected site fail over to the surviving site via the inter-DC VPN; the BGP advertisement changes to prefer the surviving site; the user-facing services are restored. Total RTO: 30 minutes.

RTO-3: Total loss (4 hours). The image-and-configuration recovery is the canonical recovery for a total loss. The operator must obtain the image from the offsite backup, boot the router from the image, restore the configuration from the offsite backup, and re-establish the BGP sessions. Total RTO: 4 hours.

RPO-1: Commit-time backup (5 minutes). The configuration backup is run on every commit (via a hook) or on a 5-minute cron. The RPO for a configuration change is the time between the last backup and the disaster.

RPO-2: Image backup (0 minutes). The image backup is run on every release upgrade. The RPO for an image change is 0 minutes (the backup is taken before the upgrade).

RPO-3: Offsite replication (1 hour). The offsite replication is run on a 1-hour cron. The RPO for an offsite replication failure is 1 hour.

The recovery runbook

The recovery runbook is the canonical procedure for recovering from every failure scenario. The runbook is stored in the operator’s documentation system (Confluence, Notion, Git) and is tested during the quarterly DR drill.

flowchart TD
  R1["Router failure"]
  R2["Site failure"]
  R3["Total loss"]
  R1 --> R1A["Step 1: Verify VRRP failover<br/>(3 minutes)"]
  R1 --> R1B["Step 2: Verify BGP re-convergence<br/>(30 seconds)"]
  R1 --> R1C["Step 3: Document the failure<br/>(post-incident review)"]
  R2 --> R2A["Step 1: Verify inter-DC VPN<br/>(5 minutes)"]
  R2 --> R2B["Step 2: Verify BGP re-convergence<br/>(2 minutes)"]
  R2 --> R2C["Step 3: Fail over application servers<br/>(15 minutes)"]
  R2 --> R2D["Step 4: Document the failure"]
  R3 --> R3A["Step 1: Obtain image from offsite<br/>(30 minutes)"]
  R3 --> R3B["Step 2: Boot router from image<br/>(30 minutes)"]
  R3 --> R3C["Step 3: Restore configuration<br/>(1 hour)"]
  R3 --> R3D["Step 4: Verify BGP and OSPF<br/>(30 minutes)"]
  R3 --> R3E["Step 5: Verify VPN and services<br/>(30 minutes)"]
  R3 --> R3F["Step 6: Document the recovery"]

The runbook covers the three canonical failure scenarios: router failure, site failure, total loss. Each scenario has a step-by-step procedure, a time estimate, and a documentation step.

The quarterly DR drill

The quarterly DR drill is the canonical validation of the DR plan. The operator must run a DR drill every quarter and document the results.

# DR drill checklist (Q3 2026)
## Scenario
Total loss of R1-A (the canonical router at DC-East).

## Pre-drill
- [ ] Confirm the offsite backup is current (last commit within 5 minutes)
- [ ] Confirm the offsite image is current (last release upgrade within 90 days)
- [ ] Confirm the recovery runbook is current (last reviewed within 30 days)
- [ ] Confirm the DR drill participants are available

## Drill
- [ ] Step 1: Power off R1-A
- [ ] Step 2: Provision a replacement router (VyOS 1.5 LTS ISO)
- [ ] Step 3: Boot the replacement router from the offsite image (target: 30 minutes)
- [ ] Step 4: Restore the configuration from the offsite backup (target: 30 minutes)
- [ ] Step 5: Verify the BGP sessions (target: 5 minutes)
- [ ] Step 6: Verify the OSPF neighbours (target: 1 minute)
- [ ] Step 7: Verify the VPN tunnels (target: 5 minutes)
- [ ] Step 8: Verify the services (VRRP, NAT, DHCP) (target: 5 minutes)
- [ ] Step 9: Document the actual RTO and RPO

## Post-drill
- [ ] Update the recovery runbook with the actual times
- [ ] Update the RTO and RPO targets if needed
- [ ] Schedule a post-drill review meeting
- [ ] Document the lessons learned

The DR drill is run quarterly (Q1, Q2, Q3, Q4) and the results are documented in the operator’s incident log. The actual RTO and RPO are compared to the targets; deviations are investigated and fixed.

Production failure modes

The DR plan failure modes the operator encounters:

• Backup not encrypted. The configuration backup is unencrypted; the backup contains sensitive information (SNMP community strings, BGP MD5 keys, IPsec pre-shared keys). Fix: encrypt the backup (AES-256); restrict access to the backup.
• Backup not offsite. The backup is stored in the same data center as the router; a data center failure loses both the router and the backup. Fix: store the backup offsite (a separate cloud provider, a separate physical location).
• Backup not tested. The backup is taken but never tested; the operator discovers the backup is corrupt during the next disaster. Fix: run the quarterly DR drill; validate the backup end-to-end.
• RTO/RPO targets not validated. The targets are aspirational; the actual RTO is 5 minutes longer than the target. Fix: validate the targets during the DR drill; document the actual values; update the targets if needed.
• Recovery runbook missing. The runbook is not documented; the operator must improvise during the disaster. Fix: document the runbook; review the runbook every 30 days; validate the runbook during the DR drill.
• Image not bootable. The offsite image is corrupt or outdated; the operator cannot boot the replacement router. Fix: validate the image end-to-end during the DR drill; keep the last 3 images offsite.

Rollback

The rollback for a DR plan change is the rollback of the configuration. The discipline:

• Backup script changes — the operator must version-control the script in Git; the rollback is git revert and re-running the script.
• Image backup changes — the operator must version-control the script in Git; the rollback is git revert and re-running the script.
• Offsite replication changes — the operator must version-control the configuration in Terraform; the rollback is terraform apply with the previous configuration.
• Recovery runbook changes — the operator must version-control the runbook in Git; the rollback is git revert and re-publishing the runbook.

Production discipline

Cross-course references

• Part LVII-01 (vyos-lvii-01-reference-topology) covers the topology that this lesson protects.
• Part LV (vyos-lv-01-saved-configuration) covers the configuration backup that this lesson extends.
• Part LV-06 (vyos-lv-06-dr-validation) covers the DR validation that this lesson applies.
• The Ansible course’s XLII-Ansible-BeyondLinux covers the automation hand-off (rolling out the backup script to a fleet via a single playbook).
• The Observability course covers the monitoring of the DR plan (backup age, offsite replication status).

Quiz