Full appliance disaster recovery — the runbook when the box is gone · OPNsense

Full appliance disaster recovery is the scenario where the firewall itself is gone — hardware failure, site loss, ransomware on the management host that took the firmware with it. The operator must rebuild from scratch on new hardware and restore traffic within the recovery time objective the business has committed to. The runbook that makes this possible is not improvisation; it is a documented procedure tested offline.

This lesson covers the full-appliance DR runbook from site loss to traffic restored, the rekeying discipline that must follow any restore where the previous appliance’s secrecy is in doubt, the offline DR test that proves the runbook, and the lessons-learned cycle that updates the runbook for the next incident.

The DR runbook, end to end

A defensible full-appliance DR runbook is a numbered procedure with explicit checkpoints. The operator who follows it under pressure makes fewer mistakes than the operator who improvises. The runbook below assumes the worst case — the original hardware is destroyed, no replacement appliance is on-site, and the off-host backup is the only artefact the operator has.

Step	Action	Time	Checkpoint
1	Confirm the loss. Distinguish hardware failure from configuration failure.	5 min	Two operators agree the appliance is non-recoverable
2	Provision a replacement appliance. Boot the OPNsense installer on identical hardware if available, or the closest available hardware if not.	60 min	Installer reaches the console prompt
3	Apply the most recent known-good backup. Use the single-user shell method or the GUI’s restore page.	15 min	Firewall boots with the restored configuration
4	Remap interfaces if hardware differs from the source.	15 min	Each physical port serves its intended logical role
5	Run the post-restore verification checklist.	30 min	Every item in the checklist passes
6	Rekey the credentials. Treat the restored appliance as compromised until rekeying is complete.	30 min	New SSH keys, new GUI passwords, new RADIUS shared secrets, new IPsec PSKs
7	Reconnect to the production network.	10 min	Traffic flows, monitoring confirms
8	Update the off-host backup.	5 min	New backup captures the post-DR configuration
9	Document the incident.	30 min	Timeline, root cause, lessons learned, runbook updates

The total elapsed time is 3-4 hours for a disciplined operator with a current backup. Without a current backup, the time stretches to days: the operator must reconstruct the configuration from documentation, screenshots, and tribal knowledge, and the configuration is approximate at best.

The rekeying discipline

A restored configuration contains every credential the previous appliance held: SSH keys, GUI passwords, RADIUS shared secrets, IPsec PSKs, certificate private keys. If the disaster that destroyed the appliance also exposed the configuration — site breach, ransomware that exfiltrated first — those credentials are compromised. The restored firewall must rekey every credential, not because the restore is broken, but because the secrets are no longer secret.

The rekeying list:

Credential	Rekey action
Local user passwords	Reset via console or API; force rotation of any external identity provider
SSH host keys	Regenerate via `/etc/rc.d/ssh_keygen`; update every known_hosts entry on management hosts
SSH user keys	Reissue to every operator; revoke old keys on the restored host
GUI certificate	Generate a new self-signed or ACME certificate
IPsec PSKs	Coordinate with every peer; rotate PSKs on a known schedule
RADIUS / TACACS shared secrets	Coordinate with the authentication server; rotate
WireGuard private keys	Generate new keys; redistribute public keys to peers
Certificate authority	If the CA private key is suspected exposed, reissue the entire PKI
API keys	Rotate every API key; revoke the previous keys at the consumer

The rekeying is not optional. The operator who restores a configuration from a compromised site and skips rekeying has restored a compromised firewall.

The offline DR test

The DR runbook that has never been tested is folklore. The offline DR test takes the runbook from the page to the bench.

The test procedure:

Schedule a maintenance window. The test is offline; production traffic is not affected. The window is announced.
Stand up an isolated DR test bench. A second appliance (the cold spare, or a borrowed unit) is racked in a separate subnet, isolated from production.
Use the latest production backup. The backup file is copied to the test bench via the off-host channel (proving the channel works).
Walk the runbook step by step. Every operator who would be on call during a real DR is on the bench. They follow the runbook exactly as written.
Measure elapsed time at each checkpoint. The measurements update the runbook’s time estimates.
Verify the test bench traffic flow. Plug a test laptop into the LAN port, confirm DHCP, GUI, and external ping.
Document gaps. Every step that took longer than expected, every missing detail, every unclear instruction becomes a runbook update.
Tear down the test bench. Wipe the test appliance to factory defaults. The test artefact must not become a backdoor into production.

The offline DR test is conducted at least annually for critical firewalls, and after every major configuration change that touches the runbook (interface renumbering, IPsec topology change, additional plugins).

Read-only / SafeDR test — backup and copy— The time output records the elapsed seconds for the backup creation and the off-host copy. The measurements feed back into the runbook's time estimates. A backup that takes 3 seconds and a copy that takes 2 seconds set the baseline for step 8 of the runbook.

$ time configctl backup create plain && time scp /conf/backup/config-*.xml dr-testbench:/tmp/

real    0m3.214s
Backup created: /conf/backup/config-2026-08-15-1438.xml
real    0m1.892s
config-2026-08-15-1438.xml           100%  384KB   1.8MB/s   00:00

Illustrative output

The lessons-learned cycle

A DR that completes without a lessons-learned document is not a DR; it is a fire drill. The lessons-learned cycle closes the loop:

Timeline. Document every step with timestamps. The timeline reveals which steps took longer than expected and which were skipped.
Root cause. Why did the disaster happen? Was the failure preventable?
Runbook gaps. Which steps were unclear? Which were missing? Which required improvisation?
Backup verification. Did the backup work as expected? Were there restore errors? Were credentials intact?
Communication. Who was notified? When? What was the escalation path?
Improvements. Specific changes to the runbook, the backup cadence, the hardware staging, the rekeying schedule.

The lessons-learned document is filed, reviewed by the team, and turned into action items with owners and dates. The next DR test exercises the updated runbook. The cycle continues.

Why DR is the moment that exposes undocumented configuration

The operator who has never had to restore a firewall under pressure discovers, at 03:00 with the production network down, that the firewall held undocumented configuration: a cron job that pushes logs to an internal syslog server, a custom IPsec proposal negotiated with a specific vendor, an alias that lists every subnet in the corporate VPN. None of these were in the runbook. None were in the backup verification. All of them surface in the DR.

The mitigation is documentation discipline during normal operation, not heroic effort during the DR. Every change window updates the runbook. Every manual intervention becomes a ticket that becomes a configuration entry. The DR runbook is the accumulation of every previous change, kept current.

Summary

The DR runbook is a documented procedure with explicit checkpoints, not improvisation.
Rekeying is mandatory when the previous appliance’s secrecy is in doubt.
The offline DR test exercises the runbook without touching production.
Lessons learned turn each DR into an improved runbook for the next incident.
The cold spare at the site is the difference between a 4-hour DR and a 24-hour DR.