Runbook: Recover a Failed Appliance
1 · Prerequisites
Confirm every item is in place before any state change.
- OPNsense appliance with console or IPMI access
- Backup configuration available on an off-appliance host
- Maintenance window scheduled for the recovery
- Knowledge of the running firmware version and the planned replacement firmware
2 · Pre-checks
Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.
- · Confirm the appliance is actually failing (power, network, software) and not just unreachable from one network
- · Capture the current console output if the appliance is intermittently rebooting
- · Capture the current configuration if the GUI is still reachable
- · If the appliance is in an HA pair, confirm the surviving appliance is healthy and the cluster is not in a split-brain
3 · Procedure
Execute each step in order. Verify the expected output of a step before moving to the next.
- 1From the console or IPMI, power cycle the appliance and capture the full boot output
- 2If the appliance boots, capture a config backup and run the firmware upgrade or revert as appropriate
- 3If the appliance does not boot, reinstall the firmware from the install media while preserving the configuration partition
- 4If the configuration must be rebuilt, restore from the off-appliance backup file and validate the basic connectivity
- 5If the appliance is in an HA pair, return the appliance to the backup role and confirm the cluster is healthy
- 6If the appliance must be replaced, install the firmware on the replacement hardware, restore the configuration, and rejoin the pair
- 7Force a failover on the surviving appliance and confirm the recovered appliance takes over the master role
- 8Update the change record with the failure mode, the recovery steps, and the operator name
4 · Verification
Confirm the procedure actually fixed the problem.
- ✓The appliance is reachable from the management subnet
- ✓The running configuration matches the expected baseline
- ✓A forced failover on the surviving appliance is handled by the recovered appliance
- ✓No new alerts fire during the observation window
5 · Rollback
If verification fails, undo the procedure in reverse order.
- ↶If the recovery fails, leave the appliance offline and let the surviving appliance continue as master
- ↶If the recovered appliance cannot rejoin the HA pair, keep it offline and schedule a rebuild
- ↶If the firmware upgrade fails, restore the previous firmware and configuration
6 · Escalation
When the runbook isn't enough, contact:
- · Engage the platform team if the appliance does not boot after the firmware reinstall
- · Engage the network team if the recovered appliance cannot rejoin the HA pair
- · Engage the security team if the failure was caused by a suspected compromise
Purpose
Return a failed OPNsense appliance to a known-good state, either by repairing the existing installation or by rebuilding the configuration on the same hardware, and rejoin the HA pair if applicable.
When to use this runbook
Use this runbook when an appliance is unreachable, when the firmware is corrupted, when the hardware has been replaced, or when the appliance has been removed from the HA pair and must be brought back into service.
Pre-checks
Before starting the procedure, confirm the prerequisites and pre-checks are met. The structured lists are rendered from the frontmatter by the page layout.
Procedure
Follow the steps from the frontmatter procedure steps. The page layout renders the steps as a checklist with copy-to-clipboard affordances.
Verification
After the procedure, the structured verification items from the frontmatter are rendered as a checklist.
Rollback
If the procedure fails or makes things worse, follow the structured rollback steps from the frontmatter.
Escalation
The structured escalation path is rendered from the frontmatter. Use it if the operator cannot complete the procedure safely.