Rollback strategy — when the upgrade fails, the firmware is broken, or the plugin is incompatible

A rollback is the operator’s safety net. Every upgrade carries the risk of failure; the rollback is the path back to a known-good state. The operator who reaches the rollback decision without a pre-planned strategy improvises under pressure, and improvisation under pressure is where the secondary incidents happen.

This lesson covers the rollback decision tree, the boot-environment rollback procedure, the configuration restore path, the partial-rollback technique for surgical fixes, and the post-rollback verification that proves the firewall is in a known-good state.

The rollback decision tree

Three failure modes drive three different rollback paths. The operator identifies the failure mode, then chooses the path.

The kernel/firmware failure is the most severe: the firewall may not boot at all. The boot-environment rollback is the only viable path; the configuration is preserved inside the boot environment, so the rollback restores both the firmware and the configuration in one step.

The configuration corruption failure is rarer: the firmware is fine but a rule, an alias, or a certificate change broke the firewall’s logic. The boot-environment rollback would rewind too much; a targeted configuration restore is the right path.

The single change failure is the most common and the easiest to fix. The change that broke things is identified, the pre-change backup is restored, and the change is rewritten correctly.

Boot-environment rollback procedure

The boot-environment rollback is performed from the boot menu. The operator needs console access (physical keyboard and monitor, IPMI, or virtual console).

The procedure:

1. Reboot the firewall. If the firewall is hung, force a power cycle via IPMI or pull the power.
2. Watch the boot menu. The boot menu appears for 3-5 seconds. Press the appropriate key (usually 2) for “Boot previous”.
3. Confirm the previous boot environment boots. The kernel messages confirm which BE is active.
4. Log in. The console prompt appears. The credentials are from the previous BE.
5. Verify. Run the post-rollback verification checklist.
6. Confirm production traffic. Watch the monitoring or poll a known-good external address from a host behind the firewall.
7. Document. Open a post-rollback ticket. Note the failure mode that triggered the rollback.

$ beadm list && beadm activate 25.7.2_1 && reboot

BE                  Active Mount   Space  Created
default             N       /      2.1G  2026-08-10 11:32
25.7.2_1            -       -      1.8G  2026-07-01 14:22
Activated successfully.

Configuration restore rollback

When the firmware is fine but the configuration is broken, the boot-environment rollback is overkill. The configuration restore reverts the XML while keeping the current firmware.

The procedure:

1. Locate the pre-change backup. The backup was taken before the change that broke things. The version tag (e.g. pre-CHG-1043) makes the right backup identifiable.
2. Apply the backup. GUI: System → Configuration → Backups → Restore. API: POST /api/core/backup/restore. Shell: cp /conf/backups/<file> /conf/config.xml && reboot.
3. Verify. The post-restore verification checklist (interfaces, routing, rules, NAT, services, GUI, SSH).
4. Confirm production traffic. The behaviour that was broken before the change should be restored.

The configuration restore preserves the firmware. A subsequent firmware rollback can still be performed if needed. The two rollbacks are independent: configuration is one artefact, firmware is another.

Partial rollback

A partial rollback is the surgical fix. One section of the configuration is reverted; the rest is preserved.

The partial rollback is most useful when:

• The change was a single rule that broke connectivity, and the rest of the configuration is correct.
• The change was a plugin configuration update that misbehaves, and the rest of the firewall is fine.
• The change was a certificate renewal that broke a service, and only the certificate needs to be reverted.

The technique:

1. Identify the section. The broken section is in the live XML. The pre-change section is in the backup XML.
2. Extract the pre-change section from the backup. Use xmllint --xpath to extract just the section.
3. Replace the section in the live XML. Either through the GUI (for sections that have a GUI editor) or through the API (for sections that don’t).
4. Validate. xmllint for well-formedness. pfctl -s rules | grep <reference> for references that are wired in.
5. Verify. Confirm the broken behaviour is fixed and nothing else regressed.

The partial rollback requires XML fluency. The operator who has never read the OPNsense XML schema is not the operator who should be performing a partial rollback at 03:00.

Post-rollback verification

The rollback is not complete when the firewall is back up. The rollback is complete when the post-rollback verification checklist passes. The checklist is the same one used after a planned upgrade or a restore — interfaces, routing, rules, NAT, services, GUI, SSH — plus the failure-mode-specific checks:

The verification produces evidence. The evidence goes into the post-rollback ticket. The post-rollback ticket is the artefact that closes the incident.

The post-rollback review

A rollback that is not reviewed is a workaround. The post-rollback review asks:

1. What was the failure mode? Why did the upgrade or change fail?
2. What was the rollback target? Which BE or backup was used?
3. Was the rollback target itself validated? A backup that has never been drill-tested is suspect.
4. What is the follow-up? Is the new firmware retried after a fix? Is the change rewritten? Is the plugin replaced?
5. What is the runbook update? Does the runbook need to capture this failure mode for next time?

The post-rollback review is the mechanism by which the operator learns from the rollback. The next upgrade benefits from the lessons; the runbook becomes more accurate; the team becomes more confident in the rollback path because it has been exercised.

The rollback discipline summary

• Plan the rollback before the upgrade, not after.
• Document the boot environment name and the backup file name in the change ticket.
• Choose the rollback path by failure mode: BE for kernel/firmware, configuration restore for configuration, partial rollback for single change.
• Verify the rollback with the same checklist as the upgrade.
• Review the rollback. Update the runbook. Plan the retry.

Summary

• Three rollback paths: boot environment, configuration restore, partial rollback.
• Failure mode drives the choice.
• The boot-environment rollback is the fastest for kernel/firmware failure.
• The configuration restore is the right path for configuration-only failure.
• The partial rollback is the surgical fix for single changes.
• Verify every rollback. Review every rollback. Update the runbook.