Rollback strategies — how to reverse a change that turned out to be wrong

A change that cannot be rolled back is not a change — it is a leap. The operator who applies a change without a tested rollback is one mistake away from a recovery that takes hours instead of minutes. The rollback is not improvisation; it is a procedure written before the change, tested before the change, and executed when the change turns out to be wrong.

This lesson covers rollback strategies: the four rollback paths the operator has, the decision tree for choosing the right one, the design of a rollback procedure, and the rollback drill that proves the procedure works.

The four rollback paths

The operator has four rollback paths, ordered from least disruptive to most disruptive:

1. In-place rollback. Reverse the change with another change. Remove the rule that was added. Re-add the rule that was removed. Restore the alias that was deleted.
2. Backup restore. Replace the running configuration with a previously-taken backup. The backup is the rollback path for changes that cannot be reversed in place — multi-rule refactors, complex NAT changes, configuration-wide updates.
3. Boot-environment rollback. Reboot into the previous boot environment. The previous firmware, the previous kernel, the previous configuration snapshot. The rollback path for upgrades that broke the system.
4. Vendor recovery. Factory reset via the boot menu. The last resort; loses all configuration.

Each path has a different cost, a different disruption, and a different recovery time. The operator chooses based on the change and the failure.

$ configctl config backup && cp /conf/backup/config-*.xml ~/backups/ && ls -la /conf/backup/

Writing backup to /conf/backup/...
-rw-------  1 root  wheel  14523 Aug 14 02:14 config-2026-08-14-0214.xml
drw-------  2 root  wheel    512 Aug 14 02:14 backups

The rollback decision tree

The decision tree is short:

1. Can the change be reversed with a single opposite change? If yes, in-place rollback. The operator applies the opposite change; the ruleset reloads; the original state is restored.
2. Is the GUI or SSH reachable? If yes, backup restore via the GUI or API. The operator uploads the backup; OPNsense restores the configuration; the firewall reloads.
3. Is the GUI or SSH unreachable but the console is reachable? If yes, console restore from a backup on a USB stick, or boot-environment rollback. The operator connects to the console, mounts the backup, restores it.
4. Is the console also unreachable? Then the rollback is to the vendor recovery path: physical access to the appliance, factory reset via the boot menu, reconfigure from scratch.

The decision tree assumes one thing: the operator has a backup stored off the firewall that is newer than the change. If the operator does not have a backup, the rollback options are limited to in-place (which requires the opposite change to be known) and boot-environment rollback (which requires the previous boot environment to still be valid).

Designing the rollback procedure

The rollback procedure is written before the change is applied. The procedure has six parts:

1. Trigger. What observation tells the operator to roll back. “No traffic from 192.0.2.0/24 reaches 8.8.8.8:53 after the change” — the trigger is the absence of expected behaviour.
2. Decision deadline. How long the operator waits after the trigger before deciding to roll back. For stateful changes, 60 seconds. For configuration changes, 5 minutes. The deadline is documented; the operator commits to the deadline.
3. Path. Which of the four rollback paths the operator will use.
4. Steps. The exact commands, in order, to execute the rollback.
5. Verification. How the operator confirms the rollback worked. The same evidence the change was supposed to produce, but inverted.
6. Notification. Who the operator tells. The peer reviewer, the on-call operator, the change-owner.

A procedure that does not have all six parts is incomplete. The operator who writes a procedure with steps but no trigger is the operator who does not know when to invoke the procedure. The operator who writes steps and trigger but no deadline is the operator who waits too long.

The rollback drill

The rollback procedure that is not tested does not exist. The operator writes the procedure before the change; the operator tests the procedure before the change. The test is the rollback drill.

The rollback drill:

1. Take a backup of the current configuration.
2. Apply a benign change (an alias entry, a harmless rule on an OPT interface, anything that can be reversed without affecting production).
3. Verify the change took effect.
4. Execute the rollback procedure.
5. Verify the rollback worked (the ruleset matches the pre-change state).
6. Document the drill result. Time to detect, time to decide, time to execute, time to verify.

The drill produces timing data. Time to detect (how long from applying the change to deciding to roll back). Time to decide (how long from detecting to choosing the rollback path). Time to execute (how long the rollback takes). Time to verify (how long to confirm the rollback worked).

The timing data feeds back into the change procedure. A drill that takes 10 minutes is fine; a drill that takes 60 minutes is too slow for a high-risk change.

Verification after rollback

The rollback is not done when the rollback completes; it is done when the verification confirms the system is back to its pre-change state. Verification has three parts:

1. Configuration verification. pfctl -s rules matches the pre-change evidence. pfctl -s nat matches the pre-change evidence. The configuration diff against the pre-change backup is empty.
2. State verification. For stateful changes, the state table is within the expected range. The new connections match the pre-change patterns.
3. Behaviour verification. The traffic patterns that existed before the change are restored. The flows the change affected behave as they did before the change.

A rollback that passes configuration but fails state is a partial rollback. The operator continues the rollback (in-place, this time) to address the state.

Summary

• Four rollback paths: in-place, backup restore, boot-environment rollback, vendor recovery. The decision tree chooses based on reachability and the nature of the change.
• The rollback procedure has six parts: trigger, decision deadline, path, steps, verification, notification. All six are written before the change.
• The rollback drill tests the procedure under realistic conditions. The drill produces timing data that feeds back into the change procedure.
• Verification after rollback has three parts: configuration, state, behaviour. All three must pass.