Safe rule change procedure — the discipline that prevents lockouts

A rule change is the most common source of firewall lockouts. The change itself is rarely wrong; the procedure around the change is what produces the lockout. The operator who applies a rule change without a backup, without a peer review, without a second-session test, and without a rollback procedure is one mistake away from a console-required recovery.

This lesson covers the safe rule change procedure — backup, peer review, second-session test, verification, and rollback — the patterns that turn a risky change into a reversible one.

The safe rule change procedure

The procedure has seven steps, executed in order:

1. Identify the change. What rule is being added, modified, or removed. What is the expected effect. What is the worst-case effect.
2. Backup the configuration. Take a backup before any change. The backup is the rollback path.
3. Peer review. A second operator reviews the change before it is applied. The review catches typos, shadowed rules, and unintended effects.
4. Verify a tested recovery path. Console access works. Break-glass SSH works. The backup is restorable.
5. Apply the change from one session. Open the GUI or SSH in from one workstation. Apply the change.
6. Test from a second session. Open a second SSH or GUI session from a different workstation or different source IP. Verify the change took effect. Verify the existing rules still work.
7. Document the change. Update the change log with what was changed, when, why, and the rollback procedure.

Each step produces a checkpoint. If any step fails, the operator stops and re-evaluates. The procedure takes longer than “click apply and hope” but produces changes that can be rolled back if they go wrong.

Step 1: identify the change

Before any GUI interaction, write down the change in plain language:

• What rule is being added, modified, or removed.
• Why the change is being made.
• What the expected effect is.
• What the worst-case effect is.

The worst-case effect is the question that catches most lockouts. “If this rule is wrong, what breaks?” If the answer involves the GUI, SSH, or a path the operator uses to reach the firewall, the change is high-risk and requires additional precautions.

A typical change description:

Change: Add rule to LAN interface to block 192.0.2.0/24 from reaching 198.51.100.50:443.
Why: 198.51.100.50 was compromised; block egress to that destination.
Expected: Hosts on 192.0.2.0/24 cannot reach 198.51.100.50:443. Other destinations unaffected.
Worst case: Rule is wrong (typo in destination, wrong source) and blocks all egress from the LAN. Operator cannot reach the GUI.
Recovery: Backup; rule removal; verification.
Risk: High — affects egress on LAN, which is the path to the GUI.

The risk assessment drives the rest of the procedure. High-risk changes require peer review, two-session test, and rollback plan verified before applying.

Step 2: backup the configuration

A configuration backup is the rollback path. Take it before any change.

OPNsense provides backups via:

• GUI: System → Configuration → Backups. Download the backup file.
• API: POST /api/core/backup/download returns the configuration XML.
• Shell: configctl config backup creates a backup in /conf/backup/.

The backup should be stored off the firewall (downloaded to a workstation, copied to a configuration management system, or stored in a backup repository). A backup stored on the firewall is recoverable only if the firewall is reachable — which is the very case the operator may be trying to recover from.

$ configctl config backup && ls -la /conf/backup/

Writing backup to /conf/backup/...
-rw-------  1 root  wheel  14523 Aug 14 02:14 config-2026-08-14-0214.xml
-rw-------  1 root  wheel  14520 Aug 13 14:22 config-2026-08-13-1422.xml

Step 3: peer review

A second operator reviews the change. The peer review catches:

• Typos in source/destination/port.
• Rules that shadow existing rules.
• Rules that do not match the stated intent.
• Missing return rules.
• Unintended effect on other interfaces.

The peer reviewer should be someone with production OPNsense experience, ideally someone who has not seen the change request before. The reviewer reads the change description and the rule as it will be applied (in the GUI or in the compiled ruleset via pfctl -s rules).

The peer reviewer signs off on the change in the change ticket. The sign-off is not “I trust the operator”; it is “I have read the change and verified the rule matches the intent”. The sign-off is part of the audit trail.

If the change is low-risk and no peer is available, the operator can self-review by reading the rule aloud (literally — read the rule’s source, destination, port, and action out loud) and confirming it matches the intent. The self-review is weaker than a peer review but better than no review.

Step 4: verify a tested recovery path

Before applying the change, verify:

• Console access works. Serial console, BMC SOL, or KVM over IP. Test with cu or equivalent from the operator’s workstation.
• Break-glass SSH works. Local account with a known password or SSH key. Log in from the operator’s workstation, confirm the shell is usable.
• Backup is restorable. The backup file is downloaded off the firewall and is readable. Test by extracting the configuration XML and confirming it has the expected sections.

If any of these three paths is not verified, the operator must verify it before applying the change. The lockout risk is the change’s effect on the path the operator uses to reach the firewall; the recovery path is the path that does not depend on the change.

Step 5: apply the change from one session

Open the GUI or SSH in from one workstation. Apply the change. PF reloads the ruleset on Apply.

The change is now live. The operator’s first session may still work (because the change does not affect the first session’s traffic) or may be broken (because the change affects the first session’s source). Either way, the operator must not close the first session until step 6 is complete.

Step 6: test from a second session

Open a second SSH or GUI session from a different workstation or different source IP. Verify:

• The new rule took effect. The flow the rule was supposed to allow (or block) behaves as expected.
• The existing rules still work. Other traffic patterns are unaffected.
• The operator’s path to the firewall still works. The first session can be opened in a new connection.

If the second session fails, the operator uses the first session to roll back the change. The first session is the escape route.

A second session on the same source IP is not a true test. The change might be working for the first source IP but blocking a different source IP. The disciplined test uses a different source IP (a different workstation, a different VLAN, a different IP on the same host).

Step 7: document the change

After the change is verified, document:

• What was changed (the rule, the interface, the action).
• When it was changed (timestamp and timezone).
• Why it was changed (the incident, the change ticket, the request).
• The peer reviewer who signed off.
• The verification evidence (the second-session test result, the log line, the state entry).

The documentation is the audit trail. The next operator who looks at the rule will know why it exists. The next incident investigation will have the change history.

Recovery from a misapplied rule change

If the change locks the operator out, the recovery is:

1. Soft lockout (SSH still works). SSH in from the operator’s workstation. Remove or fix the rule. Verify the GUI is reachable.
2. Hard lockout (SSH does not work). Use the console (serial, BMC SOL, or KVM over IP). Log in. Remove or fix the rule. Verify the GUI is reachable.
3. Last resort (no console access). Restore the backup via the OPNsense recovery mechanism (the recovery console option in the boot menu, or factory reset).

The recovery procedure should be documented before the change is applied, not after the lockout occurs. The lockout is not the time to read the manual.

UnderTheHood: why the change takes effect immediately

OPNsense’s filter generator reads the GUI configuration and produces the compiled PF ruleset. When the operator clicks Apply, the generator runs and produces a new ruleset. The new ruleset is loaded into the kernel with pfctl -f. The new ruleset replaces the old ruleset atomically.

The atomicity means the operator does not see a partial state — either the new ruleset is loaded or the old one is retained. The downside is that the new ruleset takes effect immediately, with no rollback timer. If the operator applies a change and the change is wrong, the only rollback is to apply another change (removing the rule).

The atomicity is also why the second-session test is critical. The first session’s state may be preserved (because the matching rule is still present in the new ruleset), but new connections are evaluated against the new ruleset. The operator’s first session is alive because it was created under the old ruleset; new sessions will be evaluated against the new ruleset.

Summary

• The safe rule change procedure: identify, backup, peer review, verify recovery path, apply, two-session test, document.
• The two-session test is the most important step. Verify from a different source IP, not the same workstation.
• A high-risk change (affects GUI/SSH path) requires additional precautions: peer review, tested recovery path, rollback procedure documented.
• Recovery: soft lockout via SSH, hard lockout via console, last resort via boot menu’s revert-to-previous-config.