Reference architecture — Operating the estate · OPNsense

The reference architecture is a starting point. The harder problem is operating it day after day: changing rules without breaking the estate, upgrading the firmware without downtime, responding to incidents at 03:00, and rotating the credentials and certificates before they expire.

This lesson summarises the operational model. The detailed procedures are runbooks, listed in the toolkit of this course.

Change windows

The reference architecture changes during scheduled change windows, defined by:

Day: weekday (Tue–Thu preferred). Weekends and Mondays/Fridays are reserved for incident response.
Time: low-traffic hours for the business.
Notice: 5 business days for non-emergency changes; same-day only for emergency changes with explicit approval.
Communication: change ticket in the change management system with description, impact, rollback plan, and validation steps.

Every change is preceded by a pre-change checklist (see the checklist collection of this course) and followed by a post-change validation. The change ticket is the audit trail.

Upgrade procedure for an HA pair

The reference architecture upgrades an HA pair using the “secondary first, then primary” pattern:

1. Backup both nodes' configuration (push to off-device storage).
2. Validate the backup can be restored to a disposable VM.
3. Upgrade the BACKUP node first.
4. Validate the BACKUP node boots and is functional.
5. Validate CARP: BACKUP should still show BACKUP (not promoted).
6. Validate pfsync: state replication should be healthy.
7. Run the post-upgrade validation checklist on the BACKUP node.
8. Fail over: demote the MASTER node (CARP demote).
9. Wait for CARP to switch VIPs to the (now-upgraded) BACKUP node.
10. Validate traffic flows through the new MASTER.
11. Upgrade the original MASTER (now BACKUP) node.
12. Restore CARP state.
13. Validate both nodes are healthy, pfsync is replicating, and traffic flows.
14. Run the post-upgrade validation checklist on both nodes.

The procedure assumes the configuration is compatible across the upgrade (no breaking changes). The release notes must be read before the change window.

Incident response playbook

Incidents in the reference architecture fall into four classes:

Class	Description	Response
Connectivity loss	Hosts cannot reach the Internet or specific destinations	Follow the “no Internet” runbook; identify which WAN, which VLAN, which rule
Performance degradation	Latency or throughput has changed	Check state table, CPU, interfaces; identify the saturated resource
Security event	Suspicious traffic, failed authentication, IDS alerts	Follow the security incident runbook; preserve evidence; isolate the affected host
Configuration drift	Running configuration differs from source-of-truth	Run drift detection; identify the cause; restore from source-of-truth

Every class has a runbook. The runbooks list the diagnostic commands, the typical root causes, the remediation steps, and the validation. The first responder follows the runbook, not memory.

Runbook set every operator must know

The reference architecture assumes operators know:

Deploy new OPNsense firewall. Initial install from ISO to live service.
Configure new interface. Adding a VLAN or OPT interface to an existing firewall.
Create production firewall rule. Add a rule with the correct source/destination/direction and validation.
Troubleshoot blocked traffic. Identify ingress interface, check rule, check state, check NAT, check return route.
Troubleshoot unexpected allowed traffic. Find the rule that permits it; tighten or remove.
Troubleshoot port forward. Identify DNAT, associated rule, packet path.
Add VLAN. Create VLAN, assign interface, configure addressing, DHCP, rules.
Troubleshoot VLAN connectivity. Verify trunk, interface assignment, addressing, DHCP, rule.
Configure multi-WAN. Add gateway, gateway group, NAT rules, policy routing.
Troubleshoot WAN failover. Verify gateway monitoring, NAT agreement, asymmetric routing.
Configure WireGuard tunnel. Set up instance, peer, Allowed IPs, rules.
Troubleshoot WireGuard. Check handshake, Allowed IPs, rules, NAT.
Build OPNsense HA pair. Set up CARP, pfsync, config sync.
Troubleshoot CARP. Check advertisements, skew, switch multicast.
Troubleshoot pfsync. Check sync interface, state replication.
Perform HA maintenance. Demote, upgrade, fail over, return.
Backup OPNsense configuration. Pull, encrypt, store.
Restore OPNsense. Restore from backup, verify.
Recover failed firewall appliance. Reinstall, restore, validate.
Upgrade OPNsense HA pair. The procedure above.

These runbooks live in the runbook collection of this course. They are the operator’s safety net.

What operators must NOT improvise

The operator is not authorised to:

Edit /conf/backup/.../filter.conf directly.
Modify generated configuration files in /var/etc/.
Restart services outside the GUI/API without a runbook.
Disable HA failover to “fix” a problem.
Disable the anti-lockout rule.

If the GUI or API cannot solve the problem, the operator opens a console, reads the diagnostic output, and either follows a runbook for console recovery or escalates.