HA reference topology — production-grade OPNsense pair

The HA reference topology is the architecture the rest of this part assumes. It is what an experienced network operator would deploy for a small-to-medium business or a datacentre edge. The choice of every component is justified by a failure it covers or an operational property it provides.

The topology

                    ISP-A            ISP-B
                       |                |
                  +----+----+      +----+----+
                  |         |      |         |
                WAN-A      WAN-B   WAN-A     WAN-B
                  |         |      |         |
              +---+---------+------+---+-----+---+
              |   OPNsense-A          OPNsense-B |
              |   (MASTER)            (BACKUP)   |
              |                                     |
              |   Interfaces:                        |
              |     WAN-A  ix0  203.0.113.2/29      |
              |     WAN-B  ix1  198.51.100.2/29     |
              |     LAN    ix2  (trunk, no IP)     |
              |     SYNC   ix3  10.99.0.1/30        |
              |     MGMT   ix4  10.100.0.1/24       |
              |                                     |
              +-------+---------------------+-------+
                      |                     |
              +---+---+--+            +----+---+---+
              | LAN trunk|            | MGMT switch|
              +---+---+--+            +----+---+---+
                  |   |                    |   |
              +---+   +---+           +---+   +---+
              |          |            |         |
            Servers    Users      Operators  Tools

Component-by-component justification

Two physical firewalls (OPNsense-A, OPNsense-B). A single firewall is a single point of failure. Two are the minimum for HA. Each firewall is a separate physical host with separate power supplies, separate disks, separate network interfaces.

Two ISP uplinks (ISP-A, ISP-B). A single ISP is a single point of failure. Two ISPs with gateway monitoring and policy routing allow graceful failover when one link degrades. The cost is upstream BGP or static routing complexity, and the requirement to source-route return traffic correctly.

Five physical interfaces per firewall. This is more than the typical “two-NIC” minimum, and each NIC is deliberate:

• ix0 — WAN-A. Dedicated NIC so WAN-A traffic does not share interrupt queues with other traffic.
• ix1 — WAN-B. Same reasoning.
• ix2 — LAN trunk. 802.1Q trunk carrying the production VLANs (Users, Servers, DMZ, Guest, IoT, Backup, Voice). The NIC itself has no IP; the VLAN interfaces do.
• ix3 — Sync. Dedicated NIC for pfsync and config sync to the peer. This is the most important isolation: no other host transmits on this segment.
• ix4 — Management. Dedicated NIC for the operator network. The GUI is reachable only from this segment.

CARP VIPs. Each production subnet has a CARP VIP that floats between the two firewalls. The MASTER owns the VIP and answers traffic; the BACKUP takes over if the MASTER fails.

Dedicated sync interface. pfsync and config sync run on the ix3 NIC. The two firewalls’ ix3 ports are connected by a crossover cable or a private VLAN on a dedicated switch. The pfsync traffic is unauthenticated; the isolation is the security.

Management VLAN. The operator workstations, monitoring stack, and jump hosts are on the management VLAN. The firewall GUI is reachable only from this VLAN. The production data VLANs cannot reach the GUI.

No shared broadcast domains between sync and production. The sync interface is on a separate switch (or a private VLAN). Production traffic cannot reach the sync interface. This is required for both performance (sync traffic does not contend with data traffic) and security (no host on a production VLAN can inject pfsync packets).

What this architecture covers

What this architecture does not cover

• A vulnerability in OPNsense itself (mitigated by patching)
• A stolen root credential on the firewall host
• A regional ISP outage affecting both ISPs
• A power outage on the rack (mitigated by UPS + generator)