Reference architecture — Internet edge

A reference architecture is not a template. A template tells you where to put things; a reference architecture tells you why they go there, what each choice costs, and what failure mode the architecture creates by ignoring something else.

This lesson walks through an Internet-edge OPNsense estate — two firewalls, two ISPs, segmented internal network, public services in a DMZ, edge IDS — and names the trade-offs of every decision.

The edge topology

                    ISP-A            ISP-B
                       |                |
                  +----+----+      +----+----+
                  |         |      |         |
                WAN-A      WAN-B   WAN-A     WAN-B
                  |         |      |         |
              +---+---------+------+---+-----+---+
              |   OPNsense-A          OPNsense-B |
              |   (MASTER)            (BACKUP)   |
              |                                     |
              |   CARP VIPs:                         |
              |     WAN-A: 203.0.113.2/29           |
              |     WAN-B: 198.51.100.2/29          |
              |     LAN:   10.0.0.1/24              |
              |     DMZ:   10.10.0.1/24             |
              |                                     |
              +-------+---------------------+-------+
                      |                     |
                Sync interface         Management
                (10.99.0.0/30)         (10.100.0.0/24)
                      |                     |
                      +---- dedicated ------+

Two physical firewalls. Two ISP uplinks. A dedicated sync segment for pfsync and CARP. A separate management VLAN.

The edge pair sees:

• Public traffic from the Internet
• Outbound traffic from the internal network
• DMZ traffic to/from public services (web, mail, DNS)

The edge pair does not see:

• East-west traffic between internal segments (handled by an internal routing tier — see Part XLVIII lessons 285-288)

Component-by-component justification

Two firewalls, not one. A single firewall is a single point of failure. The application of “high availability” to a single firewall is a misnomer. Two firewalls with state synchronisation are the minimum viable HA.

Two ISPs, not one. A single ISP is a single point of failure. Two ISPs with gateway monitoring and policy routing allow a graceful failover when one link degrades. The cost is upstream BGP or static routing complexity, and the requirement to source-route return traffic correctly.

Dedicated sync interface. pfsync state updates and the XMLRPC configuration sync are sent over a private segment the two firewalls own, so unauthenticated replication traffic is never exposed to a production segment. It does not move CARP: each VIP advertises on the interface it is bound to, so the LAN and WAN switches still carry 224.0.0.18 and still have to be audited for multicast handling. Cost: two extra NICs per firewall, plus a small dedicated switch (or cross-connect cable).

Separate management VLAN. The GUI is reachable only from the management VLAN, never from the LAN or DMZ. Cost: another VLAN to configure and document; benefit: the firewall management plane is isolated from data-plane incidents.

DMZ for public services. Web, mail, and authoritative DNS live on a separate segment from the corporate LAN. The firewall blocks any DMZ-initiated traffic to the LAN. Cost: another VLAN and more firewall rules; benefit: a compromised DMZ host cannot reach the corporate network directly.

Failure domains the architecture creates

The architecture does not protect against: a vulnerability in OPNsense itself, a stolen administrator credential, a global ISP outage, a power outage on the firewall rack. Those require a different layer of protection (patching, MFA, third ISP, UPS, generator).

Reading the architecture

When you are asked “why is the firewall configured this way?” the answer should be traceable to the reference architecture. “DMZ hosts cannot reach the LAN” — because the architecture separates DMZ from LAN at the firewall, and the firewall rule blocks DMZ-initiated traffic. “CARP failover takes 3 seconds” — because the CARP advertisement frequency is 1 second and the failover decision is made after missing 3 advertisements.

The architecture is the contract between operator and environment. Every configuration choice should trace back to it. If a configuration choice has no architectural reason, it should either be removed or documented as a deliberate exception.