Shared IP and virtual IPs — the abstraction that lets two firewalls look like one

Two firewalls cannot simply have the same IP address on the same network — the moment both are active, ARP becomes confused, the upstream switch’s MAC table becomes inconsistent, and the network breaks. The solution is a virtual IP (VIP): an address that belongs to one firewall at a time, with an arbitration protocol that decides which firewall holds it. This lesson covers what a VIP actually is, why the abstraction is necessary, the IP alias layer underneath, and the failure modes of a misconfigured VIP.

Why two devices cannot share an IP

IP addressing assumes a 1:1 mapping between an IP address and a host. The Address Resolution Protocol (ARP for IPv4, NDP for IPv6) maintains that mapping — a host tells the network “I am 192.0.2.10, my MAC address is aa:bb:cc:11:22:33”. Every other host on the segment caches that mapping and sends traffic for 192.0.2.10 to that MAC.

When two devices have the same IP:

1. ARP conflict. Both devices answer ARP queries for the IP. Different hosts on the segment cache different MAC addresses. Some traffic goes to device A; some to device B. The application sees intermittent failures — some connections work, some do not, depending on which device received the packet.
2. Switch confusion. The upstream switch learns MAC addresses on ports. If device A and device B have different MAC addresses, the switch sees two different MACs claiming the same IP (via ARP). The switch’s MAC table becomes inconsistent, and packets for the IP bounce between the two ports.
3. Duplicate address detection. IPv6 has a built-in mechanism — Duplicate Address Detection (DAD) — that detects when two devices claim the same address and refuses to bring the interface up. IPv4 has no equivalent; the conflict is silent.

The result: even if two devices are configured with the same IP, the network will not route traffic to them predictably. The shared-IP pattern needs a mechanism that prevents both devices from being “active” simultaneously.

The IP alias layer

The virtual IP in OPNsense is implemented as an IP alias — an additional address configured on a physical interface and tagged with a VHID. The alias answers on its own MAC address (a virtual MAC, not the physical NIC’s MAC), carries its own IP address (the VIP), and has its own CARP state.

The relationship:

Physical interface:  igb1  (MAC aa:bb:cc:11:22:33, IP 192.0.2.1)
  └── IP alias:       vip_wan (MAC 00:00:5e:00:01:01, IP 192.0.2.10) [CARP VHID 1]
Physical interface:  igb0  (MAC aa:bb:cc:11:22:44, IP 198.51.100.1)
  └── IP alias:       vip_lan (MAC 00:00:5e:00:01:02, IP 198.51.100.10) [CARP VHID 2]

The virtual MAC follows the pattern 00:00:5e:00:01:XX where XX is the VHID (Virtual Host Identifier) — the CARP group identifier. The virtual MAC is identical on both firewalls for the same VHID; that is what makes the failover transparent to the upstream switch. When the active node fails, the backup takes over the VHID, starts answering ARP for the VIP, and the upstream switch learns the same MAC on a different port — the failover looks like a port move, not an address change.

The IP alias is what OPNsense shows in the GUI (Interfaces → Virtual IPs → Settings). The configuration includes the interface, the address, the VHID, the password (used for CARP authentication), and the advertising base/skew (covered in lesson 158). The operator creates one VIP per service IP that needs to be redundant.

$ ifconfig igb1

igb1: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
      ether aa:bb:cc:11:22:33
      inet 192.0.2.1 netmask 0xffffff00 broadcast 192.0.2.255
      inet 192.0.2.10 netmask 0xffffff00 broadcast 192.0.2.255 vhid 1
      media: Ethernet autoselect (1000baseT <full-duplex>)
      status: active
      carp: MASTER vhid 1 advbase 1 advskew 0

The role of the arbitration protocol

The IP alias is the data structure; the arbitration protocol is the mechanism that decides which node holds it. Without arbitration, both nodes think they are master and the shared-IP problem returns.

OPNsense uses CARP (Common Address Redundancy Protocol), developed by the OpenBSD project. CARP is functionally similar to VRRP (Virtual Router Redundancy Protocol, RFC 5798) and HSRP (Hot Standby Router Protocol, Cisco proprietary), but is open and patent-unencumbered. The three protocols solve the same problem — shared-IP HA — with different packet formats and different default behaviours.

CARP works as follows:

1. Each node has a VHID and a skew. The VHID identifies the CARP group (the VIP). The skew determines priority — lower skew wins. A skew of 0 means “highest priority”.
2. Nodes advertise periodically. The master sends CARP advertisements every advbase seconds (default 1). The backup listens.
3. The backup promotes on silence. If the backup stops hearing advertisements for three intervals, it assumes the master is dead and promotes itself to master.
4. Both nodes use the same virtual MAC. The master and backup both know the VIP’s MAC (derived from the VHID). When the backup promotes, it does not need to ARP — it just starts answering ARP queries for the VIP, and the upstream switch learns the same MAC on a different port.

The arbitration is decentralised — there is no central arbiter, no shared database. Each node decides locally whether it should be master or backup based on the advertisements it sees. The decision is fast (failure detection in 3 seconds by default) and the failover is fast (the backup starts forwarding as soon as it promotes).

Production failure modes

The recurring failures of misconfigured VIPs:

Skew mismatch. Both nodes have advskew 0 (or both have the same skew). Neither node believes the other is the backup; both transition to master. The network sees duplicate ARP replies. The fix: explicit skew per node (the primary has advskew 0, the secondary has advskew 100 or 200).

Missing VHID. A VIP without a VHID is an IP alias but not part of a CARP group. The two nodes both believe they are master. Same outcome as skew mismatch. The fix: configure the VHID for every VIP that should be part of HA.

Password mismatch. The CARP advertisements include an authentication hash. If the two nodes have different passwords, they do not recognise each other’s advertisements. Each node believes the other is dead and both transition to master. The fix: identical passwords on both nodes.

Skew inversion. The operator intended node A to be master (skew 0) and node B to be backup (skew 100), but configured it the other way. The network now fails over to node B when node A fails — but node B has not been tested, and the operator’s runbook assumes node A is master. The fix: confirm the skew values match the documented roles.

Summary

• Two devices cannot simply share an IP address — the network cannot resolve the duplicate MAC/address mapping.
• A virtual IP is an IP alias with a virtual MAC, bound to a physical interface, with a CARP VHID that identifies the HA group.
• CARP (or VRRP, or HSRP) is the arbitration protocol that decides which node holds the VIP at any moment.
• The virtual MAC is derived from the VHID, so failover is a port move at the switch, not a new MAC — fast and transparent.
• Failure modes: skew mismatch, missing VHID, password mismatch, skew inversion. All produce duplicate-address behaviour.