CARP concepts and VHIDs — the protocol that makes a shared IP work · OPNsense

Two firewalls in HA need to share one IP address. The LAN hosts, the upstream routers, and every Internet destination see a single gateway. They do not know — and must not be able to tell — whether OPNsense-A or OPNsense-B is actually answering. The protocol that lets one of them claim the IP at any moment, and lets the other take over if the first one disappears, is CARP — the Common Address Redundancy Protocol.

This lesson covers what CARP is at the protocol level, what a Virtual Host ID (VHID) identifies, what the base advertisement skew controls, and how the master is elected. The next lessons cover the advertisement packet, the configuration in OPNsense, the multicast behaviour, the failover triggers, and the troubleshooting.

The problem CARP solves

A LAN host has a default gateway configured to one IP — say 192.0.2.1. If the firewall owning that IP fails, the host keeps sending frames to the same MAC it learned from ARP for 192.0.2.1. Until something tells the host that 192.0.2.1 is now somewhere else, the host cannot reach the Internet. The recovery time without a protocol is the time for the host’s ARP entry to age out (minutes on FreeBSD, 20 minutes default) plus the time for the backup to send a gratuitous ARP.

CARP makes the recovery time independent of the host’s ARP timeout. The firewalls coordinate among themselves: one holds the IP, the other is ready, and the secondary takes over and sends a gratuitous ARP within a second of failure. The host re-ARPs the gateway — or accepts the gratuitous ARP — and traffic resumes.

The protocol is small but precise. Each firewall on the shared segment advertises its presence on a multicast group, with a virtual host identifier, a base advertisement skew, and an advertisement interval. The firewall with the lowest effective skew becomes the master; the other becomes the backup. The backup watches for the master’s advertisements and takes over when they stop.

VHID and the virtual IP

CARP groups nodes by a Virtual Host ID — a number from 1 to 255. Each VHID identifies one shared IP (one virtual IP, one VIP) on one segment. Two firewalls in the same VHID on the same network are competing for the same VIP. Two different CARP groups (VHID 1 on LAN, VHID 2 on WAN) are independent — a node can be master on one and backup on the other.

The VHID is what the advertisements carry. When a node advertises on the segment, the packet includes the VHID, the source IP of the node (the physical IP, not the shared IP), the base skew, and the advertisement interval. Other nodes on the segment listen to the multicast and update their view of who is master for each VHID.

The CARP password is a pre-shared secret (up to 20 characters) carried in the advertisement packet. It is not a strong authentication mechanism — it is a typo check. Two firewalls with the same VHID and different passwords will not see each other as peers; both will declare themselves master. The result is the classic split-brain: two masters, both answering ARP for the VIP, and the LAN seeing inconsistent frames.

Read-only / Safeifconfig igb1— CARP state lives on the parent interface — there is no carp0 cloned device to inspect. The second inet line, tagged vhid 1, is the shared IP 192.0.2.1 that this CARP group owns; 192.0.2.2 is the node's own address on the segment. The carp: line shows the state (MASTER), the VHID (1), the base advertisement interval (advbase 1 second), and the base advertisement skew (advskew 0). The backup shows the same VHID with advskew 100 (or higher) and state BACKUP.

$ ifconfig igb1

igb1: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
      ether aa:bb:cc:11:22:33
      inet 192.0.2.2 netmask 0xffffff00 broadcast 192.0.2.255
      inet 192.0.2.1 netmask 0xffffff00 broadcast 192.0.2.255 vhid 1
      media: Ethernet autoselect (1000baseT <full-duplex>)
      status: active
      carp: MASTER vhid 1 advbase 1 advskew 0

Illustrative output

Base advertisement skew and the election

The election is “lowest effective skew wins”. Each node has a base advertisement interval (advbase, default 1 second) and a base advertisement skew (advskew, default 0). The effective advertisement interval is advbase + (advskew / 256) seconds. With advbase=1 and advskew=0, the node advertises every 1.000 seconds. With advbase=1 and advskew=100, the interval is 1.391 seconds, and the node is the “slower” advertiser.

The node with the shorter interval is the master. Two nodes both configured with advskew=0 will tie — both will claim master, and the protocol’s tie-break behaviour may flap. The OPNsense convention is to set the primary to advskew=0 (or advskew=10 for a small preference) and the secondary to advskew=100 or higher. The result is a deterministic, stable election.

The backup watches for the master’s advertisements. If three consecutive advertisements are missed (the default — configurable), the backup declares the master gone, becomes the master itself, and sends a gratuitous ARP on the segment to update the switches’ MAC tables. The total failover time is 3 × advbase plus the gratuitous ARP time — 3 seconds with the default advbase=1.

Why CARP's advertisement interval is computed in 1/256ths of a second

The expression advbase + (advskew / 256) is unusual. An operator new to CARP expects the skew to be a whole number of seconds. The reason it is fractions is to give the operator fine-grained control over the master election without leaving the integer-second scale, and the divisor is 256 because the kernel is dividing a byte-sized field, not because anyone wanted a round decimal.

With advbase=1 and advskew from 0 to 240 — 240 is the ceiling the kernel enforces — the interval ranges from 1.000 to 1.938 seconds. The operator can pick advskew=10 for a 39-millisecond preference or advskew=100 for a 391-millisecond preference, and the backup keeps advertising every ~1.4 seconds. Both are well within the “three missed advertisements” tolerance.

The trade-off is that the skew is not directly comparable to the master election time — a backup with advskew=100 is slower to advertise, which makes it the loser in the election. Slow advertisement also means slow detection of a missing master, but the backup’s behaviour is mostly driven by the master’s advertisement rate, not its own.

The three CARP states

Each CARP interface on each node is in one of three states at any moment:

INIT. The interface is up but the node has not yet heard from any peer (or has not yet started advertising). This is the state on a freshly booted node before the first advertisement.
MASTER. The node is advertising on the segment and is the rightful owner of the VIP. It responds to ARP requests for the VIP and accepts frames sent to the VIP’s MAC.
BACKUP. The node is advertising on the segment but has heard from a peer with a lower effective skew. It does not respond to ARP for the VIP and silently discards frames sent to the VIP’s MAC.

The state changes are event-driven. A backup transitions to master when the master’s advertisements stop. A master transitions to backup when it sees a peer with a lower effective skew (only possible if the operator reconfigured skew on the live system). The INIT state is usually transient — a node in INIT for more than a few seconds has either lost its network or has its CARP daemon stopped.

Read-only / Safesysctl net.inet.carp— The CARP sysctls. allow=1 means the kernel will process CARP advertisements — setting it to 0 disables CARP entirely. preempt=1 enables preemption: if a backup comes up with a lower effective skew than the current master, it will take over. log=1 enables CARP state-change logging to the system log. demotion is the current demotion factor, which the kernel adds to the advertised skew of every VHID on this node; the ifdown and senderr factors are the amounts added to that counter when an interface goes down or an advertisement fails to send.

$ sysctl net.inet.carp

net.inet.carp.allow: 1
net.inet.carp.preempt: 1
net.inet.carp.dscp: 56
net.inet.carp.log: 1
net.inet.carp.demotion: 0
net.inet.carp.ifdown_demotion_factor: 240
net.inet.carp.senderr_demotion_factor: 240

Illustrative output

Summary

CARP is the native FreeBSD/OpenBSD protocol that lets two firewalls share one IP. OPNsense uses CARP, not VRRP.
A VHID identifies one shared IP on one segment. The VHID is carried in the advertisement and the password authenticates peer advertisements.
The master is the node with the shortest advertisement interval (advbase + advskew/256 seconds). Default is one advertisement per second with skew 0 on the primary, skew 100 on the secondary.
The three states are INIT (transient), MASTER (owns the VIP), and BACKUP (watching for the master). Transition from BACKUP to MASTER happens after three missed advertisements.
The most common split-brain cause is a mismatched CARP password.