CARP multicast and switches — what the Layer 2 must do for HA to work

CARP is a multicast protocol. Two firewalls exchange advertisements on a multicast group, and both must hear the other for the protocol to converge. The Layer 2 between the two firewalls — the switch, the cabling, the VLAN — is what decides whether the advertisements reach. Most CARP failures are not CARP failures; they are Layer 2 failures that the operator sees as split-brain.

One thing to fix before anything else: CARP runs on the segment each VIP lives on. A LAN VIP advertises on the LAN, a WAN VIP on the WAN. It cannot be moved onto a dedicated back-to-back link, because the VIP has to answer ARP on the segment it serves. The dedicated sync link carries pfsync and the configuration sync; it carries no CARP at all.

That is why this lesson is about the switch in front of the production segments. This lesson covers the multicast MAC and the multicast IP group, what the switch must do to carry CARP frames between peers on those segments, what IGMP snooping does and does not do, and where the dedicated sync segment fits alongside.

The CARP multicast MAC and group

CARP uses a single, fixed multicast MAC and a single fixed multicast IPv4 group:

• MAC: 01:00:5e:00:00:12. The 01:00:5e prefix is the IANA multicast OUI; the 00:00:12 is the CARP-specific 23 bits. The low-order bit of the first byte is set (multicast), and the next 23 bits are the CARP protocol identifier.
• IPv4 group: 224.0.0.18. This is in the link-local multicast range (224.0.0.0/24), which is never routed. A router that receives a packet to 224.0.0.18 will not forward it — the TTL is forced to 1 in the software, and even if the TTL were higher, the group is reserved for link-local.
• IPv6 group: ff02::12. The link-local scope (ff02::/16) means the same thing — never routed.

The “never routed” is the key property. CARP must not cross a router. If it did, the two firewalls would be on the same CARP group across the routed network, and the failover would be driven by routing behaviours, not by the local election. The design choice is that CARP is for two firewalls on the same Layer 2 segment, and the segment is the boundary.

There is no CARP ethertype. An advertisement is an ordinary IPv4 (or IPv6) frame whose payload is IP protocol 112, so the switch sees a normal multicast IP frame and the CARP-ness only becomes visible once the IP header is parsed. That is what proto carp matches on:

$ tcpdump -n -i igb1 proto carp -c 1

12:34:56.789012 IP 198.51.100.10 > 224.0.0.18: CARPv2-advertise 36: vhid=1 advbase=1 advskew=0 authlen=7 counter=4218765

What the switch must do

A switch is a Layer 2 device. It receives a frame on one port, looks at the destination MAC, and forwards the frame to the port that owns that MAC. For unicast frames, the switch has a CAM table that maps MAC to port. For multicast and broadcast frames, the switch forwards to all ports in the same VLAN — except when the switch has been configured to do something else.

The “something else” is where CARP fails. The switch configurations that break CARP:

• Multicast suppression. Some switches (especially older or cheaper ones) treat multicast as “best effort, drop if congested”. The switch silently drops CARP frames when the segment is busy. The operator sees split-brain with no apparent cause.
• IGMP snooping without IGMP querier. IGMP snooping is a feature that lets the switch learn which hosts have joined which multicast groups and forward multicast only to the interested ports. CARP, however, does not send IGMP joins — it assumes the segment is a “multicast to all” link-local. IGMP snooping without proper configuration may drop CARP because the switch thinks no host has joined the group.
• Port isolation (private VLAN). Some switches can be configured to prevent port-to-port communication on the same VLAN. The two firewalls are on ports that cannot talk to each other. CARP cannot work.
• Asymmetric VLAN. The two firewalls are on the same VLAN but the switch has a misconfigured trunk that puts the CARP frames on the wrong VLAN on the egress port. The remote firewall never sees them.
• Port security. The switch has been configured to limit the number of MAC addresses per port. The CARP multicast MAC is one of the allowed MACs — but the per-port limit is lower than the number of MACs the firewall uses. The switch drops the frame.

The opposite — a switch that does carry CARP correctly — is the minimum. The switch must:

• Forward multicast frames to all ports in the same VLAN by default (or via IGMP snooping with a querier configured).
• Not apply multicast suppression to the CARP group.
• Not isolate the two firewall ports from each other.
• Not apply per-port MAC limits below the number of CARP groups on the firewall.

IGMP — what it does and does not do for CARP

IGMP (Internet Group Management Protocol) is the IPv4 protocol that lets hosts tell the upstream router which multicast groups they want to receive. The router uses the IGMP joins to decide which multicast to forward to which interface.

For inter-router multicast (e.g. OSPF, PIM), IGMP is essential. The router only forwards multicast to interfaces that have hosts that joined the group.

For intra-segment multicast (like CARP, which is link-local), IGMP is not essential. The hosts on the segment are supposed to receive the multicast regardless of joins. But IGMP snooping — a switch feature that watches IGMP joins to learn which ports to forward multicast to — can interfere.

IGMP snooping is configured on most managed switches. The default behaviour is to drop multicast to ports that have not sent an IGMP join. CARP, however, does not send IGMP joins (there is no router to join with). The IGMP snooper sees no joins for 224.0.0.18 and may drop the CARP frames.

The fix:

• Disable IGMP snooping on the CARP segment. The simplest option. The switch forwards multicast to all ports in the VLAN, just like broadcast.
• Configure an IGMP querier on the switch. Some enterprise switches support an IGMP querier on the VLAN. The switch periodically asks “who wants 224.0.0.18?” and the firewalls respond with a join. The switch then forwards.
• Use flood instead of drop for unknown multicast. Some switches have a setting for “unknown multicast” — flood, drop, or forward to router port. Flood is the safe choice for CARP.

The dedicated sync segment layout

The production layout is two firewalls, each with at least three interfaces:

• LAN — the interface that hosts the LAN-side CARP VIP. The LAN VIP is the default gateway for the LAN hosts.
• WAN — the interface that hosts the WAN-side CARP VIP. The WAN VIP is what the ISP sees as the customer’s gateway.
• SYNC — the dedicated sync interface. The interface is point-to-point to the other firewall’s sync interface. It carries pfsync state updates and the XMLRPC configuration sync. It does not carry CARP advertisements: those travel on the LAN and WAN interfaces, where the VIPs live. It has no other host.

The sync interface is configured with a /30 or /31 subnet — just enough addresses for the two firewalls. The interface has no CARP VIP, no firewall rules for inbound traffic other than pfsync, no NAT, no services. The interface exists to be a private channel between the two nodes.

$ ifconfig igb2

igb2: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
      ether 66:77:88:99:aa:cc
      inet 198.51.100.10 netmask 0xfffffffc broadcast 198.51.100.11
      status: active

Summary

• CARP uses multicast MAC 01:00:5e:00:00:12, IPv4 group 224.0.0.18, IPv6 group ff02::12, and IP protocol 112. There is no CARP ethertype — the frame is ordinary IPv4. The groups are link-local, so they are never routed.
• A switch must forward CARP frames between the two peers. Switch features that break CARP: multicast suppression, IGMP snooping without a querier, port isolation, asymmetric VLAN, port security.
• The advertisements travel on each VIP’s parent segment, so it is the LAN and WAN switches that have to carry them. The dedicated sync segment carries pfsync and the configuration sync, and no CARP at all.
• IGMP snooping should be disabled on every VLAN carrying a CARP VIP, or the switch should be configured to flood link-local multicast there.