Virtual IP mechanics - what actually moves during a failover · Linux

A failover is usually declared successful the moment the VIP shows up on the backup node. That check is necessary and it is not sufficient: the address moving and the traffic moving are two different events, separated by every ARP cache on the segment.

What keepalived actually does

On transition to MASTER, keepalived performs two operations:

Adds the address. The equivalent of ip addr add 203.0.113.10/24 dev eth0. This is entirely local - no other machine learns anything from it.
Announces it at layer 2. For IPv4, a gratuitous ARP (GARP): an ARP message the node sends unprompted, saying “203.0.113.10 is at this MAC”. For IPv6, an unsolicited neighbour advertisement.

Step 1 is what ip addr show reveals, and it is the step every failover check looks at. Step 2 is the one that makes the failover visible to anyone else, and it is the one that fails silently.

Why the announcement is necessary

Two separate caches have to be corrected, and neither expires quickly:

Switch MAC address tables. A switch forwards a frame to the port where it last saw that destination MAC as a source. Until the new node sends something from the VIP’s MAC, frames for that MAC keep going to the old port.
Host and router neighbour caches. Every client that has spoken to the VIP holds an IP-to-MAC mapping. Linux keeps entries usable for minutes after last use; network appliances and some embedded stacks keep them far longer.

The GARP updates both at once: it is broadcast, it carries the new MAC as the source, and receivers overwrite the mapping.

Read-only / Safethe gratuitous ARPs, sent from the new master on transition

# tcpdump -ni eth0 -c 8 arp or icmp6

14:22:07.118431 ARP, Reply 203.0.113.10 is-at 52:54:00:aa:bb:02, length 28
14:22:07.118552 ARP, Reply 203.0.113.10 is-at 52:54:00:aa:bb:02, length 28
14:22:07.118661 ARP, Reply 203.0.113.10 is-at 52:54:00:aa:bb:02, length 28
14:22:07.118742 ARP, Reply 203.0.113.10 is-at 52:54:00:aa:bb:02, length 28
14:22:07.118829 ARP, Reply 203.0.113.10 is-at 52:54:00:aa:bb:02, length 28

Illustrative output

Seeing these on the segment is the real evidence that a failover happened. Not seeing them, while ip addr shows the VIP present, is the signature of the failure below.

The VIP moved and traffic did not

ip addr shows the VIP, and clients still reach the dead node

This is the most common keepalived incident, and the check that was performed - ip addr show | grep 203.0.113.10 on the backup - is the reason nobody diagnosed it for twenty minutes. The address is local state. It says nothing about whether the network was told.

The causes, in the order they occur in practice:

The GARP never left the host. A host firewall rule, or a bridge/hypervisor filter that drops unsolicited ARP replies.
The GARP was sent and the switch discarded it. Dynamic ARP inspection or port security configured against the old MAC-to-port binding.
A static ARP entry on a router or firewall. GARP does not override a manually configured mapping, ever.
The platform does not honour GARP at all. Major cloud networks do not learn addresses from ARP; moving an address requires an API call. VRRP simply does not work there unmodified.
Asymmetric announcement. The VIP is on a bonded or tagged interface and the GARP went out on a different VLAN from the one clients are on.

Diagnose from a third host on the segment, not from either peer. The peers agree with themselves.

# From a third host: what does the segment believe?
ip neigh show 203.0.113.10

# Force a fresh resolution and see who answers
sudo ip neigh flush dev eth0 203.0.113.10
ping -c 2 203.0.113.10
ip neigh show 203.0.113.10

If the neighbour entry names the old node’s MAC after a flush and a fresh ARP exchange, the old node is still answering for the VIP - which is a dual-MASTER situation, not a stale cache. linux-keepalived-limitations covers that path.

Tuning the announcement

Keepalived exposes the GARP behaviour, because one packet at one moment is fragile:

Directive	Effect
`garp_master_delay`	Seconds to wait after becoming master before sending the GARPs
`garp_master_repeat`	How many gratuitous ARPs to send
`garp_master_refresh`	Seconds between periodic repeat announcements while master

garp_master_refresh is the one worth setting in an environment that has ever surprised you. A single burst at transition time can be lost; a periodic refresh every 30 or 60 seconds re-asserts the mapping continuously, so a device that missed the transition corrects itself within a minute instead of whenever its cache ages out.

garp_master_delay matters where the old master needs a moment to release the address. Announcing before it has stopped answering produces a brief period where both nodes reply to ARP for the same address, and whichever reply arrives last wins per client.

The virtual MAC trade-off

VRRP as specified uses a virtual MAC derived from the router id: 00:00:5e:00:01:<vrid>. The MAC moves with the address, so no client’s IP-to-MAC mapping ever becomes wrong - only the switch’s port mapping has to update.

Keepalived does not do this by default. It keeps the real interface MAC and relies on GARP to correct every cache. The use_vmac directive switches to the specification behaviour by creating a macvlan interface for the VIP.

The trade-off is real in both directions:

With use_vmac: client caches never go stale, so failover is robust against hosts that ignore GARP. But the segment now sees a MAC that moves between switch ports, which collides with port security, some bonding configurations, and most cloud virtual networks.
Without it (the default): nothing unusual on the wire, but correctness depends on every client honouring an unsolicited ARP.

Neither is wrong. Choose based on what your switching layer permits, and if you enable use_vmac, verify the switch has not learned the MAC as a security violation.

Binding to an address you do not hold

Services like HAProxy want to bind the VIP at startup, and on the backup node that address is absent. By default the bind fails and the unit will not start, so the service is not ready at the moment it becomes master.

# Allow binding to a non-local address
sudo sysctl -w net.ipv4.ip_nonlocal_bind=1
sudo sysctl -w net.ipv6.ip_nonlocal_bind=1

# Persist it
echo 'net.ipv4.ip_nonlocal_bind = 1' | sudo tee /etc/sysctl.d/60-nonlocal-bind.conf

The alternative is to bind 0.0.0.0 and filter elsewhere. The sysctl is more precise and keeps the configuration honest about which address the service serves.

What does not move with the VIP

The address moves. These do not:

Connection tracking state. Established TCP connections were tracked on the old node. The new node sees mid-stream packets with no matching entry, and a stateful firewall drops them as invalid. Clients see a hang until their TCP timeout rather than a clean reset. conntrackd can replicate state between peers for cases where this matters.
TCP connections themselves. Even with conntrack synced, the sockets and their buffers lived in the old kernel. Clients must reconnect. Design the application for it rather than trying to preserve it.
Application session state, unless it is in a shared store.
The old master’s ARP entries about its peers, which is a smaller problem but occasionally explains an odd first second after failover.

Verify a failover from a third host, over the service

Two checks, in this order:

# On the new master - necessary, not sufficient
ip -br addr show eth0 | grep 203.0.113.10

# From a third host on the segment - this is the actual test
ip neigh flush dev eth0 203.0.113.10
curl -sS -o /dev/null -w '%{http_code}\n' http://203.0.113.10/healthz
ip neigh show 203.0.113.10

The second block proves the segment learned the new mapping and that the service answered over it. A failover drill that only runs the first block has tested the half that rarely fails.