Failover, VIPs, and split-brain across two hosts

Two Docker hosts run the same stack. One is serving. The other is idle, warm, and identical. Everything is in place except the answer to one question: when host A dies, how does traffic reach host B?

Three mechanisms are available, they fail differently, and the choice between them determines whether “failover” means fifteen seconds or fifteen minutes.

DNS failover is not failover

The instinct is to point the record at the other host. It is the slowest option available, for reasons that stack:

• TTL is a floor, not a promise. A 60-second TTL means the fastest a well-behaved resolver refreshes is 60 seconds. Many are not well behaved.
• Client runtimes cache independently. A JVM configured to cache successful lookups indefinitely will hold the dead address until the process restarts. Connection pools hold open sockets and never re-resolve at all.
• Intermediate resolvers extend TTLs. Corporate and ISP resolvers routinely serve records past their nominal lifetime.
• You cannot measure the tail. There is no way to know when the last client stopped using the old address.

DNS is an excellent way to perform a planned migration over hours. It is a poor way to survive a host dying at 03:00.

A virtual IP with VRRP

The better answer for two hosts on the same L2 segment is to move the address itself. VRRP — implemented by keepalived on Linux — lets two hosts share one IP: one holds it, the other watches, and takeover is a matter of seconds.

flowchart TD
  C[Clients] -->|192.0.2.10| VIP{{VIP 192.0.2.10}}
  VIP -->|MASTER, priority 150| A[host-a 192.0.2.11<br/>Docker + proxy]
  VIP -.->|BACKUP, priority 100| B[host-b 192.0.2.12<br/>Docker + proxy]
  A --> DB[(Shared database)]
  B --> DB

The master advertises its presence periodically. When the backup stops hearing advertisements it claims the address, brings it up on its own interface, and broadcasts a gratuitous ARP so switches and neighbours update their tables. Clients keep using the same address and mostly notice only a reset connection.

Docker needs no configuration for this. A container publishing a port binds to 0.0.0.0 on the host by default, so it answers on the VIP as soon as the VIP exists on that host.

# /etc/keepalived/keepalived.conf on host-a
vrrp_script chk_proxy {
    script "/usr/local/bin/check-proxy.sh"
    interval 2
    timeout 3
    fall 2
    rise 2
    weight -60
}

vrrp_instance VI_EDGE {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 150
    advert_int 1
    unicast_src_ip 192.0.2.11
    unicast_peer {
        192.0.2.12
    }
    authentication {
        auth_type PASS
        auth_pass REPLACE_ME
    }
    virtual_ipaddress {
        192.0.2.10/24 dev eth0
    }
    track_script {
        chk_proxy
    }
}

Two details in that file do most of the work. virtual_router_id must be identical on both hosts and unique on the segment — two unrelated pairs sharing an ID will fight over each other’s addresses. And unicast_peer avoids multicast, which many cloud and virtualised networks silently drop, producing a pair where both hosts believe they are master from the first second.

Track the service, not the host

Make the check strict enough to be meaningful and loose enough not to be twitchy. A check that probes the shared database will fail on both hosts during a database blip, moving the VIP back and forth while neither host can serve — noise on top of an outage.

Split-brain: the problem two nodes cannot solve

There is no configuration that fixes this with two nodes. The reason is structural: to decide safely which side should keep serving, a node must know it is in the majority, and neither half of a two-node cluster is ever a majority of two.

What you can do is narrow it:

• A tie-breaker outside the pair. A vrrp_script that pings the default gateway and lowers priority on failure means the host that lost its network connection also gives up the VIP. This resolves the common case — one host’s uplink failing — and does nothing for a partition where both can still reach the gateway.
• A third witness. A third node, or an external arbiter, restores a majority and is the only real fix. At that point you are building a cluster, and Pacemaker or an orchestrator is a more honest tool than keepalived.
• Accept it, in writing. For a stateless service behind a VIP, split-brain means a period of confused routing and broken sessions, and it ends when the partition heals. That is a legitimate thing to accept if you have written down that you accepted it.

Verifying a failover before you need it

ip -brief addr show dev eth0 | grep -o '192\.0\.2\.10' || echo 'not here'

# And what keepalived thinks it is
journalctl -u keepalived --since '10 min ago' --no-pager | tail -20

$ systemctl stop keepalived

Aug 11 10:14:02 host-b Keepalived_vrrp: (VI_EDGE) Entering MASTER STATE
Aug 11 10:14:02 host-b Keepalived_vrrp: (VI_EDGE) setting VIPs.

Test both directions and measure both. Failing over is the easy half; failing back is where people discover that host A came up with priority 150, seized the VIP the instant it booted, and did so before its containers were ready. nopreempt on the backup, or starting the check script before keepalived, prevents that — and you only find out you needed it during a drill.

Sanity check