Sync groups and multiple VIPs - keeping a service together

Almost no real service is one address. A load balancer pair fronts an IPv4 VIP and an IPv6 VIP; a database pair has a client address on the application VLAN and a replication address on the storage VLAN. Keepalived will happily let those addresses live on different nodes at the same time, and the result is a service that is up on paper and broken in practice.

Half a service on each node

The failure is worth picturing before the configuration.

lb-01                        lb-02
  VI_FRONTEND  MASTER          VI_FRONTEND  BACKUP
  203.0.113.10                 -
  VI_BACKEND   BACKUP          VI_BACKEND   MASTER
  -                            198.51.100.10

Clients reach lb-01 on the frontend VIP. The backend address they must be proxied to lives on lb-02. Every request fails, and systemctl status keepalived is green on both nodes because each instance is doing exactly what it was told.

This happens whenever two independent VRRP instances make independent decisions - which is what they do by default. A health check that fails on one VLAN, a NIC that goes down, a priority difference, or simply the order the two nodes started in is enough to separate them.

Option one: several addresses, one instance

If the addresses belong to the same interface and the same failure domain, put them in one instance. Members of a single virtual_ipaddress block always move together, because there is only one state machine.

vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 51
    priority 150
    advert_int 1
    virtual_ipaddress {
        203.0.113.10/24 dev eth0
        203.0.113.11/24 dev eth0
        203.0.113.12/24 dev eth0
    }
}

This is the correct answer far more often than a sync group is, and it should be the default. One instance means one VRID, one election, one set of advertisements, and no possibility of the addresses separating.

Two constraints to know:

• Keepalived advertises a bounded number of addresses in the VRRP packet. Beyond that limit, additional addresses go in a virtual_ipaddress_excluded block - they still move with the instance, they are simply not carried in the advertisement. If you are configuring more than a handful of VIPs, check the limit in the documentation for your version rather than discovering it when keepalived rejects the config.
• All the addresses in one instance share a single state. There is no way to fail one over independently, which is the point and occasionally the problem.

Option two: several instances, one sync group

When the addresses genuinely must be separate instances - a different interface, a different VLAN, or a different address family - a vrrp_sync_group binds their state machines together so they transition as a unit.

vrrp_sync_group VG_SERVICE {
    group {
        VI_FRONTEND
        VI_BACKEND
    }
    notify_master "/etc/keepalived/scripts/promote.sh"
    notify_backup "/etc/keepalived/scripts/demote.sh"
}

vrrp_instance VI_FRONTEND {
    state BACKUP
    interface eth0
    virtual_router_id 51
    priority 150
    advert_int 1
    virtual_ipaddress {
        203.0.113.10/24 dev eth0
    }
}

vrrp_instance VI_BACKEND {
    state BACKUP
    interface eth1
    virtual_router_id 52
    priority 150
    advert_int 1
    virtual_ipaddress {
        198.51.100.10/24 dev eth1
    }
}

Note the distinct virtual_router_id values. Each instance is a separate VRRP router and needs its own VRID, unique among all VRRP groups on the segment it advertises on. The VRID selects the virtual MAC, so reusing 51 on both would collide with any other pair using the default.

The notify_* scripts belong on the group rather than on the instances. Placed on the group they run once per group transition; placed on the instances they run once per instance, so a two-instance group runs the promotion script twice.

Dual stack needs two instances

A single VRRP instance carries one address family. IPv4 and IPv6 VIPs therefore cannot share an instance, and a dual-stack service always needs at least two instances plus a sync group to keep them together.

vrrp_sync_group VG_WEB {
    group {
        VI_WEB_V4
        VI_WEB_V6
    }
}

vrrp_instance VI_WEB_V4 {
    state BACKUP
    interface eth0
    virtual_router_id 51
    priority 150
    advert_int 1
    virtual_ipaddress {
        203.0.113.10/24 dev eth0
    }
}

vrrp_instance VI_WEB_V6 {
    state BACKUP
    interface eth0
    virtual_router_id 52
    priority 150
    advert_int 1
    virtual_ipaddress {
        2001:db8::10/64 dev eth0
    }
}

Without the sync group, a dual-stack service can end up with IPv4 on one node and IPv6 on the other. Clients with working IPv6 reach one machine and clients without it reach the other - which is an outage that only affects some users, discovered late, and blamed on the application.

The cost of grouping

Interface tracking, and where to put it

The reason instances fault in the first place is usually track_interface or a track_script. Both are configured per instance:

vrrp_instance VI_BACKEND {
    state BACKUP
    interface eth1
    virtual_router_id 52
    priority 150
    advert_int 1
    track_interface {
        eth1
    }
    virtual_ipaddress {
        198.51.100.10/24 dev eth1
    }
}

With the instance in a sync group, eth1 going down now moves the entire service - which is exactly right when the backend path is required to serve, and exactly wrong when it is a replication link the service can briefly do without.

Which of those is true is a decision about the service, not about keepalived. Make it explicitly, and write it in the comment above the track_interface block.

Newer keepalived releases also accept tracking directives at the group level. Check the documentation for your version before relying on that, and verify the parsed result rather than the intent:

sudo keepalived -t -f /etc/keepalived/keepalived.conf
sudo systemctl reload keepalived
journalctl -u keepalived -n 40 --no-pager

The -t flag checks the configuration without running it, which is the difference between finding a typo now and finding it during the next failover.

Verifying the group moves as one

# ip -br addr show | grep -E '203\.0\.113\.10|198\.51\.100\.10'

eth0             UP             203.0.113.5/24 203.0.113.10/24
eth1             UP             198.51.100.5/24 198.51.100.10/24

The test is to run it on both nodes during a drill. One node holding both addresses and the other holding neither is a pass. Each node holding one is the split-service failure, and it will not show up in any single-host check.

# Force the group over and confirm it moved as a unit
sudo systemctl stop keepalived        # on the current master
ssh lb-02 "ip -br addr show | grep -E '203\.0\.113\.10|198\.51\.100\.10'"

Knowledge check