IPVS and the kernel load balancer - the layer 4 workhorse

IPVS (IP Virtual Server) is the Linux kernel’s layer 4 load balancer. It is built into the kernel, runs in kernel space, and handles very high throughput. This lesson covers how to use it.

What IPVS does

IPVS is a kernel feature:

• Runs in kernel space (no context switch per packet).
• Supports TCP, UDP, SCTP, ESP, AH.
• Multiple algorithms: rr, wrr, lc, wlc, lblc, dh, sh, sed, nq.
• Often used with keepalived for failover.

Install

sudo apt install ipvsadm keepalived

ipvsadm is the user-space tool to configure IPVS.

Configure IPVS

# Add a virtual service (VIP)
sudo ipvsadm -A -t 10.0.0.100:80 -s rr

# Add real servers (backends)
sudo ipvsadm -a -t 10.0.0.100:80 -r 10.0.0.10:80 -g
sudo ipvsadm -a -t 10.0.0.100:80 -r 10.0.0.11:80 -g
sudo ipvsadm -a -t 10.0.0.100:80 -r 10.0.0.12:80 -g

-g is direct routing (the default). -i is TUN, -m is NAT.

Direct routing (-g)

The director rewrites only the destination MAC address. The IP header is untouched, so the real server sees the original destination IP - the VIP - and replies straight to the client, bypassing the director entirely. Return traffic never crosses the load balancer, which is why DR scales so well.

That only works if every real server satisfies two conditions:

1. It holds the VIP on lo as a /32, so the kernel accepts packets addressed to the VIP.
2. It suppresses ARP for the VIP, so it never answers ARP requests for it.

# On EVERY real server - not the director
ip addr add 10.0.0.100/32 dev lo

sysctl -w net.ipv4.conf.all.arp_ignore=1
sysctl -w net.ipv4.conf.all.arp_announce=2
sysctl -w net.ipv4.conf.lo.arp_ignore=1
sysctl -w net.ipv4.conf.lo.arp_announce=2

Persist them, or the next reboot silently breaks the cluster:

sudo tee /etc/sysctl.d/10-lvs-dr.conf >/dev/null <<'EOF'
net.ipv4.conf.all.arp_ignore = 1
net.ipv4.conf.all.arp_announce = 2
net.ipv4.conf.lo.arp_ignore = 1
net.ipv4.conf.lo.arp_announce = 2
EOF
sudo sysctl --system

arp_ignore=1 means “only answer ARP for an address configured on the interface the request arrived on” - the VIP is on lo, not on eth0, so the reply is suppressed. arp_announce=2 means “always use the best local address for the interface” so the real server never advertises the VIP as its own source in outgoing ARP.

Verify from a client or the upstream router. Exactly one MAC must answer, and it must be the director’s:

arping -c 3 -I eth0 10.0.0.100
ip neigh show 10.0.0.100

NAT (-m)

The director rewrites the destination address from the VIP to the chosen real server. The client’s source IP is preserved, which is why applications behind LVS-NAT still see real client addresses.

Because the source is untouched, the real server would reply directly to the client with a source of its own IP - which the client’s connection tracking would drop, since it expects replies from the VIP. So return traffic must pass back through the director for the reverse translation. That means:

• Each real server’s default gateway must be the director.
• net.ipv4.ip_forward=1 must be set on the director.

NAT adds a hop and makes the director carry both directions of traffic, so it becomes the throughput ceiling.

TUN (-i)

The director encapsulates the packet in IPIP. The real server decapsulates and replies directly to the client, like DR, but the real server does not need to be on the same L2 segment. Used for cross-site or cross-region pools.

Algorithms

• -s rr: round robin.
• -s wrr: weighted round robin.
• -s lc: least connections.
• -s wlc: weighted least connections.
• -s sh: source hash (session affinity).
• -s dh: destination hash.
• -s sed: shortest expected delay.
• -s nq: never queue.

For most use cases, wlc is a good default (weighted, with connection awareness).

Persistence

For session affinity (same client to same backend):

sudo ipvsadm -A -t 10.0.0.100:80 -s rr -p 600

-p 600 means 600-second persistence. After 600 seconds, the client may go to a different backend.

Keepalived integration

IPVS is often used with keepalived for VIP failover. Be precise about where the VIP lives:

• On the director, the VIP goes on the LAN interface, so it answers ARP and receives client traffic. Let keepalived’s virtual_ipaddress place it, so it moves with the master role.
• On the real servers, the VIP goes on lo with ARP suppression, as shown above. It never moves.

# On the active director - keepalived normally does this for you
ip addr add 10.0.0.100/24 dev eth0

# Configure IPVS
ipvsadm -A -t 10.0.0.100:80 -s wlc
ipvsadm -a -t 10.0.0.100:80 -r 10.0.0.10:80 -g
ipvsadm -a -t 10.0.0.100:80 -r 10.0.0.11:80 -g

IPVS does no health checking of its own. A real server that is down stays in the table and keeps receiving connections until something removes it. Let keepalived own both the VIP and the pool:

# /etc/keepalived/keepalived.conf on both directors
virtual_server 10.0.0.100 80 {
    delay_loop 6
    lb_algo wlc
    lb_kind DR
    protocol TCP

    real_server 10.0.0.10 80 {
        weight 1
        HTTP_GET {
            url { path /health; status_code 200 }
            connect_timeout 3
            retry 3
        }
    }
    real_server 10.0.0.11 80 {
        weight 1
        HTTP_GET {
            url { path /health; status_code 200 }
            connect_timeout 3
            retry 3
        }
    }
}

When to use IPVS

• Layer 4 LB is sufficient (no layer 7 routing).
• Maximum throughput is needed.
• Latency matters (kernel-space).
• Many concurrent connections (millions).

For most web traffic, HAProxy is enough. IPVS shines for high-throughput, layer 4 use cases.

Knowledge check