Connection draining - taking a backend out without dropping requests

Every deployment, every kernel upgrade and every hardware replacement requires taking a backend out of service. It is the most frequent operation a load balancer performs, and the one most often done by stopping the service and letting the health check notice - which drops every request in flight and every request that arrives during the detection window.

Draining is the alternative: stop sending new work to a backend, let the work it already has finish, then act.

Three states, and choosing between them

HAProxy names the states explicitly, and the vocabulary transfers to other load balancers even where the mechanism differs.

The distinction that matters operationally is the health check. A drained server is still being checked, so you can see whether it is healthy while it empties. A server in maint is not checked at all, which is what you want while it is rebooting - otherwise the check failures raise alerts about a server you deliberately took out.

Draining a HAProxy backend

The runtime API is reached through the stats socket. Configure it once, with a socket that is not world-writable:

global
    stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
    stats timeout 30s
    hard-stop-after 5m

expose-fd listeners is what allows a reload to hand its listening sockets to the new process rather than closing and reopening them, which is the difference between a reload that is invisible and one that refuses connections for a few milliseconds.

# echo "set server be_app/app2 state drain" | socat stdio /run/haproxy/admin.sock; echo "show servers state be_app" | socat stdio /run/haproxy/admin.sock

1
# be_id be_name srv_id srv_name srv_addr srv_op_state srv_admin_state srv_uweight
3 be_app 1 app1 192.0.2.21 2 0 100
3 be_app 2 app2 192.0.2.22 2 8 100

Then watch it empty. This is the step people skip, and it is the only one that proves the drain worked.

# echo "show stat" | socat stdio /run/haproxy/admin.sock | awk -F, '$1=="be_app" {print $2, $5, $6, $18}'

app1 41 512 UP
app2 6 512 DRAIN
BACKEND 47 1024 UP

1. Confirm the remaining servers can carry the full load before you remove anything
2. Set the target server to drain, and record the time
3. Poll the current session count until it reaches zero, or until it stops falling
4. Set the server to maint, so its health checks stop and it does not alert while you work
5. Do the work - restart, upgrade, reboot
6. Set the server back to ready and watch the health checks pass before declaring it in service
7. Confirm the session count on it is rising again, which proves traffic is genuinely returning to it

nginx: down and a graceful reload

Open-source nginx has no runtime API for upstream membership. Removal is a configuration change plus a reload - which is graceful, and which is the whole mechanism.

upstream app {
    zone app 64k;
    server 192.0.2.21:8080 max_fails=3 fail_timeout=10s;
    server 192.0.2.22:8080 down;
}

# nginx -t && systemctl reload nginx; ss -tn state established '( dport = :8080 )' | wc -l

nginx: configuration file /etc/nginx/nginx.conf test is successful
38

Two properties of that reload are worth stating explicitly, because they are where the graceful part comes from and where it stops:

• Old worker processes keep running until their current requests complete. They are visible as processes in shutting down state, and worker_shutdown_timeout bounds how long they are allowed to linger before being terminated.
• Without that timeout, a worker holding a websocket can stay alive indefinitely, so a busy server accumulates shutting-down workers across successive reloads until memory becomes a problem.

The drain parameter and the runtime upstream API - which would let you do this without editing a file - are part of the commercial nginx subscription, not the open-source build. On open-source nginx, down plus a reload is the supported path.

IPVS: weight zero

IPVS has no drain state, but weight zero produces the same behaviour: the scheduler stops selecting the real server for new connections while existing entries in the connection table continue to be forwarded.

# ipvsadm -e -t 192.0.2.10:443 -r 192.0.2.22:443 -w 0; ipvsadm -L -n; ipvsadm -L -n -c | grep -c 192.0.2.22

Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.0.2.10:443 wlc
-> 192.0.2.21:443               Route   100    412        88
-> 192.0.2.22:443               Route   0      27         12
27

The IPVS equivalent of the persistence trap is the persistence template: with -p set on the virtual service, a client that already has a persistence entry continues to be sent to the weight-zero server until that entry expires. Same behaviour, same planning requirement.

Under keepalived, set inhibit_on_failure on the real server so that a failed health check sets its weight to zero rather than removing it from the table outright - the same graceful behaviour, applied automatically.

Make the application drain itself

The most portable mechanism needs no runtime API at all: have the application fail its own health check while continuing to serve real traffic.

1. The deployment tooling touches a flag file, or calls an admin endpoint on the application
2. The health endpoint begins returning 503 while the service endpoints keep working normally
3. The load balancer marks the backend down after its configured checks fail, and stops sending new requests
4. The application finishes its in-flight work, then exits

This works identically behind HAProxy, nginx, IPVS, a hardware load balancer or a cloud one, and it puts the decision in the process that actually knows whether it has finished. The cost is the detection delay: the backend keeps receiving new requests until the check fails enough times to count.

$ curl -sS -o /dev/null -w '%{http_code}\n' http://192.0.2.22:8080/health; sleep 6; curl -sS -o /dev/null -w '%{http_code}\n' http://192.0.2.22:8080/health

503
503

Capacity is the precondition

None of this is graceful if the servers that remain cannot carry the traffic. Before draining anything, do the arithmetic: if three servers at 70% utilisation become two, those two go to 105% and the drain converts a maintenance window into an outage.

The rule for rolling maintenance is to remove at most as many backends as leaves the remainder below the utilisation you are willing to run at during a further unplanned failure. On a three-node pool, that usually means one at a time, and it means checking the current load rather than the load at planning time.

Knowledge check