Draining behind a load balancer - rolling maintenance without a cluster manager

The rolling-maintenance workflow in this part is written around pcs node standby, because a cluster manager makes the drain step explicit. Most stateless tiers do not have one. They have a load balancer, a health check, and a pool of interchangeable hosts — and the drain is something you do to the load balancer, not to the node.

The loop is identical: drain, change, validate, return, observe, and only then the next node. What changes is the mechanism, and two things about that mechanism regularly get a maintenance window wrong.

Drain is not the same as disable

HAProxy has two ways to take a server out of service and they behave differently for clients that are already talking to it.

drain is the gentle one. The server stops taking new work but continues to serve the sessions that are stuck to it, so a user part-way through a checkout is not thrown onto another node with an empty session. maint is the full stop, and it also stops health checking, which is what you want while the node is being rebooted so the logs are not full of failed checks.

The correct sequence for a planned maintenance is therefore both of them, in order: drain, wait for the session count to fall, then maint.

HAProxy: the runtime API

The runtime API is a Unix socket. It has to be enabled in the configuration, and the privilege level matters — level admin is required to change server state:

# /etc/haproxy/haproxy.cfg
global
    stats socket /run/haproxy/admin.sock mode 660 level admin expose-fd listeners
    stats timeout 30s

$ SOCK=/run/haproxy/admin.sock

# Read-only: what does HAProxy think the pool looks like?
echo "show servers state" | sudo socat stdio "$SOCK"

# Take web01 out of rotation for new sessions
echo "set server webservers/web01 state drain" | sudo socat stdio "$SOCK"

# Watch the current session count fall to zero
echo "show stat" | sudo socat stdio "$SOCK" | awk -F, '$1=="webservers" {print $2, $5, $18}'

1
# be_id be_name srv_id srv_name srv_addr srv_op_state ...
3 webservers 1 web01 192.0.2.11 2 0 1 1 ...
3 webservers 2 web02 192.0.2.12 2 0 1 1 ...

web01 47 UP
web02 52 UP

Once scur reaches zero for that server, complete the drain and do the work:

SOCK=/run/haproxy/admin.sock

echo "set server webservers/web01 state maint" | sudo socat stdio "$SOCK"

# ... patch, reboot, validate the node ...

# Return it. Watch the health check bring it back to UP before
# you believe traffic is flowing again.
echo "set server webservers/web01 state ready" | sudo socat stdio "$SOCK"
echo "show stat" | sudo socat stdio "$SOCK" | awk -F, '$1=="webservers" {print $2, $18}'

nginx: reload, do not restart

Open-source nginx has no runtime API for upstream state. The drain is a configuration change plus a reload:

# /etc/nginx/conf.d/upstream.conf
upstream webservers {
    server 192.0.2.11:80 down;      # draining
    server 192.0.2.12:80;
    server 192.0.2.13:80;
}

sudo nginx -t && sudo systemctl reload nginx

The -t && reload pairing is not optional. A reload of a broken configuration is refused, which is the good case; a restart of a broken configuration leaves you with no proxy at all.

Reload is graceful by design: nginx starts new worker processes with the new configuration and lets the old workers finish the requests they are already handling before exiting. So in-flight requests are not dropped by the reload itself. What is dropped is anything still open when the old workers are eventually killed — worker_shutdown_timeout bounds that wait, and long polling or streaming responses will hit it.

How long does a drain actually take?

This is the arithmetic that gets skipped, and it is the reason so many maintenance windows drop requests.

When a node removes itself from service by failing its own health check — the most portable pattern, and the one that works with any load balancer — the load balancer does not find out instantly. It finds out after fall consecutive failed checks, each inter apart:

detection time = inter x fall

HAProxy default:  inter 2s, fall 3   ->  up to 6 seconds
A conservative production setting:
                  inter 5s, fall 3   ->  up to 15 seconds

During that window the node is still receiving new requests. If you stop the application the instant you touch the drain flag, every request that arrives in those seconds fails.

The pattern that avoids it:

# 1. Tell the health endpoint to start failing
sudo touch /etc/myapp/draining

# 2. Wait longer than inter x fall, plus a margin
sleep 20

# 3. Confirm no established connections remain.
#    -H suppresses the header, so the count is connections, not lines.
ss -Htn state established '( sport = :8080 )' | wc -l

# 4. Only now stop the service
sudo systemctl stop myapp

Step 3 is the one that turns a guess into evidence. A count that is not falling means something is holding connections open, and that is the next section.

The connections that never drain

Draining assumes requests are short. Three common cases where they are not:

• WebSockets and long-lived gRPC streams. A client connects once and stays for hours. scur will not fall on its own. The application has to close them, usually by sending a close frame or a GOAWAY when it sees the drain flag.
• Database and internal service connection pools. A pool is established at startup and reused indefinitely, and the client has no reason to reconnect. Bound pool connection lifetime on the client side so pools rotate, or the “drained” node keeps serving pooled traffic until it is killed.
• HTTP keep-alive to the load balancer itself. Usually short-lived enough not to matter, but a very high timeout http-keep-alive extends the drain by the same amount.

For all three, the honest answer during a maintenance is a deadline: drain, wait the agreed period, and then close what remains deliberately rather than pretending the count will reach zero. The important part is that closing them is a decision you made, at a moment you chose, rather than a side effect of systemctl stop at the end of a window.

Where the health check should live

The most robust arrangement makes the node responsible for declaring itself out of service, because that survives whichever load balancer is in front of it:

GET /healthz
  -> 503 if /etc/myapp/draining exists
  -> 503 if the app cannot reach its database
  -> 200 otherwise

The load balancer configuration then never changes during a maintenance. You touch a file on the node, the health check fails, the load balancer removes it, and returning the node is removing the file. That works identically behind HAProxy, nginx, a cloud load balancer or a service mesh, and it means the drain step in your automation is one line that does not need to know what is in front of the tier.

It also means the check has to test something real. A /healthz that returns 200 as long as the process is listening will report a node healthy when its database connection has been dead for an hour — and validating a change against that check proves nothing, exactly as the health-validation lesson says.

Knowledge check