External load balancing — HA without an orchestrator · Docker & Containers

The previous lesson established that standalone Docker cannot fail over, because nothing outside the host observes it. This lesson builds the thing that observes it: a proxy in front of two or more Compose hosts, which polls each one and routes around the ones that stop answering.

flowchart LR
  Internet((Clients)) --> LB[Proxy with health checks]
  LB -->|checked every 2s| Host1[host-01: Compose stack]
  LB -->|checked every 2s| Host2[host-02: Compose stack]
  Host1 --> DB[(Managed or replicated DB)]
  Host2 --> DB
  Host1 --> Obj[(Object storage)]
  Host2 --> Obj

The picture is easy. Everything difficult is in two places: what the health check tests, and what happens to connections when a backend leaves the pool.

Prerequisite: the state has to move first

Before any of this is worth doing, deal with the fact that a named volume is local to one host. app_pgdata on host-01 is not visible from host-02 and never will be.

State	Where it must go
Relational database	A managed service, or a replicated cluster with a defined promotion procedure
User uploads	Object storage (S3-compatible), or a shared filesystem both hosts mount
Sessions	Redis or the database — never in-process memory
Cache	Shared Redis, or genuinely per-host and cold-start tolerant
Certificates	Terminated at the proxy, or a shared store both hosts read

Skipping this produces the failure named in the previous lesson: two hosts, one database on host A, and a load balancer that confidently routes to a survivor that cannot serve a request. Adding a second host improves availability only after the single-instance state is gone from both.

Sessions deserve a specific mention because they fail subtly rather than loudly. In-process sessions plus round-robin load balancing means every second request lands on a host that has never seen the user, and the symptom is “randomly logged out” rather than an outage. The usual workaround — sticky sessions at the proxy — makes it work and quietly recreates the problem: a host failure now logs out every user pinned to it, and it prevents clean draining. Move sessions to shared storage; do not paper over them with stickiness.

The health check is the whole design

The proxy’s entire model of reality is one endpoint’s response. If that endpoint returns 200 while the service is unusable, the proxy will keep routing traffic into it, and no amount of good configuration elsewhere helps.

Read-only / Safethe two kinds of endpoint— One tells you the process is alive. The other tells you it can serve a request.

HOST=127.0.0.1:8080

# Liveness: is the process running? Cheap, no dependencies.
# Used by the container health check and by process supervision.
curl -sS "http://$HOST/healthz"

# Readiness: can this instance serve a real request right now?
# Checks the database connection, the cache, the disk it writes to.
# This is what the LOAD BALANCER must poll.
curl -sS "http://$HOST/readyz"

Read-only / Safereadiness that means something— The database is unreachable. Liveness is green; readiness is not.

$ curl -sS -w '\\n%{http_code}\\n' http://127.0.0.1:8080/readyz

{
"status": "unavailable",
"checks": {
  "database": "FAIL: dial tcp 10.0.0.9:5432: connect: connection refused",
  "objectstore": "ok",
  "diskspace": "ok"
}
}
503

Illustrative output

The health check that only proves the process is alive

The default health check people write is GET / returning 200, or a Compose health check of curl -f http://localhost:8080/. It confirms that a socket is listening and a handler ran.

It does not confirm that the database connection pool is alive, that the object store is reachable, that the disk the application writes to has space, or that the migration the last deploy started ever finished.

So the failure mode is: the database goes away, both application hosts keep returning 200 on /, the load balancer keeps both in the pool, and every real request returns 500. Availability monitoring is green. The proxy is doing exactly what it was told. Nobody is paged by the load balancer, because from its point of view nothing is wrong.

The rule that avoids it: the health check must exercise the dependencies the request path uses. If a request needs the database, the readiness check queries the database.

The counter-rule matters just as much: do not check dependencies you can survive without. If the readiness endpoint checks a non-critical downstream API, an outage in that API marks every backend unhealthy simultaneously, the proxy has nothing left to route to, and a partial degradation becomes a total one. Check what you cannot serve without; degrade gracefully on everything else.

Configuring the proxy

HAProxy

Configuration changehaproxy.cfg— Active checks with an explicit expectation, plus a drain-capable runtime socket.

global
  stats socket /run/haproxy/admin.sock mode 660 level admin
  # Bound wait for in-flight sessions on reload, so old workers cannot linger
  hard-stop-after 120s

defaults
  mode http
  timeout connect 5s
  timeout client  60s
  timeout server  60s

backend app_backend
  balance roundrobin
  option httpchk GET /readyz
  http-check expect status 200
  # inter: check every 2s. fall: 3 failed checks removes it.
  # rise: 2 good checks return it. slowstart ramps traffic back gradually.
  server app-01 192.0.2.11:8080 check inter 2s fall 3 rise 2 slowstart 30s
  server app-02 192.0.2.12:8080 check inter 2s fall 3 rise 2 slowstart 30s

The four numbers on the server line are the availability policy, and they are a trade-off in both directions:

inter 2s fall 3 means a dead backend is detected in about six seconds. Lowering fall to 1 detects faster and makes a single dropped packet eject a healthy host. Raising it makes the pool stable and slow to react.
rise 2 stops a flapping backend from being readmitted on one lucky check.
slowstart 30s ramps a returning backend’s weight up over 30 seconds. This is the one people omit and then wonder why a recovered host immediately falls over: a host that has just started has cold caches and an empty connection pool, and sending it half of production instantly is how you get it ejected again. HAProxy has this natively; nginx open source offers slow_start as a server parameter, though its interaction with active checks is a commercial feature.

nginx, and an important limitation

Configuration changenginx upstream— Open source nginx has PASSIVE checks only. Read the caveat below before relying on this.

upstream app_backend {
  # max_fails failures within fail_timeout marks the server unavailable
  # for fail_timeout. This is PASSIVE - it only reacts to real traffic.
  server 192.0.2.11:8080 max_fails=3 fail_timeout=30s;
  server 192.0.2.12:8080 max_fails=3 fail_timeout=30s;
  keepalive 32;
}

server {
  listen 443 ssl;
  server_name app.example.com;

  location / {
      proxy_pass http://app_backend;
      proxy_next_upstream error timeout http_502 http_503 http_504;
      proxy_set_header Host $host;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  }
}

Open source nginx cannot actively health check

This is a real limitation and it is regularly missed. The health_check directive — periodic active probing of an endpoint — is part of NGINX Plus, the commercial subscription. Open source nginx offers only passive health checking through max_fails and fail_timeout: it counts failures that occur while proxying real user requests, and marks a server unavailable for fail_timeout once max_fails is reached within that window.

Three consequences:

Failures are detected by users, not by the proxy. The first max_fails requests to a dead backend fail for real people before it is ejected. proxy_next_upstream mitigates this by retrying the failed request on another server, which is why that directive matters more here than with HAProxy.
A backend cannot be marked healthy again by probing; it is simply retried after fail_timeout expires, with live traffic.
A backend that is up but not ready is invisible. Passive checking sees only what user requests reveal, so /readyz returning 503 is never consulted unless a user happens to hit a path that fails.

If you are choosing a proxy for an availability architecture and want active checks without a subscription, HAProxy, Traefik and Caddy all provide them. Choosing nginx open source is defensible — just do it knowing that detection is user-driven, and set proxy_next_upstream accordingly.

What the proxy is actually connecting to

The backend address in the proxy configuration is the host’s published port, not the container’s. 192.0.2.11:8080 reaches the host’s network stack, where Docker’s port publishing forwards to the container.

For a bridge-network container with -p 8080:8080, the path is a DNAT rule in the DOCKER chain of the nat table, rewriting the destination to the container’s address on docker0 or a user-defined bridge. docker-proxy handles the cases iptables does not cover, notably loopback connections. On Engine 29 with the experimental nftables firewall backend selected via the firewall-backend daemon option, the equivalent rules live in nftables instead — the behaviour is the same, the place you look for the rules is not.

Two operational consequences follow.

The health check tests the whole path. A check against 192.0.2.11:8080 exercises host networking, the NAT rule and the container. That is what you want — it is the same path a user takes. A check against the container’s internal address would pass while port publishing was broken.

Removing the container removes the rule. When a container stops, its DNAT rule goes with it, so the host’s port stops accepting connections and the proxy’s check fails with connection refused rather than timing out. Refused is faster than a timeout, which is why a stopped container is ejected from the pool noticeably quicker than a hung one — and why a hung application is the worse failure. A check with a timeout shorter than the check interval is what stops a hung backend holding a check slot.

Draining, not cutting

Removing a backend from the pool has two meanings and they are not the same operation.

Drain — stop sending new requests; let existing ones finish.
Disable / down — cut the backend out immediately, including in-flight requests.

Going straight to the second for a planned change drops every request currently being served on that host. For a 30-second report or a file upload, that is a user-visible error caused entirely by the maintenance procedure.

Service impact possibledrain a backend— Drain, wait for quiet, then act. Never straight to maint for a planned change.

SOCK=/run/haproxy/admin.sock
BACKEND=app_backend
SERVER=app-01

# 1. Drain: no new sessions, existing ones continue
echo "set server $BACKEND/$SERVER state drain" | sudo socat stdio "$SOCK"

# 2. Wait for the current session count to reach zero
for _ in $(seq 1 60); do
n=$(echo "show stat" | sudo socat stdio "$SOCK"     | awk -F, -v b="$BACKEND" -v s="$SERVER" '$1==b && $2==s {print $5}')
echo "sessions still open: $n"
[ "$n" -eq 0 ] && break
sleep 5
done

# 3. Only now is it safe to stop the stack on that host
# docker compose -f /srv/app/compose.yaml stop

# 4. After maintenance, return it - slowstart ramps the traffic
echo "set server $BACKEND/$SERVER state ready" | sudo socat stdio "$SOCK"

Read-only / Safedraining in progress— Nineteen requests were still in flight when the drain started.

$ show stat | socat stdio /run/haproxy/admin.sock, sampled every 5s

sessions still open: 19
sessions still open: 11
sessions still open: 4
sessions still open: 1
sessions still open: 0

Illustrative output

Those nineteen requests are the ones a state maint would have destroyed. The whole drain took twenty seconds.

Rolling a deploy across two hosts

With draining available, the downtime that docker compose up imposes — it stops and recreates a changed service, with no overlap — becomes invisible to users, because the host is out of the pool while it happens.

Service impact possiblerolling deploy— One host at a time. Verify before returning each to the pool.

SOCK=/run/haproxy/admin.sock
STACK=/srv/app/compose.yaml

for srv in app-01 app-02; do
echo "set server app_backend/$srv state drain" | sudo socat stdio "$SOCK"
sleep 30   # replace with the session-count wait loop above

ssh "$srv" "docker compose -f $STACK pull &&             docker compose -f $STACK up -d --wait --wait-timeout 120"

# Verify locally on that host, bypassing the proxy, BEFORE returning it
ssh "$srv" 'curl -fsS http://127.0.0.1:8080/readyz > /dev/null' || {
  echo "FAIL: $srv did not come back ready - stopping the rollout" >&2
  exit 1
}

echo "set server app_backend/$srv state ready" | sudo socat stdio "$SOCK"
sleep 60   # watch one host in the new version before touching the next
done

Two properties make this a deploy procedure rather than a script. --wait returns non-zero if any service fails to reach healthy, so a broken image stops the rollout at host one instead of taking out both. And the local curl before returning the host to the pool means a host that started but is not serving correctly never receives traffic — the same “verify before repointing” rule as the DR rebuild.

What if the health check passes on both hosts but users still get errors?

The proxy has done its job; the problem is somewhere its model cannot see. Work through these in order, because they are roughly in order of frequency.

The health check is not exercising the failing path. /readyz checks the database; the failure is in the object store used only by uploads. Everything else works. The fix is to widen the readiness check to cover the paths you cannot serve without — and to accept that it will never cover everything, which is why synthetic monitoring of real user journeys exists as a separate layer.

The failure is per-request, not per-host. A specific tenant, a specific route, or a poison record produces 500s regardless of which backend serves it. No health check finds this because every backend is equally affected. Look at the error rate broken down by route and by tenant, not by host.

Sessions are pinned and one host is degraded. If stickiness is in use, users pinned to the bad host see errors while the aggregate error rate stays low and the average looks fine. Break the metrics down by backend before concluding things are healthy.

The proxy itself is the single point of failure. One HAProxy in front of two application hosts moved the availability problem; it did not remove it. Two proxies with a floating address (keepalived / VRRP) or a managed cloud load balancer is the next step, and it is worth being explicit that you have not taken it yet.

Capacity, not availability. One host is out, the survivor is at 100% CPU, and requests are timing out rather than failing. The health check still passes because the check is cheap and the real requests are not. N+1 means each host must be able to carry the full load alone — if two hosts each run at 70%, losing one gives you a 140% demand on a 100% host, which is an outage with a healthy health check.