HAProxy — TCP and HTTP load balancing · Docker & Containers

nginx is a web server that learned to proxy. HAProxy is a load balancer that learned to speak HTTP. The difference is not marketing: it decides what each one does well, and it is why the previous lesson had a section admitting nginx cannot poll a backend on a timer while this one spends a third of its length on exactly that.

Three capabilities are the reason to reach for HAProxy in front of containers:

Active health checks in the open-source build, on a timer, independent of client traffic.
A Runtime API that changes backend state — drain, maintenance, weight — with no reload and no dropped connection.
L4 (mode tcp) as a first-class citizen, which is what you need in front of a database, a message broker, or TLS you do not want to terminate.

The shape of the file

HAProxy configuration is four kinds of section, and the ordering is conventional rather than enforced:

Section	Holds
`global`	process-wide settings: user, logging, the Runtime API socket, TLS defaults
`defaults`	values inherited by every frontend and backend below it
`frontend`	what to listen on, and how to choose a backend
`backend`	the servers, the health check, and the balancing algorithm

defaults is the section that matters more than it looks, because timeouts have no useful default and HAProxy warns at startup if you omit them. A configuration with no timeout client is not one that picks something sensible; it is one that will hold connections until something else runs out of file descriptors.

A configuration that survives production

Configuration change/usr/local/etc/haproxy/haproxy.cfg— HTTP mode, TLS termination, active checks, DNS-discovered container backends.

global
  log stdout format raw local0 info
  # The Runtime API. 'level admin' allows state changes; see the
  # drain section below. Keep this socket off the network.
  stats socket /var/run/haproxy/admin.sock mode 660 level admin
  stats timeout 30s
  ssl-default-bind-options ssl-min-ver TLSv1.2 no-tls-tickets

defaults
  log     global
  mode    http
  option  httplog
  # None of these four are optional. Omit one and HAProxy warns.
  timeout connect 5s
  timeout client  30s
  timeout server  30s
  timeout http-request 10s
  # Long-lived upgraded connections (WebSocket) use this instead
  # of timeout client/server once the tunnel is established.
  timeout tunnel  1h
  retries 3
  option  redispatch

# Docker's embedded DNS, so backend names track container recreation.
resolvers docker
  nameserver dns1 127.0.0.11:53
  resolve_retries 3
  timeout resolve 1s
  timeout retry   1s
  hold valid    10s
  hold obsolete 5s
  hold nx       5s

frontend https_in
  # 'crt' is ONE file containing cert, chain and key concatenated.
  bind *:443 ssl crt /etc/haproxy/certs/app.example.com.pem alpn h2,http/1.1
  bind *:80

  http-request redirect scheme https unless { ssl_fc }

  # Append the client address. The backend must read the LAST
  # occurrence of this header, never the first.
  option forwardfor

  acl is_api path_beg /api/
  use_backend api_servers if is_api
  default_backend web_servers

backend web_servers
  balance roundrobin

  # Active checks: GET /healthz, accept 200 only.
  option httpchk
  http-check send meth GET uri /healthz ver HTTP/1.1 hdr host app.example.com
  http-check expect status 200

  # 'resolvers docker' re-resolves in the background.
  # 'init-addr none' lets HAProxy start with this server DOWN rather
  # than refusing to start when the container is not up yet.
  # 'resolve-prefer ipv4' because the default is ipv6.
  server-template web 3 web:8080 check inter 2s rise 2 fall 3 \
      resolvers docker init-addr none resolve-prefer ipv4

backend api_servers
  balance leastconn
  option httpchk
  http-check send meth GET uri /api/healthz ver HTTP/1.1 hdr host app.example.com
  http-check expect status 200
  server-template api 3 api:9090 check inter 2s rise 2 fall 3 \
      resolvers docker init-addr none resolve-prefer ipv4

Active health checks, and the arithmetic behind them

check inter 2s rise 2 fall 3 is four decisions in one line. The defaults, if you write only check, are inter 2000ms, rise 2, fall 3.

Parameter	Default	Meaning
`inter`	`2000ms`	interval between two consecutive checks
`fall`	`3`	consecutive failures before the server is DOWN
`rise`	`2`	consecutive successes before a DOWN server is UP again
`fastinter`	unset	interval while transitioning either way
`downinter`	unset	interval while the server is fully DOWN

The number that matters operationally is not any of them alone:

Worst-case detection time = inter × fall. With the defaults that is 2s × 3 = 6 seconds during which requests are still being sent to a dead backend.

Six seconds of errors is fine for a web application and unacceptable for a payment path. Shorten it with fastinter, which applies the moment a check first fails rather than to every check:

Configuration changefaster detection without more idle load— Normal cadence 5s; once something looks wrong, 500ms.

server web1 web-a:8080 check inter 5s fastinter 500ms downinter 5s \
  rise 2 fall 3

That is 5-second polling while everything is healthy, and worst-case detection of roughly 500ms + 2 × 500ms = 1.5 seconds once the first check fails — at a tenth of the steady-state check load of inter 1s.

The health check passes and the application is broken

The single most common way a health check lies is by not checking anything. Written as option httpchk with nothing else, HAProxy sends OPTIONS / and accepts any 2xx or 3xx — so a backend that answers 301 from a catch-all redirect, or a framework that returns 200 for every unmatched route, is permanently healthy no matter what state its database connection is in.

It is worse with check on its own: with no option httpchk at all, the check is a bare TCP connect. The listening socket answers, so the server is UP — which means a process that has deadlocked every worker thread but still has a listening socket is, as far as HAProxy is concerned, fine.

The fix is to be specific in both directions: name the URI with http-check send, and name the acceptable status with http-check expect status 200. Then make the endpoint behind it exercise the dependency it claims to cover — which is the subject of the health-check lesson later in this course, and the same mistake in a different tool.

Discovering containers by DNS

This is HAProxy’s answer to the problem that made the previous lesson 502: a backend name whose address changes on every deploy.

What server-template and init-addr actually do at startup

HAProxy’s internal model is a fixed array of servers, allocated at startup. It has no concept of adding one later. server-template web 3 web:8080 therefore does exactly what the manual says it does: it creates three server slots named web1, web2, web3, all pointed at the same FQDN, sharing one set of parameters. The resolvers section fills those slots from the DNS answer — Docker’s embedded DNS returns one A record per container behind that service name — and re-fills them as the answer changes.

The consequences follow directly from “fixed array”:

The count is a ceiling. Scale to five replicas behind a server-template web 3 and two of them receive no traffic at all, silently. Size the template above your maximum replica count.
init-addr decides what happens when the name does not resolve at startup. The default is last,libc: try the state file, then the libc resolver, and throw an error if neither works. In Compose that means HAProxy refuses to start whenever it comes up before the application container — which is a startup-order race you will lose on a host reboot. init-addr none starts the server in a DOWN state with no address, and the active check brings it up seconds later.
resolve-prefer defaults to ipv6. On a Docker network that is usually IPv4-only, so the default is harmless; on a dual-stack network it will quietly prefer an address the application may not be listening on. Write resolve-prefer ipv4 rather than inherit it.
hold decides how long a bad answer is tolerated. hold nx 5s means an NXDOMAIN — which is exactly what you get in the second between the old container going away and the new one registering — does not immediately mark the server down.

Read-only / Safeprove discovery is working— Slot count, resolved addresses, and the DNS counters HAProxy keeps.

PROXY=proxy
SOCK=/var/run/haproxy/admin.sock

# Which slots exist and what address is in each one
docker compose exec -T "$PROXY" sh -c "echo 'show servers state' | socat stdio $SOCK"

# HAProxy's own view of the resolvers section: sent, valid, nx, timeout
docker compose exec -T "$PROXY" sh -c "echo 'show resolvers docker' | socat stdio $SOCK"

A non-zero and climbing nx counter with a flat valid counter means the name is not resolving — the backend is on a different network, or the service name is misspelled. That distinction is invisible from the error pages, which say 503 either way.

502, 503, 504 — HAProxy tells you which

HAProxy is unusually precise here, and the manual is explicit about what each code means:

Code	HAProxy’s definition	What to go and look at
`502`	the server returned an empty, invalid or incomplete response	the application crashed mid-response, or is speaking something that is not HTTP on that port
`503`	no server was available to handle the request	every server is DOWN — a health-check or discovery problem, not an application one
`504`	the response timeout struck before the server responded	the server is alive and too slow; compare against `timeout server`

The one that gets misread is 503. It is not “the backend returned an error”; it is “there was no backend to ask”. When you see 503 the application logs will be empty, because no request ever reached it, and half an hour disappears looking for an error that was never written. Go straight to show stat instead.

Read-only / Safewhich backends are up— One line per server with its status and the reason it changed.

PROXY=proxy
SOCK=/var/run/haproxy/admin.sock

docker compose exec -T "$PROXY" sh -c "echo 'show stat' | socat stdio $SOCK" | awk -F, 'NR==1 || $2!~/^(FRONTEND|BACKEND)$/ {print $1, $2, $18, $37, $38, $39}'

Read-only / Safereading check_status— One line per server. The check status is the diagnosis, not the UP/DOWN column.

$ echo 'show stat' | socat stdio /var/run/haproxy/admin.sock | cut -d, -f1,2,18,37,38

# pxname,svname,status,check_status,check_code
web_servers,web1,UP,L7OK,200
web_servers,web2,DOWN,L7STS,503
web_servers,web3,DOWN,L4CON,
api_servers,api1,UP,L7OK,200
api_servers,api2,DOWN,L7TOUT,

Illustrative output

The check_status column is the answer, and each value points somewhere different:

Value	The check	Go and look at
`L7OK`	answered with an accepted status	nothing
`L7STS`	answered, status rejected — `check_code` has it	the application; it is up and unhealthy
`L4CON`	the TCP connection was refused	the container is down, or on the wrong port
`L4TOUT`	the connection attempt timed out	networking — no route, or packets dropped
`L7TOUT`	connected, then no response within `timeout check`	the application is hanging, not erroring

L4CON and L7STS are the two that get conflated, and they are opposite problems: one means nothing is listening, the other means something is listening and telling you it is not well.

Timeouts

The four in defaults above cover four genuinely different waits, and mismatching them against the application is what produces confusing 504s.

Directive	Bounds
`timeout connect`	establishing the TCP connection to the server
`timeout client`	client-side inactivity
`timeout server`	server-side inactivity — most importantly, how long the server may take to start sending response headers
`timeout http-request`	receiving a complete request from the client (slow-loris defence)
`timeout tunnel`	inactivity once a connection has been upgraded, e.g. WebSocket

timeout server is the one to size deliberately: the manual’s advice is to start from what you would consider an unacceptable response time and check the log distribution. Setting it far above the application’s own request timeout means HAProxy waits patiently for a request the application has already abandoned; setting it below means HAProxy returns 504 for requests that would have succeeded, and the application log shows a completed 200 for a request the user saw fail. That disagreement — 504 at the edge, 200 in the application — is the signature of a proxy read timeout shorter than the application’s work.

Draining without a reload

This is HAProxy’s headline advantage over nginx, and it is one line.

Service impact possibledrain, deploy, return— No reload, no configuration edit, no dropped connection.

PROXY=proxy
SOCK=/var/run/haproxy/admin.sock
TARGET=web_servers/web1

runtime() { docker compose exec -T "$PROXY" sh -c "echo '$1' | socat stdio $SOCK"; }

# 1. Stop sending NEW sessions here; existing ones finish.
runtime "set server $TARGET state drain"

# 2. Wait for the current session count to reach zero.
until runtime 'show stat' | awk -F, '$1=="web_servers" && $2=="web1" {exit ($5==0)?0:1}'; do
sleep 2
done

# 3. Now it is safe to replace the container.
docker compose up -d --no-deps web

# 4. Return it. 'ready' re-enables checks; it comes back UP after 'rise'.
runtime "set server $TARGET state ready"

The client address, in both modes

In mode http it is one line — option forwardfor in the frontend — with one rule attached to it that is easy to get wrong:

HAProxy appends X-Forwarded-For to the existing header list. The manual is explicit: the server must use only the last occurrence, “since it is really possible that the client has already brought one”.

An application that reads the first X-Forwarded-For value is reading whatever the client chose to send. option forwardfor if-none adds the header only when one is not already present, which is the correct choice for an inner proxy that trusts the outer one — and exactly the wrong choice at the internet edge, where you want to overwrite. The client-identity lesson later in this part covers the trust boundary properly.

In mode tcp there is no header to add, because there is no HTTP. That is what the PROXY protocol exists for:

Configuration changeL4 in front of PostgreSQL, with client address preserved— No HTTP anywhere. The address travels in a preamble on the TCP connection itself.

frontend pg_in
  bind *:5432
  mode tcp
  option tcplog
  default_backend pg_servers

backend pg_servers
  mode tcp
  balance leastconn
  option pgsql-check user haproxy
  # send-proxy-v2 prepends a PROXY protocol v2 header describing the
  # original layer 3/4 addresses. The receiver must expect it.
  server pg1 pg-a:5432 check send-proxy-v2 inter 2s rise 2 fall 3
  server pg2 pg-b:5432 check send-proxy-v2 inter 2s rise 2 fall 3 backup

Note also that option pgsql-check is a real protocol-level check — HAProxy speaks enough of the PostgreSQL startup handshake to know the server answered as a database. A bare TCP connect would not distinguish a live PostgreSQL from a port that something else grabbed.

TLS: one file, concatenated

HAProxy will not start and the certificate is definitely there

bind *:443 ssl crt /etc/haproxy/certs/app.example.com.pem and HAProxy exits with a message about being unable to load the SSL certificate or find a private key. The file exists. The permissions are right. You have just come from nginx, where ssl_certificate and ssl_certificate_key are two separate directives.

HAProxy’s crt takes one PEM file containing the certificate, any intermediates, and the private key, concatenated. The manual gives the recipe directly: cat cert.pem key.pem > combined.pem. A Let’s Encrypt directory has fullchain.pem and privkey.pem as two files, so the conversion step is mandatory and easy to forget in the renewal hook.

If the file has no private key in it, HAProxy falls back to looking for the same path with .key appended — which is why some setups appear to work without concatenation and then break when someone renames a file.

Configuration changebuild the combined PEM and reload— Order matters less than completeness: leaf, chain, then key is conventional.

set -euo pipefail
DOMAIN=app.example.com
LIVE=/etc/letsencrypt/live/$DOMAIN
OUT=/etc/haproxy/certs/$DOMAIN.pem

sudo install -d -m 0750 /etc/haproxy/certs
sudo sh -c "cat $LIVE/fullchain.pem $LIVE/privkey.pem > $OUT.tmp"
sudo chmod 0640 "$OUT.tmp"
sudo mv "$OUT.tmp" "$OUT"

# Validate before you make it live. -c checks the config and exits.
docker compose exec -T proxy haproxy -c -f /usr/local/etc/haproxy/haproxy.cfg

# Reload. HAProxy starts a new process and lets the old one finish.
docker compose kill -s HUP proxy

Writing to a temporary file and moving it into place is not fussiness. crt pointed at a file that is half-written when HAProxy reads it fails the same way a missing key does, and the window is exactly as long as cat takes.

When HAProxy is the right choice

Situation	HAProxy	Why
L4 in front of a database or broker	yes	`mode tcp` plus protocol-aware checks (`pgsql-check`, `mysql-check`, `redis-check`)
Backends must be drained on every deploy	yes	Runtime API, no reload
You need detection under a second	yes	`fastinter`
You want automatic certificates	no	no ACME client; pair with certbot, or use Caddy or Traefik
Containers appear and disappear by label	no	Traefik reads labels; HAProxy needs DNS or a config generator
The proxy must also serve static files	no	HAProxy does not serve content
The team has never seen an HAProxy config	consider carefully	the language is precise and unforgiving, and that is a real cost

What if the health check endpoint is on a different port from the traffic?

This is common and correct — an application that serves users on 8080 and exposes /healthz on a separate admin port 9000 so the check cannot be reached from outside. HAProxy handles it with port on the server line:

server web1 web-a:8080 check port 9000 inter 2s

Two things change quietly when you do that, and both have bitten people:

PROXY protocol stops applying to checks. The manual is explicit: the PROXY protocol is enabled for health checks automatically if it is enabled for normal traffic and no port or addr directive is present. Add check port and the checks stop sending it, so a backend that requires the preamble now sees every check as a malformed connection and every server goes DOWN. check-send-proxy forces it back on.
You are no longer checking the path traffic takes. A check against port 9000 proves the admin listener is alive. It says nothing about whether the worker pool serving 8080 has deadlocked. That is the false-green problem again, one layer down.

The undo is to remove check port and reload. Nothing persists.

Knowledge check

Where next

Traefik takes the opposite approach to discovery: instead of asking DNS, it watches the Docker daemon directly. That is genuinely convenient and it is also a privilege grant, which the next lesson takes seriously.