HAProxy configuration - the production-grade layer 7 LB

HAProxy is the production-grade layer 7 load balancer. This lesson covers the configuration for production use.

Install

sudo apt install haproxy

Basic configuration

# /etc/haproxy/haproxy.cfg
global
    log /dev/log local0
    maxconn 4096
    user haproxy
    group haproxy
    daemon

defaults
    log     global
    mode    http
    option  httplog
    option  dontlognull
    option  forwardfor except 127.0.0.0/8
    option  redispatch
    retries 3
    timeout connect          5s
    timeout client          30s
    timeout server          30s
    timeout http-request    10s
    timeout http-keep-alive 10s
    timeout queue           30s

frontend http-in
    bind *:80
    bind *:443 ssl crt /etc/ssl/private.pem
    redirect scheme https if !{ ssl_fc }
    default_backend webservers

backend webservers
    balance roundrobin
    option httpchk GET /health
    http-check expect status 200
    server web1 10.0.0.10:80 check inter 2s fall 3 rise 2
    server web2 10.0.0.11:80 check inter 2s fall 3 rise 2
    server web3 10.0.0.12:80 check inter 2s fall 3 rise 2

Four lines in that defaults block are the difference between a demo and something you leave running.

timeout http-request 10s bounds how long a client may take to send a complete request header. Without it, timeout client is your only defence, and timeout client is an inactivity timeout: a client that sends one byte every twenty seconds never triggers it. That is Slowloris. A few thousand such connections exhaust maxconn and the load balancer stops accepting anyone, with no attack traffic worth the name and nothing in the logs but healthy-looking open connections. timeout http-request aborts them regardless of how slowly they type.

option forwardfor inserts X-Forwarded-For. Without it every backend log line, every rate limiter and every geo-IP rule sees the load balancer’s address, and you lose the ability to identify a client at all. except 127.0.0.0/8 skips the header for local health checks.

option redispatch with retries 3 lets a request that fails to connect be retried against a different server. Without redispatch, the retries all go back to the same dead backend and the client gets the error anyway.

Frontends

A frontend listens for traffic and routes to a backend:

frontend http-in
    bind *:80
    default_backend webservers

For TLS termination:

frontend https-in
    bind *:443 ssl crt /etc/ssl/private.pem
    http-request set-header X-Forwarded-Proto https
    default_backend webservers

For multiple domains:

frontend http-in
    bind *:80
    acl is_app hdr(host) -i app.example.com
    acl is_api hdr(host) -i api.example.com
    use_backend app if is_app
    use_backend api if is_api
    default_backend app

Backends

A backend is a pool of servers:

backend webservers
    balance roundrobin
    option httpchk GET /health
    http-check expect status 200
    server web1 10.0.0.10:80 check
    server web2 10.0.0.11:80 check
    server web3 10.0.0.12:80 check

The check keyword enables health checks. The httpchk GET /health and expect status 200 define the check.

Health checks

HAProxy health checks:

• httpchk: HTTP health check (default GET /).
• tcp-check: TCP connect.
• mysql-check: MySQL ping.
• redis-check: Redis PING.
• Custom scripts via external-check.

For HTTP:

# The modern form: send an explicit request, then assert on it
http-check send meth GET uri /health
http-check expect status 200

The check runs every inter seconds (default 2s). After rise successes, the server is marked up. After fall failures, it is marked down.

The <match> keyword after http-check expect is one of status, rstatus, hdr, fhdr, string or rstring. Those are the only ones. header and body look plausible, read naturally, and are not keywords - HAProxy rejects the line at parse time and, because a reload of a bad configuration is refused, you find out during the change window:

# Assert on a response header - the keyword is hdr, not header
http-check expect hdr name Content-Type value -m sub application/json

# Assert on the response body - the keyword is string, not body
http-check expect string OK

Session persistence

For cookie-based persistence:

backend webservers
    cookie SERVERID insert indirect nocache
    server web1 10.0.0.10:80 cookie web1 check
    server web2 10.0.0.11:80 cookie web2 check

HAProxy sets a cookie that pins the client to a specific backend. Subsequent requests go to the same backend.

For IP-based persistence (simpler):

backend webservers
    balance source
    hash-type consistent
    server web1 10.0.0.10:80 check
    server web2 10.0.0.11:80 check

Same source IP always to the same backend. Useful for applications with session state on the backend.

hash-type consistent is the line that makes this survivable. The default is map-based, which selects a server by its position in a static array of live servers - so when one backend of three goes down, the array changes shape and every client is rehashed, not just the third that was on the failed node. For a session-affinity backend that means every logged-in user is thrown onto a server that has never seen them, from a single health-check failure. Consistent hashing moves only the clients that were on the departed server.

If you can, prefer cookie-based persistence over balance source entirely: source hashing puts every client behind a single NAT gateway on the same backend, which is a load distribution problem that only shows up in production.

Monitoring

HAProxy exposes a stats page. It is genuinely useful and it is also the most over-exposed endpoint on a typical load balancer, so configure it defensively from the start:

frontend stats
    # loopback only - reach it through an SSH tunnel
    bind 127.0.0.1:8404
    # or, on a dedicated management interface, with TLS:
    # bind 10.0.99.5:8404 ssl crt /etc/haproxy/certs/stats.pem

    stats enable
    stats uri /stats
    stats refresh 10s
    stats realm HAProxy\ Statistics
    stats auth admin:__REPLACE_WITH_GENERATED_SECRET__
    stats hide-version

    # defence in depth: even on a management interface, scope the source
    acl from_mgmt src 10.0.99.0/24
    http-request deny unless from_mgmt

Reach a loopback-bound stats page over a tunnel rather than opening the port:

ssh -L 8404:127.0.0.1:8404 lb.example.com
# then browse http://127.0.0.1:8404/stats on your workstation

Generate the password rather than typing one:

openssl rand -base64 24

For Prometheus integration, prefer HAProxy’s built-in Prometheus endpoint over scraping the HTML page, and scope it the same way:

frontend metrics
    bind 127.0.0.1:8405
    http-request use-service prometheus-exporter if { path /metrics }
    http-request deny

Production discipline

• Use the latest stable version.
• Set appropriate timeouts.
• Use health checks.
• Monitor the stats page — bound to loopback or a management address, behind TLS, with a generated credential and a source ACL. Never bind *:8404.
• Validate with haproxy -c -f before every reload.
• Test failover (stop a backend, verify traffic moves).
• Log to a central system.

Knowledge check