Lab: HAProxy layer 7 load balancer with health checks

This lab configures HAProxy for production: frontends, backends with health checks, session persistence, TLS termination, and the stats page.

Tasks

Task 1: Set up the test environment

Three hosts:

• lb: the HAProxy load balancer
• web1, web2: the backend web servers

Each web server runs nginx with a /health endpoint:

# /etc/nginx/sites-available/default
server {
    listen 80 default_server;
    server_name _;

    location / {
        return 200 "web1\n";
    }

    location /health {
        access_log off;
        return 200 "ok\n";
    }
}

Task 2: Install HAProxy

On lb:

sudo apt install haproxy

Task 3: Configure HAProxy

Generate the stats password first. Never type one in.

STATS_PW=$(openssl rand -base64 24)
printf '%s\n' "$STATS_PW"   # note it down; it is not stored anywhere else

sudo tee /etc/haproxy/haproxy.cfg <<EOF
global
    log /dev/log local0
    maxconn 4096
    user haproxy
    group haproxy
    daemon

defaults
    log     global
    mode    http
    option  httplog
    option  dontlognull
    timeout connect 5s
    timeout client  30s
    timeout server  30s

frontend http-in
    bind *:80
    default_backend webservers

backend webservers
    balance roundrobin
    cookie SERVERID insert indirect nocache
    option httpchk GET /health
    http-check expect status 200
    server web1 10.0.0.10:80 check inter 2s fall 3 rise 2 cookie web1
    server web2 10.0.0.11:80 check inter 2s fall 3 rise 2 cookie web2

frontend stats
    # loopback only - reach it through an SSH tunnel
    bind 127.0.0.1:8404
    # or, on a dedicated management interface, with TLS:
    # bind 10.0.99.5:8404 ssl crt /etc/haproxy/certs/stats.pem
    stats enable
    stats uri /stats
    stats refresh 10s
    stats auth admin:$STATS_PW
    stats hide-version
EOF

Task 4: Start HAProxy

sudo systemctl enable --now haproxy

Task 5: Verify

From a client:

# Test load balancing
curl http://lb/
# Should return "web1" or "web2"

curl http://lb/
# Should return the other

# Test session persistence
curl -c /tmp/cookies -b /tmp/cookies http://lb/
curl -c /tmp/cookies -b /tmp/cookies http://lb/
# Should return the same backend each time

Task 6: Test backend failover

Stop nginx on web1:

ssh web1 "sudo systemctl stop nginx"

HAProxy should mark web1 as down and route all traffic to web2.

Verify:

curl http://lb/
# Should return "web2" for every request

# Check the stats page - the port is bound to loopback on lb,
# so forward it over SSH rather than opening it to the network
ssh -L 8404:127.0.0.1:8404 lb
# then, in a second terminal on your workstation:
curl -u admin http://127.0.0.1:8404/stats   # prompts for the password
# web1 should be marked down

The tunnel carries the credential inside the SSH session, and it changes nothing on the load balancer — there is no window during which the endpoint is exposed and no temporary change to forget to revert. curl -u admin:password http://lb:8404/stats would have put the credential on the wire in cleartext.

Restart nginx on web1:

ssh web1 "sudo systemctl start nginx"

After the rise threshold (2 successes), web1 returns to the pool.

Task 7: Add TLS

HAProxy’s crt takes one PEM file containing the certificate chain and the private key together (or a directory of such files). Two separate files, which is what openssl req produces, will not work — so concatenate them:

# -subj is not optional here: without it openssl prompts
# interactively and the lab stalls waiting for input.
sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
    -subj "/CN=lb.example.com" \
    -keyout /tmp/lb.key -out /tmp/lb.crt

# One file, certificate first then key, readable only by HAProxy
sudo install -d -m 0750 -o root -g haproxy /etc/haproxy/certs
sudo sh -c 'cat /tmp/lb.crt /tmp/lb.key > /etc/haproxy/certs/lb.pem'
sudo chown root:haproxy /etc/haproxy/certs/lb.pem
sudo chmod 0640 /etc/haproxy/certs/lb.pem
shred -u /tmp/lb.key /tmp/lb.crt

Add the TLS listener; do not replace the plaintext one. The sed form that rewrites bind *:80 leaves the frontend with no HTTP listener at all, so every existing client gets connection refused the moment HAProxy restarts — and the redirect below is what makes port 80 useful anyway:

frontend http-in
    bind *:80
    bind *:443 ssl crt /etc/haproxy/certs/lb.pem
    http-request redirect scheme https unless { ssl_fc }
    default_backend webservers

Validate before restarting. HAProxy refuses to start on a bad config, and a restart is the point at which you lose the listener:

sudo haproxy -c -f /etc/haproxy/haproxy.cfg
sudo systemctl reload haproxy      # reload, not restart: no dropped connections

# Prove it: -k because the certificate is self-signed
curl -kIs https://localhost/ | head -1
curl -Is  http://localhost/  | head -1     # expect 301 to https

Task 8: Document

HAPROXY LAB
==========
Host: lb (10.0.0.100)
Backends: web1 (10.0.0.10), web2 (10.0.0.11)
Algorithm: round robin
Health check: GET /health, expect 200
Session persistence: cookie SERVERID
TLS: cert at /etc/ssl/cert.pem

Tests:
- Load balancing: PASS (web1 and web2 alternate)
- Session persistence: PASS (same backend for same cookie)
- Backend failover: PASS (web1 down, all to web2)
- TLS: PASS

Findings:
- HAProxy configured correctly
- All health checks work
- Failover is automatic