Health checks and persistence - the safety net of load balancing

A load balancer without health checks sends traffic to broken backends. Without session persistence, users get logged out on every request. This lesson covers both.

Active vs passive health checks

Active health check: the LB actively probes each backend. Examples: TCP connect, HTTP GET /health, custom script.

Passive health check: the LB observes real traffic. If a backend returns errors, it is marked down. Examples: HTTP 5xx, connection refused, timeout.

Active: proactive, catches silent failures. Adds load on backends.

Passive: lightweight, but only catches failures that the LB sees in real traffic.

For production, use both: active checks for routine verification, passive for real-time detection.

Configure active checks in HAProxy

backend webservers
    option httpchk
    http-check send meth GET uri /health hdr Host example.com
    http-check expect status 200
    server web1 10.0.0.10:80 check inter 2s fall 3 rise 2
    server web2 10.0.0.11:80 check inter 2s fall 3 rise 2

• inter 2s: check every 2 seconds.
• fall 3: after 3 failures, mark down.
• rise 2: after 2 successes, mark up.

Configure passive checks

Passive checks are typically the LB’s default behaviour: if a backend fails to respond, the LB marks it down after a timeout.

For HAProxy:

option httpchk
http-check expect status 200

# Mark down on errors
server web1 10.0.0.10:80 check on-error mark-down

Session persistence

Some applications need a client to reach the same backend for the duration of a session (e.g. session state on the backend).

Cookie-based persistence

The LB sets a cookie that pins the client to a backend:

backend webservers
    cookie SERVERID insert indirect nocache
    server web1 10.0.0.10:80 cookie web1 check
    server web2 10.0.0.11:80 cookie web2 check

The client receives Cookie: SERVERID=web1. Subsequent requests with that cookie go to web1.

IP-based persistence

Same source IP goes to the same backend:

backend webservers
    balance source
    server web1 10.0.0.10:80 check
    server web2 10.0.0.11:80 check

Simpler but less precise (NAT, mobile clients).

Sticky session timeout

How long does the stickiness last? After the timeout, the client may be re-routed:

cookie SERVERID insert indirect nocache maxlife 30m

Trade-offs

For stateless applications, no persistence is best. For stateful applications, choose the persistence that matches the app.

Knowledge check