Runbook: Load balancer failure - recover and restore

This runbook recovers from a load balancer failure. The LB can fail in several ways: the LB process, the network, or all backends. Each has a different recovery.

When to use this runbook

Use this runbook when:

• The LB is down (HAProxy is not responding).
• All backends are down (no healthy backend).
• Traffic is not flowing to the application.
• The VIP is not on any LB.

Inputs

Gather before starting:

• The LB configuration (HAProxy or other).
• The backend list.
• The VIP and DNS.
• Out-of-band access (console, IPMI, cloud).

Procedure

Step 1: Identify the failure

# Check LB status
sudo systemctl status haproxy

# Check VIP
ip addr show | grep <vip>

# Check backends
curl http://<backend>/health

The failure type determines the response:

• LB down: HAProxy process is not running.
• Backends down: HAProxy is up but all backends are marked down.
• VIP not on any LB: keepalived is not working.
• Network issue: traffic is not reaching the LB.

Step 2: If the LB is down

Collect the evidence first. A restart destroys the only record of why the process stopped, and if the cause is still present the restart either fails identically or succeeds and fails again in ten minutes.

# Why did it stop? Read this before touching anything.
sudo journalctl -u haproxy -n 100 --no-pager

# Parse-only. Mutates nothing, starts nothing.
sudo haproxy -c -f /etc/haproxy/haproxy.cfg

# Is anything listening on the front ends?
sudo ss -tlnp | grep -E ':(80|443)'

# Preserve the log before rotation or a restart overwrites the window
sudo cp /var/log/haproxy.log /tmp/incident-$(date +%s).log

Once the journal has given you a cause, restart:

sudo systemctl restart haproxy

# If keepalived is also down, restart it after checking its journal too
sudo journalctl -u keepalived -n 50 --no-pager
sudo systemctl restart keepalived

If HAProxy will not start, do not escalate to rebooting the VM or power-cycling the host. A reboot removes every remaining piece of evidence, and the process failed for a reason that will still be there afterwards. If the host itself is unreachable rather than the service, that is a host incident: fail the VIP over to the standby LB (see the rollback section) and treat the dead host separately.

Verify:

# The VIP should now be on this LB
ip addr show | grep <vip>
curl http://<vip>/health

Step 3: If backends are down

# Check each backend
for backend in web1 web2 web3; do
  curl -s -o /dev/null -w "%{http_code}\n" http://$backend/health
done

Read the pattern before you read the individual hosts.

• Some backends down, the rest healthy: the fault is per-host. Work them one at a time.
• All backends down at once: the fault is almost never in all of them simultaneously. It is shared - a bad deploy, the database, an expired certificate, a DNS change, a configuration push. Restarting the backends cannot fix any of those, and it destroys the evidence needed to find them.

When the cause is shared, prove it from one host rather than guessing across all of them:

# Bypass the LB - talk to a single backend directly
curl -sv http://web1:8080/health

# What does that backend say about itself?
ssh web1 "sudo journalctl -u myservice -n 100 --no-pager"

# Is the shared dependency reachable from the backend?
ssh web1 "sudo ss -tnp state established | head; getent hosts db.internal"

If one backend fails in isolation and the others are fine, diagnose then restart that host, and confirm it before touching the next:

ssh web1 "sudo systemctl status myservice"
ssh web1 "sudo journalctl -u myservice -n 100 --no-pager"

# Restart one host only
ssh web1 "sudo systemctl restart myservice"

# Confirm it is healthy before moving to the next host
curl -s -o /dev/null -w "%{http_code}\n" http://web1:8080/health

One at a time, with a health confirmation between each, is what keeps the remaining capacity serving while you work.

The LB’s health check should mark the backend up after a successful health response.

Step 4: If the VIP is not on any LB

systemctl status keepalived --no-pager
sudo journalctl -u keepalived -n 100 --no-pager   # VRRP transitions: "Entering MASTER STATE"

# Is the VIP actually present on an interface? This is the ground truth.
ip -brief addr show

# Parse-only config check - changes nothing
sudo keepalived --config-test -f /etc/keepalived/keepalived.conf

Run these on both load balancers before concluding anything. The failure modes look identical from one side: a VIP on neither host and a VIP on both hosts are different incidents with different fixes, and you cannot tell them apart from a single node.

Note --no-pager and -n 100 rather than tail -f: a runbook step has to terminate so the next one can run. Keep the follow form for watching a failover you are about to trigger, not for a diagnostic step.

If keepalived is not running, restart it. If the VIP is on neither LB, the issue may be:

• Network partition.
• Both LBs have lower priority.
• Keepalived configuration error.

Step 5: If the issue is the LB configuration

# Check the configuration syntax
sudo haproxy -c -f /etc/haproxy/haproxy.cfg

# Reload
sudo systemctl reload haproxy

If the reload fails, revert to the last known good config:

sudo cp /etc/haproxy/haproxy.cfg.bak /etc/haproxy/haproxy.cfg
sudo systemctl restart haproxy

Step 6: Verify

# LB is up
curl http://<vip>/health

# All backends are healthy
curl -s -o /dev/null -w "%{http_code}\n" http://<vip>/health

# Real traffic works
# Wait for the load balancer to send real traffic
# Or test with curl
curl http://<vip>/

Step 7: Document

In the incident log:

• Time of failure.
• What failed (LB, backends, network).
• Recovery time.
• Improvements (HA pair, more capacity, better monitoring).

Common patterns

Every row is diagnosis first. The middle column is what you must establish before the right-hand column is safe to run.

Knowledge check