Load balancer — HAProxy, keepalived, MetalLB, cloud LB

The load balancer is the kubeconfig’s connection endpoint. Two patterns dominate: HAProxy + keepalived for on-prem clusters, and cloud-managed LBs for cloud clusters. MetalLB is a separate concern (advertising Service IPs to the local network). This lesson walks the LB configurations and the production discipline.

The LB role

flowchart LR
    C[kubectl / clients] --> VIP[VIP: api.example:6443]
    VIP --> LB[Load balancer]
    LB --> CP1[API server cp-1:6443]
    LB --> CP2[API server cp-2:6443]
    LB --> CP3[API server cp-3:6443]

The kubeconfig points at the LB’s VIP (or DNS name). The LB routes traffic to one of the API server instances.

$ kubectl config view --minify | grep server

- server: https://api.example:6443

HAProxy for the API server

HAProxy is a fast, reliable TCP/HTTP load balancer:

# /etc/haproxy/haproxy.cfg
global
    log /dev/log local0
    maxconn 4096
    user haproxy
    group haproxy
    daemon

defaults
    log     global
    mode    tcp
    option  tcplog
    timeout connect 5s
    timeout client  30s
    timeout server  30s

frontend k8s-api
    bind *:6443
    mode tcp
    default_backend k8s-api-servers

backend k8s-api-servers
    mode tcp
    balance roundrobin
    option ssl-hello-chk
    server cp-1 10.0.1.10:6443 check inter 5s fall 3 rise 2
    server cp-2 10.0.1.11:6443 check inter 5s fall 3 rise 2
    server cp-3 10.0.1.12:6443 check inter 5s fall 3 rise 2

The option ssl-hello-chk is a TCP-level TLS handshake health check; HAProxy does not decrypt and cannot verify the API server’s response.

keepalived for the floating VIP

keepalived provides the VIP that the kubeconfig references:

# /etc/keepalived/keepalived.conf
vrrp_script check_haproxy {
    script "/usr/bin/killall -0 haproxy"
    interval 2
    weight 2
}

vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 100
    advert_int 1
    virtual_ipaddress {
        10.0.1.254
    }
    track_script {
        check_haproxy
    }
}

The VIP (10.0.1.254) is held by the master keepalived node; on failure, the backup takes over.

sequenceDiagram
    autonumber
    participant M as Master node
    participant B as Backup node
    Note over M: VIP 10.0.1.254 active
    M->>M: heartbeat
    M-->>M: track script fails
    M-->>M: state BACKUP
    B->>B: state MASTER
    Note over B: VIP 10.0.1.254 active

The failover time is typically 3-10 seconds.

The two-node HAProxy + keepalived setup

For an HA LB:

• Two hosts run both HAProxy and keepalived.
• One is MASTER (initial state), one BACKUP.
• MASTER holds the VIP.
• On MASTER failure, BACKUP takes the VIP.

# On both nodes:
sudo systemctl enable haproxy keepalived

# On master:
# priority 100

# On backup:
# priority 50

Production: 2 LB hosts is enough; 3 is rare for the LB layer.

Cloud load balancers

AWS Network Load Balancer

# Create a target group of API server IPs
aws elbv2 create-target-group \
  --name k8s-api \
  --protocol TCP \
  --port 6443 \
  --vpc-id vpc-... \
  --health-check-protocol TCP \
  --health-check-port 6443

# Create the NLB
aws elbv2 create-load-balancer \
  --name k8s-api \
  --type network \
  --subnets subnet-... subnet-... \
  --scheme internal

# Register targets
aws elbv2 register-targets \
  --target-group-arn ... \
  --targets Id=i-... Id=i-... Id=i-...

The NLB health-check on TCP port 6443 ensures unhealthy API servers are removed.

GCP TCP Load Balancer

# Create an instance group
gcloud compute instance-groups managed create k8s-api \
  --zone us-east1-a \
  --template ...

# Create a backend service
gcloud compute backend-services create k8s-api-backend \
  --load-balancing-scheme EXTERNAL \
  --protocol TCP \
  --health-checks k8s-api-health

# Create the LB
gcloud compute forwarding-rules create k8s-api \
  --load-balancing-scheme EXTERNAL \
  --ports 6443 \
  --backend-service k8s-api-backend

Azure Load Balancer

Standard SKU LB with TCP health checks.

MetalLB for Service LoadBalancer

MetalLB provides LoadBalancer services on bare metal clusters:

apiVersion: v1
kind: ConfigMap
metadata:
  name: metallb
  namespace: metallb-system
data:
  config: |
    address-pools:
    - name: my-pool
      protocol: layer2
      addresses:
      - 10.0.10.0/24

MetalLB announces Service IPs to the local network via ARP / NDP / BGP.

sequenceDiagram
    participant U as User
    participant SVC as Service (type=LoadBalancer)
    participant ML as MetalLB
    participant Net
    U->>SVC: connect to <SVC-IP>
    Net->>SVC: ARP for <SVC-IP>
    ML->>Net: yes, <my-mac>
    Net->>SVC: route to my node
    SVC->>SVC: kube-proxy routes to Pod

MetalLB is for Service IPs, not for the API server VIP. The two concerns are separate.

The DNS for the LB

The kubeconfig typically references a DNS name rather than a raw IP:

# In /etc/hosts (for tests) or DNS (production)
10.0.1.254 api.cluster.example

DNS records with low TTL enable clean failover when the VIP moves.

The LB health-check tuning

Tighter health checks = faster failover = more LB chatter. A balance is 5-second interval, 3 unhealthy threshold.

$ haproxy -c -f /etc/haproxy/haproxy.cfg

Configuration file is valid

Production discipline

The LB is the cluster’s front door. Operating it well is keeping clients able to reach the API server.

• Test failover. Kill an LB node and verify the VIP moves and that clients still reach the API server.
• Tighten health checks. 5-second interval, 3 failures for unhealthy — tight, but not so tight as to flag transient blips.
• Use DNS for kubeconfig. The DNS name survives VIP changes, and a low TTL enables clean failover.
• Document the LB configuration. Runbook entry on the LB layout, failover, health checks, and restart.
• Distinguish from MetalLB. The API LB and the Service LB (MetalLB) are separate concerns.

Quiz