Load balancer for the cluster — keepalived, HAProxy, cloud LB

The Kubernetes cluster has two load-balancing concerns: the control plane LB for the API server (covered in Part LXXIII), and the Service LB for Service-type=LoadBalancer. This lesson walks both, with a focus on Service-type load balancing: cloud-managed LBs for cloud clusters, MetalLB for on-prem.

The two LB concerns

flowchart LR
    C[Clients] -->|kubectl| LB_API[API server LB]
    LB_API --> CP[API servers]
    C -->|HTTPS| LB_SVC[Service LB]
    LB_SVC -->|HTTP| SVC[Service]
    SVC --> P[Pods]

• API server LB. HAProxy/keepalived (on-prem) or cloud LB. Endpoints: API servers.
• Service LB. Cloud LB (most) or MetalLB (on-prem). Endpoints: Service IPs backed by Pods.

The two are independent.

The API server LB recap

For on-prem:

• HAProxy for L4 TCP proxy.
• keepalived for the floating VIP.
• TCP health check on port 6443.

For cloud:

• AWS NLB / GCP LB / Azure LB.

Details in Part LXXIII.

The Service LB — overview

Production services expose themselves via:

• Ingress / Gateway API. HTTP routing at L7.
• Service Type=LoadBalancer. L4 TCP/UDP to a cloud LB.
• Service Type=NodePort. Direct port on each node.
• HostPort / hostNetwork. Direct to host.

The LoadBalancer type is the standard for stateful services (databases, message queues) that need persistent TCP connections.

flowchart LR
    U[User] -->|TCP| LB[Cloud LB]
    LB -->|port-forward| N[Node]
    N -->|kube-proxy| P[Pod]

For cloud: the cloud-managed LB is provisioned when the Service is created.

For on-prem: MetalLB is the standard.

MetalLB

MetalLB is the on-prem Service-LB solution. It uses either Layer 2 (ARP/NDP) or BGP to advertise Service IPs.

apiVersion: v1
kind: ConfigMap
metadata:
  name: metallb
  namespace: metallb-system
data:
  config: |
    address-pools:
    - name: my-pool
      protocol: layer2
      addresses:
      - 10.0.10.0/24

sequenceDiagram
    participant U as User
    participant SVC as Service (type=LoadBalancer)
    participant ML as MetalLB
    participant Net
    U->>SVC: connect to <SVC-IP>
    Net->>SVC: ARP for <SVC-IP>
    ML->>Net: yes, <my-mac>
    Net->>SVC: route to my node
    SVC->>SVC: kube-proxy routes to Pod

MetalLB Layer 2

Layer 2 mode announces the Service IP via gratuitous ARP. The “speaker” node responds; the IP is reachable on the local network.

Pros:

• Simple; works with most switches.
• No BGP-required routers.

Cons:

• One node is the “speaker” for an IP; failover is slower (~10 seconds).
• L2 flooding can be heavy in dense networks.

MetalLB BGP

BGP mode runs a BGP session with the cluster’s routers and announces Service IPs:

data:
  config: |
    peers:
    - peer-address: 10.0.0.1
      peer-asn: 64512
      my-asn: 64512
    address-pools:
    - name: my-pool
      protocol: bgp
      addresses:
      - 10.0.10.0/24

Each MetalLB node runs a BGP speaker; routers see ECMP routes to any node.

Pros:

• ECMP for load distribution.
• Faster failover (BGP convergence).
• Works for large clusters.

Cons:

• Requires BGP-capable routers (e.g., FRR, BIRD, VyOS, Cisco).
• More operational complexity.

The MetalLB install

kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.14.x/config/manifests/metallb-native.yaml

A single YAML installs:

• metallb-system namespace.
• controller (cluster-wide).
• speakers (per-node agents).
• ConfigMap (the address pool + peers).

After install, a Service of type LoadBalancer gets an IP from the pool:

apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  type: LoadBalancer
  selector:
    app: my-app
  ports:
  - port: 80
    targetPort: 8080

kubectl get svc my-service
# NAME         TYPE           CLUSTER-IP    EXTERNAL-IP    PORT(S)
# my-service   LoadBalancer   10.96.0.42    10.0.10.5      80:31789/TCP

The cloud-managed Service LB

Cloud providers integrate with Service-type=LoadBalancer:

# AWS (annotations)
apiVersion: v1
kind: Service
metadata:
  name: my-service
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-name: my-service
    service.beta.kubernetes.io/aws-load-balancer-type: nlb
spec:
  type: LoadBalancer
  ...

When the Service is created, the cloud controller (in kube-controller-manager) provisions an AWS NLB / GCP LB / Azure LB automatically.

$ kubectl get svc my-service -o jsonpath='{.status.loadBalancer.ingress}'

...

The “Service LB matters” reason

Service-type=LoadBalancer is widely used because:

• Stateful services. Databases, message brokers.
• TCP / UDP. Protocol-agnostic.
• Direct connection. No Ingress controller needed.

The alternative (Ingress / Gateway API) is L7 and HTTP only. For raw TCP, the LoadBalancer service is the choice.

The Service LB on a single host (lab / edge)

For lab clusters or edge deployments, MetalLB in single-speaker mode is sufficient:

data:
  config: |
    address-pools:
    - name: my-pool
      protocol: layer2
      addresses:
      - 10.0.10.0/24

A single speaker announces all IPs.

The BGP configuration with FRR / VyOS

On-prem routers (FRR, VyOS, BIRD, Cisco) must accept the BGP advertisements:

# FRR / VyOS
ip bgp <asn>
 neighbor <metallb-ip> remote-as <asn>
 address-family ipv4 unicast
  neighbor <metallb-ip> activate

The router learns the Service IPs from the MetalLB speaker.

The “no Service LB” mistake

Some operators skip the Service LB and rely on Ingress-only. This works for HTTP services but blocks stateful / non-HTTP traffic. For a complete cluster, Service-type=LoadBalancer is necessary.

The drift from design

A cluster’s LB choice is set at build time and rarely changes. Documenting the choice matters:

LB CHOICE FOR PRODUCTION CLUSTER
================================

API server LB:
  - On-prem: HAProxy + keepalived
  - VIP: 10.0.1.254
  - Health check: TCP 6443

Service LB:
  - MetalLB in Layer 2 mode
  - Address pool: 10.0.10.0/24
  - Speaker selection: automatic (one speaker per IP)

Backup plan:
  - For Service LB: BGP mode (router reconfiguration)
  - For API LB: cloud LB (for cloud migration)

The discipline

• Choose LB solutions deliberately. Document the rationale.
• Test failover. LB failures happen; the recovery must work.
• Use DNS where possible. VIP changes are transparent via DNS.
• Document the configuration. Runbook entry on LB layout, failover, restart.
• Verify Service exposure. A Service of type=LoadBalancer without LB is an orphan.

Quiz

Production discipline

• API server and Service LBs are separate. Document each.
• Choose MetalLB for on-prem Service LB. Configure L2 or BGP based on infrastructure.
• Test failover. Both LBs must work under failure.
• Document the LB choice. A runbook entry is essential.
• Verify Service exposure. A Service without a reachable external IP is an orphan.

Load balancers are the cluster’s external interface. Operating them well is making a deliberate choice and keeping it functional.