kube-apiserver HA and load balancers — the cluster's front door at scale

Three API servers behind a load balancer turn a total outage into a partial one, and a partial one is easy to miss: with one instance down, kubectl still works and nothing announces that the cluster is one failure away from being unreachable. The failures move to the layer in front — a virtual IP that keepalived has not re-elected, a health check that passes while the backend refuses connections, or a partition that leaves two API servers serving different views of the same cluster. Diagnosis therefore starts by asking each API server for its own health directly, rather than through the address every client uses.

The HA topology

An HA cluster has multiple API server instances, each on a separate control-plane node. The instances are fronted by a load balancer (e.g., HAProxy, NGINX, cloud LB).

flowchart LR
    A[kubectl] --> B[Load balancer]
    B --> C[API server 1]
    B --> D[API server 2]
    B --> E[API server 3]
    C --> F[etcd 1]
    C --> G[etcd 2]
    C --> H[etcd 3]

The HA topology is the cluster’s front door at scale.

The diagnostic

The canonical diagnostic:

# Substitute your own values before running:
LB=192.0.2.10
API1=192.0.2.11
API2=192.0.2.12
API3=192.0.2.13
CP_NODE=cp-01.example.com

# 1. Check the load balancer
curl -k "https://$LB:6443/healthz"

# 2. Check each API server
curl -k "https://$API1:6443/healthz"
curl -k "https://$API2:6443/healthz"
curl -k "https://$API3:6443/healthz"

# 3. Check the API server's logs
kubectl logs -n kube-system -l component=kube-apiserver --tail=200

# 4. Check the etcd
ssh "$CP_NODE" "ETCDCTL_API=3 etcdctl endpoint health"

The diagnostic is the load balancer, the API servers, and the etcd.

Common failures

• Load balancer is failing. The load balancer cannot reach the API servers. The remediation is to fix the load balancer.
• API server is failing. One or more API servers are failing. The remediation is to restart the affected static Pods, one node at a time.
• Split brain. The API servers are partitioned; each is serving its own cluster view. The remediation is to fix the network.
• Keepalived is failing. The keepalived is failing to elect a master. The remediation is to fix the keepalived.

flowchart TD
    A[HA failing] --> B{Load balancer OK?}
    B -->|No| C[Fix the load balancer]
    B -->|Yes| D{API servers OK?}
    D---|No| E[Restart the API servers]
    D---|Yes| F{Split brain?}
    F -->|Yes| G[Fix the network]
    F -->|No| H{Keepalived OK?}
    H -->|No| I[Fix the keepalived]
    H -->|Yes| J[Unknown]

The remediation

The remediation depends on the cause:

# Option 1: Restart the load balancer
systemctl restart haproxy

# Option 2: Restart one API server.
# haproxy and keepalived are ordinary systemd services, but
# kube-apiserver is a static Pod: move its manifest out of the
# kubelet's watch directory and back again on that node only.
mv /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/
# wait for the kubelet's fileCheckFrequency, 20s by default
mv /tmp/kube-apiserver.yaml /etc/kubernetes/manifests/

# Option 3: Fix the network
# (cluster-specific)

# Option 4: Restart the keepalived
systemctl restart keepalived

The remediation is the HA recovery.

The API server’s view

The API server’s view is the cluster’s state. The view is stored in etcd. The HA cluster has multiple etcd instances and the API server is the gateway.

# Check the etcd's view
ETCDCTL_API=3 etcdctl get /registry --prefix --keys-only | head -20

The etcd is the cluster’s authoritative state.

Production discipline

An HA failure is the cluster’s hypothesis. The discipline is to walk the 11-step methodology applied to the control plane, identify the cause, apply the remediation. The HA is the cluster’s front door at scale; the remediation is the HA recovery.

• Check the load balancer. The load balancer is the cluster’s front door.
• Check the API servers. The API servers are the cluster’s gateway.
• Check the etcd. The etcd is the cluster’s state.

Quiz