API server HA — multiple instances, load balancing, failover

A single API server is a single point of failure. Every production cluster runs at least 2 (often 3) API server instances behind a load balancer, so that no single host failure takes the cluster down. This lesson walks the HA topology, the load balancer configuration, and the client-side behaviour during failover.

The HA topology

flowchart LR
    C[Clients] --> LB[Load balancer VIP]
    LB --> AS1[API server cp-1]
    LB --> AS2[API server cp-2]
    LB --> AS3[API server cp-3]
    AS1 --> E[etcd cluster]
    AS2 --> E
    AS3 --> E

The client’s view is the load balancer’s IP/port. The load balancer routes to one of the three API server instances. Each instance is independent; each has its own in-memory state. The cluster’s API server traffic spreads across them.

Why HA matters

A single API server is fragile:

• Hardware failure. If the host dies, the entire cluster is unreachable.
• Software failure. If the API server process crashes, the cluster is unreachable.
• Maintenance. Every upgrade, certificate renewal, or restart is a cluster-wide outage.

HA addresses all three.

$ kubectl get --raw='/livez'

ok

# Each API server returns its hostname
curl -k https://api.example/version | jq '.hostname'
# Different across calls (round-robin)

A single API server also affects watch: a watcher connected to the dying instance loses its connection; the client reconnects. Failover at the load balancer layer minimises the disruption.

The load balancer

The load balancer is the cluster’s API gateway. Two common patterns:

HAProxy + keepalived (on-prem)

# /etc/haproxy/haproxy.cfg
frontend k8s-api
    bind *:6443
    mode tcp
    default_backend k8s-api-servers

backend k8s-api-servers
    mode tcp
    balance roundrobin
    option ssl-hello-chk
    server cp-1 10.0.1.10:6443 check
    server cp-2 10.0.1.11:6443 check
    server cp-3 10.0.1.12:6443 check

The option ssl-hello-chk is the only viable health check for TLS; HAProxy does not decrypt.

# keepalived config
vrrp_script check_haproxy {
    script "killall -0 haproxy"
    interval 2
}
vrrp_instance VI_1 {
    state MASTER
    interface eth0
    virtual_router_id 51
    priority 100
    virtual_ipaddress {
        10.0.1.254
    }
    track_script {
        check_haproxy
    }
}

keepalived provides the floating VIP that the kubeconfig points to. Two haproxy+keepalived instances fail over the VIP.

Cloud-managed LB

• AWS NLB / ALB. NLB is L4; the target group contains the API server IPs.
• GCP HTTPS load balancer. L7; target pool contains the API servers.
• Azure Load Balancer. Standard SKU.

Each cloud LB comes with its own health check options; the production rule is “TCP on port 6443”.

Health checks

The health check defines whether the API server is in the load balancer’s pool:

# Standard API server health check
curl -k https://api.example/livez
# Returns "ok" if the API server is healthy

curl -k https://api.example/readyz
# Returns "ok" if the API server is ready to serve

# /livez is for liveness (load-balancer pool membership)
# /readyz is for readiness (the API server is past startup)

The load balancer’s check interval and timeout determine the failover speed:

The total failover time: ~10-30 seconds for the load balancer to detect an unhealthy API server and route around it.

Active-active vs active-passive

Two strategies:

• Active-active. All API servers receive traffic; the load balancer round-robins.
• Active-passive. One API server is active; failover moves to a passive on failure.

Active-active is the production standard. Each API server is treated as equivalent; the LB routes traffic evenly.

flowchart LR
    subgraph active-active
        LB1[LB] --> AS1
        LB1 --> AS2
        LB1 --> AS3
    end
    subgraph active-passive
        LB2[LB] --> AS4[active]
        AS4 -.->|failover| AS5[passive]
    end

Active-passive leaves capacity unused; active-active uses all.

Client behaviour during failover

A client (kubectl, controller, kubelet) that is in the middle of an HTTP request when the load balancer fails over:

• HTTP/1.1: The TCP connection is severed; the request fails. The client retries on a new connection that lands on a different API server.
• HTTP/2 (kubectl ≥ 1.13): Persistent connection; the client may retry without the user knowing.

The client library (client-go) typically retries transient failures (5xx, network errors) with exponential backoff. Failover at the LB layer is invisible to a well- written client.

Watch and failover

A kubelet watching the API server may experience:

• A brief connection drop; the watcher reconnects automatically (Part LXX-03).
• Loss of events during the drop; the watcher re-lists to recover state.
• Resource version mismatch; the watcher uses the last acknowledged resourceVersion.

The watch cache on the new API server is independent of the old one’s; the kubelet sees the same cluster state without loss (the watch cache is updated from the same etcd).

The VIP and DNS

The load balancer’s IP is typically a floating VIP that the kubeconfig points to. Two DNS patterns:

• Direct IP. https://10.0.1.254:6443 (the VIP).
• DNS name. https://api.example:6443 resolves to the VIP via A record.

The DNS approach is recommended for clean failover (DNS TTL controls how quickly clients switch).

Maintenance during HA

A rolling restart of API servers is the standard for maintenance:

gantt
    title HA rolling maintenance
    dateFormat HH:mm
    axisFormat %H:%M
    section Restart cp-1
    Stop pod :a1, 00:00, 30s
    Wait drain :a2, after a1, 30s
    Start pod :a3, after a2, 30s
    section Restart cp-2
    Stop pod :b1, 01:00, 30s
    Wait drain :b2, after b1, 30s
    Start pod :b3, after b2, 30s
    section Restart cp-3
    Stop pod :c1, 02:00, 30s
    Wait drain :c2, after c1, 30s
    Start pod :c3, after c2, 30s

At any moment, at least 2 of 3 API servers are serving traffic. The cluster is observable during the rolling.

What doesn’t fail over

The LB only routes API server traffic. Other components (scheduler, controller-manager, etcd) have their own high-availability patterns:

• scheduler / controller-manager typically use leader election: only one active at a time, with a backup failover on leader failure.
• etcd uses Raft consensus (Part LXVI).

The API server HA pattern is specifically for the API server surface.

Quiz

Production discipline

• 3 API servers is the standard. Active-active; spread across hosts.
• Configure the LB with TCP health checks. Avoid HTTP-based checks that require deeper inspection.
• Verify failover. Kill a host and watch the cluster remain operational.
• Use DNS for the kubeconfig endpoint. A records with low TTL enable clean failover.
• Test rolling maintenance. A maintenance process that takes 1 instance off-line at a time is the basis for upgrade and restart operations.

HA for the API server is the foundation that makes every other operational procedure safe.