Control plane HA — responsibility and architecture

Control plane HA is one of the most consequential operational concerns in Kubernetes. This lesson walks the responsibility split (managed vs self-managed), the HA architecture, the failure modes, etcd quorum, and the operational discipline.

The responsibility split

flowchart LR
    A["Managed: HA"] --> A1[Provider manages 3+ control-plane nodes]
    A --> A2[Provider manages etcd quorum]
    A --> A3[Provider handles control-plane upgrades]
    B["Self-managed: HA"] --> B1[Operator provisions 3+ control-plane nodes]
    B --> B2[Operator configures etcd cluster]
    B --> B3[Operator handles control-plane upgrades]

The split:

Managed (EKS, AKS, GKE, OKE):

• The provider provisions and manages 3+ control-plane nodes (typically 3 for production).
• The provider manages etcd clustering and quorum.
• The provider handles control-plane upgrades.
• The provider monitors control-plane health.
• The operator does not see the control-plane nodes in kubectl get nodes (they are hidden or marked as control-plane).

Self-managed (kubeadm):

• The operator provisions 3+ control-plane nodes.
• The operator configures etcd as a clustered service (typically static pod or external etcd).
• The operator handles control-plane upgrades (kubeadm upgrade on each node).
• The operator monitors control-plane health.
• The operator sees all control-plane nodes in kubectl get nodes.

The HA architecture

flowchart LR
    A[Control-plane node 1] --> A1[etcd member 1]
    B[Control-plane node 2] --> B1[etcd member 2]
    C[Control-plane node 3] --> C1[etcd member 3]
    A1 <--> B1
    B1 <--> C1
    A1 <--> C1
    D[Load balancer] --> A
    D --> B
    D --> C

The HA architecture:

• 3 control-plane nodes (5 for larger clusters, e.g., 100+ nodes).
• etcd clustered — each control-plane node runs an etcd member; the etcd members form a Raft cluster.
• Load balancer fronts the API servers; clients connect to the load balancer, which distributes traffic across the API servers.
• Stacked or external etcd. In stacked, etcd runs on the same nodes as the control plane. In external, etcd runs on separate nodes.

Stacked is more common in kubeadm deployments (3 nodes running both control plane and etcd). External is used when etcd needs dedicated resources or when the cluster is large.

The failure modes

flowchart TD
    A[3 control-plane nodes] --> B{How many dead?}
    B -->|0 dead| C["Healthy, 3 nodes serving"]
    B -->|1 dead| D["Quorum intact, 2 nodes serving"]
    B -->|2 dead| E["Quorum lost, read-only"]
    B -->|3 dead| F[Cluster gone]
    D --> G["Replace dead node, etcd rebalances"]
    E --> H[Force new cluster from surviving member or restore from snapshot]
    F --> I[Restore from snapshot to new cluster]

The failure modes:

• 0 dead. Healthy. 3 nodes serving; load balancer distributes traffic.
• 1 dead. Quorum intact (2 of 3 is majority). The cluster continues to serve traffic. Replace the dead node; etcd rebalances.
• 2 dead. Quorum lost; etcd refuses writes. The cluster is read-only at best. Recovery requires either --force-new-cluster on the surviving member or restore from snapshot.
• 3 dead. Cluster is gone. Recovery requires restore from snapshot to a new cluster.

Etcd quorum

flowchart LR
    A[3-member etcd] --> B{Quorum: 2 of 3}
    B -->|Yes| C[Writes accepted]
    B -->|No| D[Writes refused]

Etcd uses Raft consensus. A quorum is a majority of members. For 3 members, quorum is 2. For 5 members, quorum is 3.

Quorum loss is the most operationally subtle scenario: the cluster appears up (API server responds to reads) but writes fail. The diagnostic is etcdctl endpoint status — if only 1 member is healthy, quorum is lost.

The upgrade procedure

Managed:

• Trigger upgrade via portal or CLI.
• Provider upgrades control plane first.
• Operator upgrades worker nodes.

Self-managed:

• kubeadm upgrade plan on the first control-plane node.
• kubeadm upgrade apply v1.34.x on the first node.
• Drain and upgrade each additional control-plane node.
• Drain and upgrade each worker node.

The self-managed procedure is more complex but gives the operator full visibility.

The operational failure modes

Control plane HA fails for predictable reasons:

• Single control-plane node in production. A 1-node control plane is a single point of failure. Always 3+ for production.
• Load balancer misconfiguration. The load balancer does not health-check the API servers; it sends traffic to a dead node.
• etcd not clustered. A kubeadm cluster with etcd on a single node (or with etcd on the control-plane nodes but not clustered) loses all etcd data if one node dies.
• Certificate expiry. The API server’s serving certificates expire; kubelets refuse to authenticate.
• Upgrade without testing. A control-plane upgrade that breaks in production because it was not tested in staging.

Quiz

Production discipline

Control plane HA in production rests on five non-negotiable elements:

• 3 control-plane nodes minimum. A single-node control plane is a single point of failure.
• Distribute across failure domains. Different racks, different power, different network.
• Test quorum-loss recovery quarterly. The 2-of-3 scenario is the most operationally subtle.
• Monitor etcd health. etcdctl endpoint status is the diagnostic. Alert on member loss.
• Plan certificate rotation. API server certificates expire; rotate before they do.

Control plane HA is the foundation of cluster reliability. A control plane that has never been tested is unreliable. The discipline is to test quarterly.