Control plane rebuild — kubeadm init, HA topology, and the join sequence

Phase 2 of complete cluster loss recovery is rebuilding the control plane via kubeadm init on the first node, then joining the additional control-plane nodes for HA. This lesson walks the kubeadm init procedure, the join sequence, the API server endpoint configuration, and the operational discipline.

The kubeadm init procedure

flowchart LR
    A[kubeadm init on cp-1] --> B[API server up]
    B --> C[kubeadm join on cp-2]
    B --> D[kubeadm join on cp-3]
    C --> E[HA control plane]
    D --> E

The procedure:

# On the first control-plane node (cp-1)
kubeadm init \
  --control-plane-endpoint "lb-endpoint:6443" \
  --upload-certs \
  --pod-network-cidr=10.244.0.0/16 \
  --service-dns-domain=cluster.local

# The output includes:
# - kubeadm join commands for additional control-plane nodes
# - kubeadm join command for worker nodes
# - certificate key for uploading certificates

The flags:

• --control-plane-endpoint — the load balancer DNS or IP that fronts the API server. All nodes connect to this endpoint, not to individual control-plane IPs.
• --upload-certs — upload the control-plane certificates to a Secret in the cluster, so additional control-plane nodes can fetch them via --certificate-key.
• --pod-network-cidr — the CIDR for pod IPs. Must match the CNI’s expected range.
• --service-dns-domain — the cluster DNS suffix. Default cluster.local.

Joining additional control-plane nodes

# All three values come from the saved `kubeadm init` output:
TOKEN=abcdef.0123456789abcdef
CA_CERT_HASH=sha256:6f0a1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8
CERT_KEY=f8e7d6c5b4a3928170615243f8e7d6c5b4a3928170615243f8e7d6c5b4a39281

# On cp-2 and cp-3
kubeadm join lb-endpoint:6443 \
  --token "$TOKEN" \
  --discovery-token-ca-cert-hash "$CA_CERT_HASH" \
  --control-plane \
  --certificate-key "$CERT_KEY"

The flags:

• --token — the bootstrap token from kubeadm init.
• --discovery-token-ca-cert-hash — the CA cert hash; ensures the joining node trusts the cluster’s CA.
• --control-plane — mark this node as a control-plane node (not a worker).
• --certificate-key — the key from --upload-certs in kubeadm init; used to fetch the cluster certificates from the Secret.

The join procedure:

1. The joining node connects to the API server at lb-endpoint:6443.
2. The joining node authenticates with the bootstrap token.
3. The joining node verifies the CA cert hash.
4. The joining node fetches the cluster certificates using the certificate key.
5. The joining node starts kube-apiserver, kube-controller-manager, kube-scheduler, etcd.
6. The new control-plane node joins etcd’s Raft cluster.

The API server endpoint

flowchart LR
    A[External client] --> B[Load balancer]
    B --> C[cp-1 API server]
    B --> D[cp-2 API server]
    B --> E[cp-3 API server]
    C --> F[etcd on cp-1]
    D --> G[etcd on cp-2]
    E --> H[etcd on cp-3]

The --control-plane-endpoint is the load balancer DNS or IP. The load balancer fronts the three control-plane nodes’ API servers. Clients connect to the load balancer; the load balancer distributes requests across the API servers.

The load balancer must:

• Listen on port 6443 (API server).
• Health-check the API servers (kube-apiserver’s /healthz endpoint).
• Distribute TCP traffic across the API servers.

A misconfigured load balancer makes the API server unreachable; kubeadm init may succeed but workers cannot join.

Verifying the control plane

After kubeadm init and the joins:

# On cp-1 (or any control-plane node)
kubectl get nodes

NAME    STATUS   ROLES           AGE   VERSION
cp-1    Ready    control-plane   5m    v1.34.0
cp-2    Ready    control-plane   2m    v1.34.0
cp-3    Ready    control-plane   2m    v1.34.0

The three nodes should be Ready with the control-plane role. If any are NotReady, the control plane is not fully HA.

kubectl get pods -n kube-system

NAME                             READY   STATUS    RESTARTS   AGE
kube-apiserver-cp-1              1/1     Running   0          5m
kube-apiserver-cp-2              1/1     Running   0          2m
kube-apiserver-cp-3              1/1     Running   0          2m
kube-controller-manager-cp-1     1/1     Running   0          5m
kube-controller-manager-cp-2     1/1     Running   0          2m
kube-controller-manager-cp-3     1/1     Running   0          2m
kube-scheduler-cp-1              1/1     Running   0          5m
kube-scheduler-cp-2              1/1     Running   0          2m
kube-scheduler-cp-3              1/1     Running   0          2m
etcd-cp-1                        1/1     Running   0          5m
etcd-cp-2                        1/1     Running   0          2m
etcd-cp-3                        1/1     Running   0          2m

All control-plane components should be Running. If any are CrashLoopBackOff or Pending, the control plane is not functional.

The operational failure modes

Phase 2 fails for predictable reasons:

• kubeadm init output not saved. The join commands and certificate key are lost; subsequent nodes cannot join.
• Load balancer not configured. The --control-plane-endpoint points to a DNS or IP that does not yet resolve or route.
• Token expired. The bootstrap token has a default TTL of 24 hours. If the join happens later, the token is invalid; a new token must be created with kubeadm token create.
• Certificate key lost. The --upload-certs certificate key is required for additional nodes. Without it, the additional nodes cannot fetch cluster certificates.
• etcd quorum not formed. Two of three control-plane nodes joined but the third has a clock skew or network issue; etcd cannot form quorum.

Quiz

Production discipline

Phase 2 (control plane rebuild) in production rests on five non-negotiable elements:

• Save the kubeadm init output. The output is the source of truth for the join commands and certificate key. Save it to a file immediately.
• Rehearse the join procedure quarterly. The flags, the token, the certificate key — under pressure the operator forgets. Quarterly rehearsals build muscle memory.
• Verify the load balancer first. The —control-plane-endpoint must resolve and route before kubeadm init starts. A misconfigured load balancer produces a non-functional cluster.
• Check etcd quorum. After all three control-plane nodes join, verify etcd has quorum with etcdctl endpoint status. Quorum loss is the most operationally subtle failure.
• Document the kubeadm version and flags. The kubeadm version and the exact flags used must be in the runbook. A different version or flags produce a different cluster.

The control plane is the foundation. A control plane that has never been rebuilt via kubeadm is a control plane whose recovery is unknown. The discipline is to rehearse quarterly.