Control plane failure recovery — the cluster's worst day

Restoring etcd from a snapshot rewinds every object in the cluster to the moment that snapshot was taken; Secrets, PVC bindings, RBAC grants and Deployments created since are gone, and no controller will bring them back. The one mercy is that the workloads do not stop — kubelets keep running the Pods they already have, so a control plane outage is not immediately an application outage, and that buys the time to do the restore carefully. What decides the outcome is whether the snapshot was ever tested, because a restore is the wrong moment to discover the file is truncated or the data directory is owned by the wrong user.

The disaster recovery

The control plane failure recovery is the cluster’s disaster recovery. The recovery is the last resort when the control plane is not recoverable.

flowchart TD
    A[Control plane failing] --> B{Recoverable?}
    B -->|Yes| C[Fix the control plane]
    B -->|No| D[Disaster recovery]
    D --> E[Restore from etcd snapshot]
    E --> F[Rebuild the cluster]

The disaster recovery is the cluster’s worst day.

The diagnostic

The canonical diagnostic:

# 1. Check the API server
kubectl get --raw /healthz

# 2. Check the etcd
ETCDCTL_API=3 etcdctl endpoint health

# 3. Check the kubelet
journalctl -u kubelet -n 200

# 4. Check the etcd snapshot
ls -la /var/backups/etcd/

# 5. Check the cluster's state
kubectl get nodes

The diagnostic is the API server, the etcd, and the kubelet.

The recovery

The recovery is the canonical disaster recovery:

# 1. Stop the control plane. kubeadm runs all four components
# as static Pods, so move their manifests out of the kubelet's
# watch directory rather than reaching for systemctl.
mkdir -p /root/manifests-parked
mv /etc/kubernetes/manifests/*.yaml /root/manifests-parked/
# wait for the kubelet's fileCheckFrequency, 20s by default,
# then confirm nothing is left running
crictl ps

# 2. Restore the etcd from the snapshot
etcdutl snapshot restore /var/backups/etcd/snapshot.db \
  --data-dir /var/lib/etcd-restore

# 3. Point the etcd static Pod at the restored data directory
# Edit the hostPath volume in
# /root/manifests-parked/etcd.yaml so it maps
# /var/lib/etcd-restore into the container

# 4. Start the etcd first and let it come up alone
mv /root/manifests-parked/etcd.yaml /etc/kubernetes/manifests/

# 5. Start the rest of the control plane
mv /root/manifests-parked/*.yaml /etc/kubernetes/manifests/

# 6. Verify the cluster
kubectl get nodes
kubectl get pods -A

The recovery is the cluster’s restoration.

The kubeadm reset

The kubeadm reset is the alternative to disaster recovery. The reset tears down the cluster and re-initializes it.

# Substitute your own values before running:
CP_ENDPOINT=k8s-api.example.com   # the control-plane load balancer
CP_NODE_2=192.0.2.12              # second control-plane node
WORKER_NODE=192.0.2.21            # one worker node

# 1. Reset the kubeadm
kubeadm reset

# 2. Re-initialize the cluster
kubeadm init --control-plane-endpoint "$CP_ENDPOINT:6443" \
  --upload-certs

# The token, CA cert hash and certificate key are printed by the
# `kubeadm init --upload-certs` run in step 2 - copy them from there:
JOIN_TOKEN=abcdef.0123456789abcdef
CA_CERT_HASH=sha256:8cb2de97839780ac5162a5b3dc0f0f4b3aa25b3f19d0c9d4e5e3f2a1b0c9d8e7
CERT_KEY=f8902e114ef119b7cbee4b4cb1adf3f9a4b3f9d8c7e6f5a4b3c2d1e0f9a8b7c6

# 3. Re-join the other control-plane nodes
ssh "$CP_NODE_2" "sudo kubeadm join $CP_ENDPOINT:6443 \
  --token $JOIN_TOKEN \
  --discovery-token-ca-cert-hash $CA_CERT_HASH \
  --control-plane --certificate-key $CERT_KEY"

# 4. Re-join the worker nodes
ssh "$WORKER_NODE" "sudo kubeadm join $CP_ENDPOINT:6443 \
  --token $JOIN_TOKEN \
  --discovery-token-ca-cert-hash $CA_CERT_HASH"

# 5. Re-apply the workloads
kubectl apply -f workloads/

The reset is the cluster’s rebuild.

The remediation

The remediation depends on the failure:

# Option 1: Fix the API server by restarting its static Pod
mv /etc/kubernetes/manifests/kube-apiserver.yaml /tmp/
# wait for the kubelet's fileCheckFrequency, 20s by default
mv /tmp/kube-apiserver.yaml /etc/kubernetes/manifests/

# Option 2: Restore the etcd
etcdutl snapshot restore

# Option 3: Reset the cluster
kubeadm reset
kubeadm init

The remediation is the control plane recovery.

Production discipline

A control plane failure is the cluster’s hypothesis. The discipline is to walk the 11-step methodology applied to the control plane, identify the cause, apply the remediation. The control plane is the cluster’s brain; the remediation is the control plane recovery.

• Have a verified snapshot. The snapshot is the cluster’s last resort.
• Rehearse the restoration. The rehearsal is the cluster’s preparedness.

Quiz