critical riskcluster affecting~180 min

Runbook: Recover a Failed Control Plane

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Confirm the failure mode: which control-plane nodes are up, which are reachable, and which have lost quorum
· Capture the etcd cluster state from a surviving member: etcdctl endpoint status --cluster -w table
· Capture a recent etcd snapshot: restic -r sftp:backup.internal:/srv/etcd snapshots
· Capture the load balancer state: which control-plane nodes it sees as healthy
· Confirm the recovery decision is communicated to cluster users
· Confirm the change ticket documents the recovery point and the recovery plan

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Classify the failure: single node down, quorum loss, or total control-plane loss
2For a single node down: bring it back via kubeadm upgrade node or by restoring the node from a known-good image
3For quorum loss with one surviving node: read-only operations only; restore from snapshot (see kubernetes-rb-restore-etcd)
4For total control-plane loss: rebuild control-plane nodes from the bootstrap path and restore etcd from snapshot
5For HA load balancer pointing at zero healthy backends: repair the load balancer before rebuilding the control plane
6For total infrastructure loss: rebuild on new infrastructure (compute, network, storage) and restore from snapshot
7Validate every component: API server, scheduler, controller-manager, etcd, kubelet, CNI, CoreDNS
8Validate every workload namespace: spot-check 5 representative workloads
9Re-attach and validate worker nodes: re-bootstrap workers if their client certs expired during the outage
10Communicate recovery and capture the post-mortem

4 · Verification

Confirm the procedure actually fixed the problem.

✓etcdctl ... endpoint health --cluster -w table reports every member healthy with one leader
✓curl -k https://<api-vip>:6443/healthz returns ok
✓kubectl get nodes -o wide returns every node Ready (control plane + workers)
✓kubectl get pods -A returns the expected set of Pods
✓A test workload runs end-to-end on the recovered cluster
✓A representative public endpoint returns 200
✓RBAC, NetworkPolicies and PDBs are intact: kubectl auth can-i get pods, kubectl get netpol -A, kubectl get pdb -A
✓No x509, clock skew, or auth errors in the API server, kubelet, or etcd logs

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶If recovery fails partway, restore the previous control-plane state from the most recent snapshot
↶If a worker cannot re-bootstrap, leave it out of the cluster and re-provision it from a known-good image
↶If the recovered cluster is missing workloads, the snapshot pre-dates them; pull the next-most-recent snapshot
↶Capture every step of the failed recovery to the change ticket
↶For a recovery that cannot succeed, escalate to a full cluster rebuild from the bootstrap path

6 · Escalation

When the runbook isn't enough, contact:

· Recovery exceeds the change window: schedule the next attempt in a fresh window; do not improvise under time pressure
· The etcd snapshot is corrupt: escalate to platform ownership; do not proceed with a partial recovery
· Workers cannot rejoin because their client certs expired: re-bootstrap them with the new CA
· CNI cannot re-establish on the recovered cluster: rebuild CNI DaemonSet pods from the operator
· A recovered workload cannot reach its backing services: NetworkPolicy was over-restrictive; review and adjust

A failed control plane is the cluster’s worst incident. The runbook classifies the failure first, then chooses the recovery path that matches. There are three: single-node recovery, etcd restore after quorum loss, and full cluster rebuild on new infrastructure.

1. Classify the failure

Read-only / SafeClassify the failure


for n in <cp1> <cp2> <cp3>; do
echo "=== $n ==="
ssh -o ConnectTimeout=5 "$n" -- uptime 2>&1 || echo "$n: unreachable"
done

# From a surviving node, etcd health
ETCDCTL_API=3 etcdctl \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
endpoint health --cluster -w table 2>&1 || echo "etcd unreachable from this node"

# HA load balancer
ssh <lb-node> -- sudo curl -k https://<lb-vip>:6443/healthz --max-time 3 || echo "LB unhealthy"

Failure mode	Description	Recovery path
Single node down	One control-plane node unreachable	Recover the node (kubeadm upgrade node or rebuild from image)
Quorum loss with one surviving node	One control-plane node up, etcd has no quorum	Restore etcd from snapshot
Quorum loss with no surviving node	All control-plane nodes down	Rebuild control plane + restore etcd from snapshot
Total infrastructure loss	Network, compute, storage all down	Rebuild on new infrastructure + restore from snapshot

2. Single-node recovery

Read-only / SafeSingle-node recovery


ssh <cp-failed> -- sudo kubeadm reset --force
ssh <cp-failed> -- sudo rm -rf /etc/kubernetes/pki /var/lib/etcd /var/lib/kubelet

# Re-join as a control-plane node
ssh <cp-failed> -- sudo kubeadm join <lb-vip>:6443 \
--token <token> \
--discovery-token-ca-cert-hash sha256:<hash> \
--control-plane \
--certificate-key <key> \
--cri-socket unix:///run/containerd/containerd.sock

kubectl get nodes -o wide

3. Quorum loss with one surviving node

Read-only / SafeQuorum loss with one surviving node


for n in <cp1> <cp2> <cp3>; do
ssh "$n" -- bash -c '
  sudo crictl stop $(sudo crictl ps -a -q --name kube-apiserver) 2>/dev/null || true
  sudo crictl stop $(sudo crictl ps -a -q --name kube-scheduler) 2>/dev/null || true
  sudo crictl stop $(sudo crictl ps -a -q --name kube-controller-manager) 2>/dev/null || true
  sudo crictl stop $(sudo crictl ps -a -q --name etcd) 2>/dev/null || true
'
done

# Restore from snapshot on every node (see kubernetes-rb-restore-etcd for full steps)
SNAP=/var/backups/etcd/recovery.db
for n in <cp1> <cp2> <cp3>; do
ssh "$n" -- sudo mv /var/lib/etcd /var/lib/etcd.bak
ssh "$n" -- sudo etcdutl snapshot restore "$SNAP" \
  --data-dir=/var/lib/etcd \
  --name=$(ssh "$n" -- sudo grep -oE '--name=[^ ]+' /etc/kubernetes/manifests/etcd.yaml | head -1 | cut -d= -f2)
done

# Restart everything (kubelet recreates static pods)
sleep 60

etcdctl ... endpoint health --cluster -w table

4. Total control-plane loss

Read-only / SafeTotal control-plane loss


ssh <cp1> -- sudo kubeadm init \
--config kubeadm-config.yaml \
--upload-certs

# Capture the join commands
JOIN_WORKER=$(ssh <cp1> -- sudo kubeadm token create --print-join-command)
JOIN_CP=$(ssh <cp1> -- sudo kubeadm init phase upload-certs --certificate-key <new-key> 2>&1)

# Join additional control-plane nodes
for n in <cp2> <cp3>; do
ssh "$n" -- sudo kubeadm reset --force
ssh "$n" -- sudo $JOIN_CP
done

# Join workers
for w in <worker1> <worker2> ...; do
ssh "$w" -- sudo kubeadm reset --force
ssh "$w" -- sudo $JOIN_WORKER
done

# Restore etcd from snapshot (this must be done before admitting workloads)
# See kubernetes-rb-restore-etcd

5. Validate

Read-only / SafeValidate


kubectl get nodes -o wide
kubectl get pods -A | wc -l
kubectl get csr | head

etcdctl ... endpoint health --cluster -w table
etcdctl ... endpoint status --cluster -w table

# Three representative workloads
kubectl describe deploy/web -n prod | head -30
kubectl describe sts/db -n prod | head -30
kubectl describe deploy/queue -n prod | head -30

# Three checks: auth, network policy, PDB
kubectl auth can-i get pods
kubectl get netpol -A | wc -l
kubectl get pdb -A | wc -l

# A workload smoke test
kubectl run smoketest --image=registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7 \
--restart=Never --command -- sleep 30
kubectl wait --for=condition=Ready pod/smoketest --timeout=60s
kubectl delete pod smoketest --wait=false

6. Re-attach workers (if their certs expired)

Read-only / SafeRe-attach workers (if their certs expired)


ssh <worker> -- sudo kubeadm reset --force
ssh <worker> -- sudo $JOIN_WORKER

# Or for kubelet-only rotation:
ssh <worker> -- sudo systemctl restart kubelet

7. Post-mortem

Read-only / SafePost-mortem


# Control plane recovery summary

Failure mode: <single|quorum|total>
Detection time: <first alert timestamp>
Recovery start: <recovery start timestamp>
Recovery end: <recovery end timestamp>
Data loss since snapshot: <duration>
Operator(s): <names>
Change ticket: <ticket>

Decisions made:
- Recovery point: <snapshot id>
- Recovery path: <single|restore|rebuild>

Follow-up:
- [ ] Post-mortem written
- [ ] Backup pipeline improved
- [ ] Monitoring improved to detect this earlier
- [ ] Runbook improved
EOF

Common pitfalls

Symptom	Cause	Action
Recovery succeeded but workers do not join	Workers’ client certs expired during the outage	Re-bootstrap the workers
Recovery succeeded but workloads are missing	Snapshot pre-dates them	Pull the next-most-recent snapshot
Recovery succeeded but `kubectl auth can-i` denies everyone	RBAC was lost in the snapshot	Restore from a snapshot that predates the RBAC issue
Recovery succeeded but `kubectl get pdb -A` is empty	PDB objects were on a different API version	Restore from the correct snapshot
Recovery succeeded but the cluster reports version skew	Workers were on the previous version	Upgrade the workers

A control-plane recovery is the most expensive runbook in the collection. It is also the one that proves the backup pipeline worked. Treat it as a learning, not a one-off.