Recovery architecture for control-plane loss — the design choices

Control-plane loss is the most consequential DR event because every operation flows through the API server and every state decision is held in etcd. This lesson walks the recovery architecture: etcd restore, control-plane rebuild, restore-in-place vs restore-to-new-cluster, quorum-loss scenarios, and the operational discipline of testing control-plane recovery quarterly.

The two recovery paths

flowchart TD
    A[Control-plane loss] --> B{What survived?}
    B -->|etcd intact| C[Rebuild API server only]
    B -->|etcd gone| D{etcd snapshot available?}
    D -->|Yes| E[etcd restore from snapshot]
    D -->|No| F[Rebuild from scratch]
    B -->|API server certificates gone| G[Rebuild API server]

The two primary recovery paths:

• etcd restore. etcd is gone but a snapshot exists (from etcdctl snapshot save or velero backup). Stop the API server, restore etcd from the snapshot, restart the API server. The cluster’s objects return to the snapshot’s state.
• Control-plane rebuild. etcd is gone and no usable snapshot exists. Re-provision the control plane via kubeadm (or the cloud’s managed service), re-apply all manifests from Git, re-provision all PVs from Velero or application-level backups.

The choice depends on what survived. A cluster with intact etcd data (corrupt API server, lost kubelet TLS) can be recovered by rebuilding only the API server. A cluster with lost etcd data needs a rebuild.

Restore-in-place vs restore-to-new-cluster

flowchart LR
    A[etcd snapshot] --> B{Restore target}
    B -->|Same cluster| C[Restore in place]
    B -->|New cluster| D[Restore to new]
    C --> E["Same IP, same certificates, same workers"]
    D --> F["New IPs, new certificates, workers rejoin"]

• Restore-in-place. The same nodes are reused. Same IP addresses, same certificates, same worker nodes. Workers rejoin without re-tls. The cluster’s external references (DNS, load balancer) do not need to change.
• Restore-to-new-cluster. New nodes are provisioned. New IPs, new certificates. Workers must re-tls and rejoin. External references (DNS, load balancer) must be updated.

Restore-in-place is faster but requires the original nodes to be available. Restore-to-new-cluster is slower but more robust — it works even when the original infrastructure is gone.

Quorum-loss scenarios

etcd quorum loss has three sub-scenarios:

flowchart TD
    A[3-member etcd cluster] --> B{How many dead?}
    B -->|1 of 3| C[Quorum intact]
    B -->|2 of 3| D[Quorum lost]
    B -->|3 of 3| E[Cluster gone]
    C --> F["Replace dead member, no restore needed"]
    D --> G["Force quorum with --initial-cluster, restore from snapshot"]
    E --> H[Restore from snapshot to new etcd cluster]

• 1 of 3 dead. Quorum is intact (2 of 3 is a majority). The dead member is replaced by adding a new member; etcd rebalances. No data loss.
• 2 of 3 dead. Quorum is lost; etcd refuses writes. The cluster is read-only at best. Recovery requires forcing a new quorum and restoring from a snapshot.
• 3 of 3 dead. The cluster is gone. Recovery requires restoring from a snapshot to a new etcd cluster.

The 2-of-3 scenario is the most operationally subtle. etcd’s safety properties prevent split-brain, so the cluster refuses writes. Recovery requires explicit intervention (etcdutl snapshot restore with --initial-cluster and --initial-advertise-peer-urls pointing to the surviving member).

The snapshot vs live etcd trade-off

flowchart LR
    A[etcd snapshot] -->|RPO = snapshot interval| B[Recovery to snapshot state]
    C[Live etcd] -->|RPO = 0| D[Recovery to last write]

• Snapshot recovery. The cluster returns to the state at the snapshot time. Anything written between the snapshot and the disaster is lost. RPO = snapshot interval.
• Live etcd recovery. If at least one etcd member survived and quorum can be re-established, the cluster returns to the state at the last write before the failure. RPO ≈ 0.

Live recovery is preferred when possible. But live recovery requires that at least one member’s data directory is intact — which is rare in a true disaster (the data directory lives on the same machine as the failed control plane).

The control-plane rebuild procedure

When etcd and the API server are both gone:

# 1. Provision new control-plane nodes
for node in cp-1 cp-2 cp-3; do
  # Provision via your IaC (Terraform, OpenTofu, Ansible)
  echo "provision ${node}"
done

# 2. Install kubeadm, kubelet, kubectl on each

# 3. On the first node, kubeadm init
kubeadm init --control-plane-endpoint "lb-endpoint:6443" --upload-certs

# 4. Join the other two control-plane nodes, using the token, CA cert hash
#    and certificate key that step 3 printed:
JOIN_TOKEN=abcdef.0123456789abcdef
CA_CERT_HASH=sha256:56b65c61aafa58096802e3926e7d7a04b78df0d3f1cc3d425d4dbcb2be3582d1
CERT_KEY=fbc0c2654e39178a02482a554d4b9b1f30ee4a99c27e5f2ed4dcbae3dab96d61

kubeadm join lb-endpoint:6443 \
  --token "$JOIN_TOKEN" \
  --discovery-token-ca-cert-hash "$CA_CERT_HASH" \
  --control-plane \
  --certificate-key "$CERT_KEY"

# 5. Install the CNI
kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

# 6. Restore workloads from Git or Velero
velero restore create --from-backup latest

# 7. Restore persistent data from Velero/CSI snapshots

The procedure takes hours; the RTO is the procedure duration plus the validation.

The operational failure modes

Control-plane recovery fails in production for predictable reasons:

• No recent etcd snapshot. The snapshot cadence was weekly; the disaster lost 5 days of changes. The RPO is far worse than designed.
• Snapshot stored on the same cluster. The etcd snapshot was stored in the cluster’s own object storage, which is gone with the cluster. The snapshot is not accessible for restore.
• Certificates expired. The API server’s serving certificates expired during the recovery; workers refuse to join. The recovery stalls.
• CNI not installed. The control plane is up but the CNI is not installed; Pods stay Pending. The recovery is not functional.
• Workers cannot rejoin. The workers’ kubeconfigs point to the old control plane’s IP. New control plane has a new IP. Workers must be re-bootstrapped.

Quiz

Production discipline

Control-plane recovery in production rests on five non-negotiable elements:

• Test the recovery quarterly. The procedure that has never been executed takes longer than the RTO. Quarterly tests prove the recovery time and reveal missing prerequisites.
• Store snapshots off-cluster. An etcd snapshot stored on the cluster it backs up is not a backup; it is a hope. Store snapshots in S3 or another off-cluster location.
• Document the recovery runbook. The runbook lists the procedure, the prerequisites, and the validation steps. A recovery during an incident is not the time to compose the procedure from memory.
• Use restore-to-new-cluster for true DR. The standard production DR scenario assumes the original infrastructure is gone. Rebuild on new nodes with new IPs and certificates.
• Rehearse the worst case. The 2-of-3 etcd quorum loss is the most operationally subtle scenario. Rehearse it specifically.

Control-plane recovery is the most consequential DR event. A runbook that has never been tested is not a runbook — it is a hope that someone remembers the procedure under pressure.