Cluster loss — the end-to-end recovery sequence

Complete cluster loss is the worst-case DR scenario: every node is gone, every PVC is detached, every manifest is lost. Recovery is a nine-phase sequence that must be executed correctly in order. This lesson walks the sequence, the dependencies, the parallelism opportunities, and the operational discipline.

The nine-phase sequence

flowchart TD
    A[1. New infrastructure] --> B[2. Control plane]
    B --> C[3. Networking]
    C --> D[4. Storage]
    A -.-> C
    A -.-> D
    B --> E["5. Cluster state: CRDs, add-ons"]
    E --> F[6. Workers]
    F --> G[7. Workloads]
    G --> H[8. Persistent data]
    H --> I[9. Validation]

The phases:

1. New infrastructure. Provision new control-plane and worker nodes via IaC (Terraform, Cluster API).
2. Control plane. kubeadm init on the control-plane nodes; join to form HA.
3. Networking. Install the CNI; configure cluster DNS; verify pod-to-pod and pod-to-service.
4. Storage. Install the CSI driver; verify StorageClasses; create or restore VolumeSnapshots.
5. Cluster state. Apply CRDs, ConfigMaps, Secrets (from Git or external store), RBAC, ServiceAccounts.
6. Workers. Workers join the cluster; verify they are Ready.
7. Workloads. Apply Deployments, StatefulSets, Services. The controllers begin reconciling.
8. Persistent data. Restore PVCs from Velero or CSI snapshots; the workloads start with their data.
9. Validation. End-to-end tests against the live service.

The dependencies

flowchart LR
    A["Phase 1: infra"] --> B["Phase 2: control plane"]
    B --> C["Phase 3: networking"]
    C --> D["Phase 5: cluster state"]
    D --> E["Phase 6: workers"]
    E --> F["Phase 7: workloads"]
    F --> G["Phase 8: persistent data"]
    G --> H["Phase 9: validation"]
    A -.->|parallel| I["Phase 4: storage"]

The dependencies:

• Phase 2 (control plane) depends on Phase 1 (infra).
• Phase 3 (networking) depends on Phase 2.
• Phase 5 (cluster state) depends on Phase 3 (CRDs must be applied over a working network).
• Phase 6 (workers) depends on Phase 5 (workers register against the API server which needs RBAC).
• Phase 7 (workloads) depends on Phase 6 (workloads schedule on workers).
• Phase 8 (persistent data) depends on Phase 7 (PVCs bind to workloads’ nodes).
• Phase 9 (validation) depends on all of the above.

The parallelism opportunities

Some phases can run in parallel:

flowchart TD
    A["Phase 1: infra"] --> B["Phase 2: control plane"]
    A --> C["Phase 4: storage prep"]
    B --> D["Phase 3: networking"]
    C --> D
    D --> E["Phase 5: cluster state"]
    E --> F["Phase 6: workers"]
    F --> G["Phase 7: workloads"]
    G --> H["Phase 8: persistent data"]
    H --> I["Phase 9: validation"]

• Phase 1 (infra) and Phase 4 (storage prep). Storage backends (EBS volumes, S3 buckets) can be provisioned in parallel with the cluster nodes.
• Phase 2 (control plane) and Phase 4 (storage prep). While kubeadm init runs on the first control-plane node, the operator can pre-create StorageClasses and CSI driver configs.
• Phase 3 (networking) and Phase 5 (cluster state). After the CNI is installed, CRDs and ConfigMaps can be applied while workers are joining.

Parallelism reduces the recovery time. The RTO target should account for the parallelism used in the production runbook.

The recovery time budget

A production recovery time budget for Tier 3 DR:

The RTO target should be at least 4 hours for Tier 3. A shorter RTO requires Tier 4 or Tier 5 (active/passive or active/active).

The operational failure modes

The complete sequence fails for predictable reasons:

• Phase ordering wrong. Workloads applied before CRDs are rejected. The cluster appears up but workloads do not reconcile.
• Phases not parallelised. The sequence is run serially when parallelism was possible; RTO is exceeded.
• Storage restored before CSI driver. PVCs cannot bind because the CSI driver is not installed.
• Workers not verified. Workers joined but are NotReady because the CNI is not running on them.
• Validation skipped. The recovery is declared complete before workloads are functional.

Quiz

Production discipline

Complete cluster loss recovery in production rests on five non-negotiable elements:

• Test the complete sequence quarterly. Partial tests miss ordering bugs and dependency gaps. The full sequence is the only valid test.
• Parallelise where possible. Storage prep with control plane; cluster state with workers. The parallelism reduces RTO.
• Enforce phase ordering in tooling. Argo CD sync waves, Helm hooks, or imperative ordering in the runbook script. Manual ordering relies on the operator’s memory.
• Document the recovery time budget. Each phase has an expected time; the sum is the RTO. The RTO must match the business’s commitment.
• Validate end-to-end. Phase 9 is not optional. A recovery that has not been validated is not complete.

The complete sequence is the only valid recovery path. Skipping phases produces a non-functional cluster. The discipline is to execute the full sequence, in order, with parallelism, and to validate the result.