New infrastructure — provisioning nodes from scratch

Phase 1 of complete cluster loss recovery is provisioning new infrastructure — the nodes on which the new cluster will run. This lesson walks the IaC approach (Terraform, Cluster API), the node specifications, the network and storage prerequisites, the time budget, and the operational discipline.

The IaC approach

flowchart LR
    A[IaC repository] --> B{Tool}
    B -->|Terraform| C["Cloud VMs, networks, storage"]
    B -->|Cluster API| D[Kubernetes-native nodes]
    C --> E[terraform apply]
    D --> F[clusterctl apply]
    E --> G[New nodes]
    F --> G

Two approaches:

• Terraform — provisions cloud resources directly: VMs, networks, security groups, IAM roles, S3 buckets. The output is a set of nodes ready for kubeadm. Used for cloud-managed infrastructure or on-prem VMs.
• Cluster API (CAPI) — provisions nodes as Kubernetes objects (Machine, MachineDeployment). The output is a Kubernetes-native cluster managed by CAPI providers. Used for declarative, GitOps-driven cluster lifecycle.

Most production uses Terraform for the initial provisioning and CAPI for ongoing worker node management. The combination gives IaC at both layers.

The node specifications

The control-plane and worker nodes have different requirements:

flowchart LR
    A[Control plane] --> B[3 nodes minimum for HA]
    A --> C[etcd data on dedicated disk]
    A --> D["CPU/memory for kube-apiserver, etcd, scheduler"]
    E[Workers] --> F[3+ nodes for workload distribution]
    E --> G["CPU/memory for application workloads"]
    E --> H[Local storage or attached volumes for kubelet]

Control plane:

• 3 nodes minimum for HA (5 for larger clusters).
• CPU/memory — 4 vCPU, 8 GiB minimum; 8 vCPU, 16 GiB for production.
• etcd disk — dedicated SSD; 100 GB minimum; performance-critical.
• Network — low-latency between control-plane nodes for etcd consensus.

Workers:

• 3+ nodes for workload distribution and HA.
• CPU/memory — sized for the workload profile.
• Container runtime disk — fast SSD for image layers and ephemeral storage.
• Network — high-bandwidth for pod-to-pod and external traffic.

The network prerequisites

flowchart LR
    A[VPC] --> B["Subnet: control plane"]
    A --> C["Subnet: workers"]
    A --> D["Subnet: pods"]
    A --> E["Subnet: services"]
    A --> F["Internet gateway / NAT"]
    A --> G[Security groups]
    A --> H[Route tables]
    A --> I[DNS zone]

The network must be ready before the cluster can function:

• VPC with subnets for control plane, workers, pods, services.
• Internet gateway or NAT for outbound traffic (image pulls, S3 access).
• Security groups allowing API server (6443), kubelet (10250), etcd peer (2379-2380), pod CIDR.
• Route tables for pod-to-pod and external.
• DNS zone for the cluster’s external records.

The storage prerequisites

Storage must be ready before PVCs can be provisioned:

flowchart LR
    A[Storage backends] --> B[Cloud block storage class]
    A --> C[Object storage bucket]
    A --> D["NFS / shared filesystem"]
    A --> E[CSI driver IAM role]

• Block storage for PVs (EBS, Azure Disk, vSphere VMDK, Ceph RBD).
• Object storage bucket for Velero backups, application data, container images.
• Shared filesystem if used (NFS, CephFS, Longhorn).
• IAM roles for the CSI driver to provision volumes (e.g., the EBS CSI driver’s IAM role for CreateVolume permission).

Parallelising with kubeadm init

flowchart TD
    A["terraform apply: nodes"] --> B["kubeadm init: phase 2"]
    A --> C["terraform apply: storage prep"]
    A --> D["terraform apply: DNS zone"]
    C --> E["CSI driver install: phase 4"]
    D --> F["External DNS config: phase 3"]
    B --> G[Cluster up]
    E --> G
    F --> G

Phase 1 (infrastructure) can run in parallel with:

• Phase 4 prep (cloud volumes, S3 buckets) — these are IaC resources provisioned by the same Terraform apply.
• DNS zone creation — Terraform creates the zone; external DNS provider is configured to delegate.

The parallelisation reduces RTO. The IaC runbook lists which resources can be provisioned in parallel.

The time budget

Phase 1 time budget for cloud-managed IaC:

For on-prem infrastructure, phase 1 may take hours (physical hardware, network cabling). Cloud-managed IaC is faster.

The operational failure modes

Phase 1 fails for predictable reasons:

• IaC not maintained. The Terraform configs have drifted from the actual cluster. Apply fails.
• Cloud quotas exceeded. The cloud account has hit a quota for VMs, IPs, or volumes. Provisioning fails.
• Region unavailable. The disaster is regional; the same region cannot provision new resources. The IaC must target a different region.
• DNS not delegated. The DNS zone is created but not delegated to the cluster’s nameservers.
• IAM roles missing. The CSI driver cannot provision volumes without its IAM role.

Quiz

Production discipline

Phase 1 (infrastructure provisioning) in production rests on five non-negotiable elements:

• IaC for every resource. Terraform or CAPI for every node, network, storage, IAM role, DNS zone. No manual provisioning during a disaster.
• Test the IaC quarterly. The IaC that has never been applied in a real environment may fail at the worst moment. Quarterly dry-runs catch drift and quota issues.
• Maintain a DR region. A regional DR requires pre-provisioned capacity in another region. The cost is justified by the RTO.
• Parallelise storage and DNS prep. These can run in parallel with node provisioning. The parallel runbook lists which resources.
• Document the topology. The architecture diagram must show the DR region, the IaC repository, the backup locations, and the dependency graph.

Phase 1 is the foundation. A foundation built under pressure is fragile. The discipline is to build it in advance, in IaC, and to test it quarterly.