Surge and replace — the cloud-native worker upgrade

The surge-and-replace pattern is the cloud-native approach to worker upgrades. The old worker is cordoned; a new worker is launched at the new version; the old worker is terminated. The cluster autoscaler integrates with the pattern. This lesson walks the implementation, the cluster autoscaler integration, and the lifecycle of the old worker.

The pattern

flowchart LR
    A[Worker at v1.33.0] --> B[Cordon worker]
    B --> C[Old worker is unhealthy for the ASG]
    C --> D[Cluster autoscaler launches new worker at v1.34.0]
    D --> E[New worker joins the cluster]
    E --> F[Old worker is drained and terminated]

The cluster autoscaler detects the unhealthy worker (the cordon marks it as unschedulable) and launches a new worker at the new version.

The launch template

The cluster autoscaler uses a launch template:

# AWS EKS example
apiVersion: eksctl.io/v1alpha5
kind: ClusterConfig
metadata:
  name: my-cluster
managedNodeGroups:
  - name: workers
    instanceType: t3.large
    desiredCapacity: 5
    minSize: 3
    maxSize: 10
    ami: ami-0abcdef1234567890  # v1.34.0 AMI
    ssh:
      allow: true
    iam:
      withAddonPolicies:
        autoScaler: true

The ami field is the new kubelet version. Updating the launch template triggers the autoscaler to use the new AMI for new workers.

The cordoning

kubectl cordon worker-1
kubectl drain worker-1 --ignore-daemonsets --delete-emptydir-data

The cordon marks the worker as unschedulable. The drain evicts the pods. The cluster autoscaler notices the worker is unhealthy (no pods, no scheduling) and launches a new worker.

The autoscaler launch

The cluster autoscaler launches a new worker:

cluster-autoscaler: detected unhealthy node worker-1 (Ready,SchedulingDisabled, no pods)
cluster-autoscaler: scaling up node group workers (desired 5 → 6)
cluster-autoscaler: launching new worker at v1.34.0

The new worker is at the new version. The cluster scales up briefly to maintain the desired capacity.

The new worker joins

kubectl get nodes -o wide

NAME         STATUS   ROLES    AGE   VERSION   INTERNAL-IP
worker-1     Ready    <none>   30d   v1.33.0   10.0.1.20 (old)
worker-2     Ready    <none>   30d   v1.33.0   10.0.1.21
worker-3     Ready    <none>   30d   v1.33.0   10.0.1.22
worker-4     Ready    <none>   30d   v1.33.0   10.0.1.23
worker-5     Ready    <none>   30d   v1.33.0   10.0.1.24
worker-6     Ready    <none>   1m    v1.34.0   10.0.1.25 (new)

The new worker is at v1.34.0. The cluster has 6 workers briefly.

The old worker termination

After the new worker is Ready and the cluster has the desired capacity:

kubectl delete node worker-1

The kubectl delete removes the node object from the cluster. The cloud provider (AWS, GCP, Azure) terminates the VM.

sequenceDiagram
    participant O as Operator
    participant K as kubectl
    participant AS as API server
    participant CA as Cluster autoscaler
    participant CW as Cloud provider
    O->>K: kubectl cordon worker-1
    O->>K: kubectl drain worker-1
    K->>AS: evict pods
    CA->>CW: detect unhealthy worker
    CA->>CW: launch new worker at v1.34.0
    CW-->>CA: new worker ready
    CA->>AS: register new worker
    O->>K: kubectl delete node worker-1
    K->>AS: remove node object
    AS->>CW: terminate VM

The wave strategy

The surge pattern is applied in waves:

# Wave 1: worker-1
kubectl cordon worker-1
kubectl drain worker-1 --ignore-daemonsets --delete-emptydir-data
# Wait for the new worker to be Ready
kubectl delete node worker-1

# Wave 2: worker-2
kubectl cordon worker-2
kubectl drain worker-2 --ignore-daemonsets --delete-emptydir-data
# Wait for the new worker to be Ready
kubectl delete node worker-2

# ... continue for waves 3, 4, 5

The wave size is bounded by the workload’s PDB.

The Terraform approach

The surge pattern is repeatable in Terraform:

resource "aws_launch_template" "workers" {
  name_prefix   = "k8s-workers-"
  image_id      = "ami-v1.34.0"
  instance_type = "t3.large"
  user_data = <<-EOF
    #!/bin/bash
    # ... kubeadm join at v1.34.0
  EOF
}

resource "aws_autoscaling_group" "workers" {
  launch_template {
    id      = aws_launch_template.workers.id
    version = "$Latest"
  }
  min_size         = 3
  max_size         = 10
  desired_capacity = 5
}

Updating the image_id triggers the autoscaling group to use the new AMI for new workers.

The observability

The autoscaler emits metrics:

cluster_autoscaler_nodes_count
cluster_autoscaler_unschedulable_pods_count
cluster_autoscaler_node_group_size

The metrics are emitted by the cluster autoscaler’s /metrics endpoint. Prometheus scrapes them.

Cross-course references

• The Terraform course covers IaC-managed worker replacement.
• The Observability course covers autoscaler metrics.
• The AWS EKS course covers managed node groups.

Quiz

Production discipline

• Update the launch template first. The new AMI is the trigger.
• Cordon before drain. The worker is unschedulable before eviction.
• Wait for the new worker to be Ready. Validate before terminating the old.
• Terminate the old worker. kubectl delete node + cloud termination.
• Validate the cluster after each wave. kubectl get nodes, workload smoke tests.
• Document the surge plan. The launch template, the wave size, the PDB constraint.

The surge-and-replace is the cloud-native upgrade. Operating it well is integrating with the cluster autoscaler and validating at each wave.