Worker upgrade patterns — in-place vs surge

Workers can be upgraded in two patterns: in-place (the host is reused, the kubelet is upgraded on the existing host) or surge (a new VM is launched at the new version, the old VM is drained and terminated). The choice is per-cluster: on-prem prefers in-place; cloud prefers surge. This lesson walks both patterns and the decision criteria.

The two patterns

flowchart LR
    A[Cluster at v1.33.0] --> B{Upgrade pattern?}
    B -->|in-place| C[Drain worker]
    C --> D[kubeadm upgrade node]
    D --> E[Uncordon worker]
    B -->|surge| F[Drain worker]
    F --> G[Launch new worker at v1.34.0]
    G --> H[Terminate old worker]

The patterns differ in cost (in-place reuses the host; surge requires a new VM) and disruption (in-place has the kubelet downtime; surge has the pod reschedule).

The in-place pattern

The in-place pattern reuses the host:

# On the worker
kubectl drain worker-1 --ignore-daemonsets --delete-emptydir-data

ssh worker-1 sudo kubeadm upgrade node
ssh worker-1 sudo systemctl restart kubelet

kubectl uncordon worker-1

The kubelet is restarted with the new version. The host is the same VM; only the kubelet binary is updated.

The in-place pattern is preferred when:

• VMs are long-lived (on-prem).
• The cluster has fewer workers than the new VMs would cost.
• The kubelet downtime is acceptable (minutes).

The surge pattern

The surge pattern replaces the worker with a new VM:

# Mark the old worker for retirement
kubectl cordon worker-1
kubectl drain worker-1 --ignore-daemonsets --delete-emptydir-data

# Launch a new worker at the new version
# (Via cluster autoscaler, Terraform, or cloud CLI)

# Once the new worker is Ready, terminate the old worker
# (Via kubectl delete node, cloud CLI, or Terraform)

The new VM is at the new version; the old VM is terminated.

The surge pattern is preferred when:

• VMs are ephemeral (cloud).
• The cluster has a cluster autoscaler that can launch new workers.
• The new VM cost is acceptable.

The cluster autoscaler integration

The cluster autoscaler (Part LXXXI) integrates with the surge pattern:

flowchart LR
    A[Cordon worker] --> B[The worker is unhealthy for the ASG]
    B --> C[Cluster autoscaler launches a new worker]
    C --> D[The new worker joins the cluster]
    D --> E[The old worker is terminated]

The cluster autoscaler replaces the unhealthy worker with a new one. The integration is automatic.

The surge strategy in Terraform

The surge pattern is reproducible in Terraform:

resource "aws_instance" "worker" {
  count         = 5
  ami           = "ami-v1.34.0"
  instance_type = "t3.large"
  user_data = <<-EOF
    #!/bin/bash
    # ... kubeadm join at v1.34.0
  EOF
}

The Terraform state tracks the workers. Replacing the AMIs (machine images) and applying Terraform launches new workers at the new version.

The decision criteria

The decision framework:

The decision is per-cluster. Both patterns are valid.

The hybrid pattern

Some clusters use a hybrid:

• In-place for control plane and certain workers.
• Surge for the bulk of workers.

The hybrid is common in mixed environments (e.g., on-prem control plane + cloud workers).

Cross-course references

• The Terraform course covers IaC-managed worker replacement.
• The Proxmox course covers VM lifecycle for on-prem.
• The Ansible course covers idempotent worker configuration.

Quiz

Production discipline

• Choose the pattern at cluster design time. The decision is per-cluster.
• Document the pattern in the runbook. The decision, the rationale, the procedure.
• Test the pattern on staging. Catch the issues before production.
• Hybrid is acceptable. Some clusters use both patterns.
• Validate the cluster after each wave. kubectl get nodes, workload smoke tests.

The worker upgrade pattern is the cluster’s lifecycle decision. Operating it well is choosing the right pattern and following it consistently.