Worker drain and upgrade — the in-place upgrade pattern

The worker upgrade is the second half of the Kubernetes upgrade sequence. The control plane is at the new version; the workers follow. This lesson walks the per-worker upgrade pattern: drain, upgrade, restart, uncordon, and validate.

The drain

Before the upgrade, the worker is drained:

kubectl drain worker-1 \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --force

The flags:

• --ignore-daemonsets: DaemonSet pods are not evicted (they are system pods; they survive on every node).
• --delete-emptydir-data: pods with emptyDir volumes are evicted (their data is lost on the new pod).
• --force: pods that are not managed by a controller (RS, RC, Deployment, etc.) are forcefully deleted.

The drain respects PDBs. If a workload’s PDB would be violated by the eviction, the drain fails.

flowchart LR
    A[Worker Ready] --> B[kubectl drain]
    B --> C[Pods evicted / rescheduled]
    C --> D[Worker cordoned]
    D --> E[Worker empty]

The upgrade

On the worker host:

ssh worker-1 sudo kubeadm upgrade node

The command:

1. Upgrades the kubeadm binary.
2. Upgrades the kubelet binary.
3. Restarts the kubelet service.

The static pod manifests were already updated by the leader’s kubeadm upgrade apply; the workers’ kubeadm upgrade node does not re-apply them.

The kubelet service is restarted:

ssh worker-1 sudo systemctl restart kubelet

The kubelet starts with the new version.

The validation

The worker’s kubelet registers with the API server:

kubectl get nodes worker-1 -o wide

NAME       STATUS   ROLES    AGE   VERSION   INTERNAL-IP   ...
worker-1   Ready    <none>   24h   v1.34.1   10.0.1.20     ...

The VERSION column shows v1.34.1 — the new version.

kubectl get pods -A -o wide | grep worker-1

NAMESPACE   NAME                              READY   STATUS    NODE
kube-system kube-proxy-xyz                   1/1     Running   worker-1
kube-system cilium-xyz                       1/1     Running   worker-1
...

The DaemonSet pods are back on the worker.

The uncordon

The worker is uncordoned:

kubectl uncordon worker-1

The worker is now available for new pod scheduling.

The wave strategy

The worker upgrade is done in waves. A wave is a group of workers that are drained and upgraded together. The wave size is bounded by the workload’s PDB:

# Wave 1: worker-1, worker-2 (4 workers, maxUnavailable: 2)
for w in worker-1 worker-2; do
  kubectl drain $w --ignore-daemonsets --delete-emptydir-data
  ssh $w sudo kubeadm upgrade node
  ssh $w sudo systemctl restart kubelet
  kubectl uncordon $w
done

# Validate wave 1
kubectl get nodes -o wide

# Wave 2: worker-3, worker-4
for w in worker-3 worker-4; do
  kubectl drain $w --ignore-daemonsets --delete-emptydir-data
  ssh $w sudo kubeadm upgrade node
  ssh $w sudo systemctl restart kubelet
  kubectl uncordon $w
done

The wave size is bounded by the lowest PDB in the cluster.

The skip-options

Some workers are skipped during the upgrade:

• Spot / preemptible workers. Spot workers may be terminated by the cloud provider; not upgrading them is acceptable.
• Workers running critical workloads. Workers hosting workloads with maxUnavailable: 0 require a separate plan (Part LXXX).
• Workers in a different cluster. Workers in a regional cluster are upgraded independently.

The skip is recorded in the runbook.

The surge strategy

A surge strategy replaces workers with new ones (scheduled elsewhere) rather than upgrading in place:

# Drain the worker
kubectl drain worker-1 --ignore-daemonsets --delete-emptydir-data

# Mark the worker for retirement
kubectl label node worker-1 kubelet.kubernetes.io/retire-=true

# The cluster autoscaler (Part LXXXI) launches a new worker
# to replace the drained one

The drained worker is terminated (or left for a separate retirement process). The new worker is launched at the new kubelet version. This is the AWS / GCP / Azure pattern.

The surge strategy is preferred for cloud-managed node groups but can be reproduced on-prem with Cluster Autoscaler.

The in-place vs surge tradeoffs

The choice is per-cluster:

• In-place for on-prem clusters where VMs are long-lived.
• Surge for cloud-managed clusters where VMs are ephemeral.

Cross-course references

• The Linux course covers service restart semantics.
• The Terraform course covers IaC-managed node replacement.
• The Proxmox course covers VM lifecycle for on-prem clusters.

Quiz

Production discipline

• Drain before upgrade. Always.
• Respect the PDB. Wave size is bounded by the lowest PDB.
• Verify at each wave. kubectl get nodes, workload smoke tests.
• Take a wave-by-wave snapshot. Long upgrades benefit from intermediate snapshots.
• Document the wave plan. The wave order, the PDB constraint, the rollback path.
• Test the upgrade on staging. Catch the wave-size issue before production.

The worker upgrade is the second half of the upgrade. Operating it well is draining, upgrading, restarting, and uncordoning in waves.