high riskservice affecting~90 min

Runbook: Upgrade Worker Nodes

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Confirm the control plane is at the target version: kubectl get nodes -o wide | grep control-plane
· Confirm cluster capacity can absorb one worker at a time being drained: kubectl describe nodes | grep -E "Allocatable|Allocated"
· Confirm every PDB allows the drain: kubectl get pdb -A -o jsonpath='{.items[*].status}' | jq
· Confirm the target kubelet version is within the supported version skew (control plane minus 3 minor versions)
· Confirm the container runtime version is compatible with the target kubelet
· Capture the current kubelet version per node: kubectl get nodes -o jsonpath='{.items[*].status.nodeInfo.kubeletVersion}'
· Capture the drain order (by AZ or node group): kubectl get nodes -L topology.kubernetes.io/zone,node.kubernetes.io/instance-type

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Upgrade kubeadm on the worker node first: apt-mark unhold kubeadm && apt-get install -y kubeadm=<target-version>
2Drain the worker node: kubectl drain <node> --ignore-daemonsets --delete-emptydir-data
3Upgrade kubelet and kubectl on the worker: apt-mark unhold kubelet kubectl && apt-get install -y kubelet=<target-version> kubectl=<target-version>
4Restart kubelet: sudo systemctl restart kubelet
5Upgrade the container runtime if needed: apt-get install -y containerd=<target-version> && systemctl restart containerd
6Confirm the kubelet registers with the new version: kubectl get node <node> -o jsonpath='{.status.nodeInfo.kubeletVersion}'
7Confirm the node is Ready: kubectl get node <node>
8Uncordon: kubectl uncordon <node>
9Repeat for every worker node, one at a time, validating between each
10After all workers are upgraded, validate cluster-wide: kubectl get nodes -o wide, workload smoke tests

4 · Verification

Confirm the procedure actually fixed the problem.

✓kubectl get nodes -o wide shows every node Ready with the target kubelet version
✓kubectl version confirms the cluster-wide version consistency
✓kubectl get pods -A -o wide --field-selector spec.nodeName=<node> shows Pods Running on the upgraded node
✓kubectl get pdb -A -o jsonpath='{.items[*].status}' | jq shows no Disallowed pods
✓No Warning events in the last 5 minutes
✓A representative workload runs correctly on the upgraded node
✓Dashboard metrics show no error spike or latency increase

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶If the kubelet upgrade fails to start, restore the previous kubelet binary and config: apt-get install -y kubelet=<previous-version>
↶If the node joins but Pods do not schedule, the kubelet config may have lost a flag; restore /var/lib/kubelet/config.yaml from backup
↶If the upgrade succeeded but a workload is broken, the workload is the issue; roll back the workload, not the kubelet
↶Capture the kubelet journal and the node condition before any rollback that may lose them
↶If multiple worker upgrades fail, escalate to platform ownership before attempting any further changes

6 · Escalation

When the runbook isn't enough, contact:

· Kubelet repeatedly fails to register: certificate issue; see kubernetes-rb-renew-cluster-certs
· Container runtime upgrade breaks Pod scheduling: restore the runtime binary and config from backup
· Node rejoins but Pods are stuck in ContainerCreating: CNI is incompatible with the new runtime; see kubernetes-rb-troubleshoot-cni
· A worker upgrade breaks PDB compliance across the cluster: the upgrade is moving too many nodes; reduce concurrency
· All workers upgraded but the cluster reports version skew: a control-plane or system component is on the previous version

Worker upgrades move through every node in the fleet, one at a time. The drain / upgrade / uncordon cycle is the runbook for each node; the planning is what makes the operation safe across the fleet.

1. Plan the order

Read-only / SafePlan the order


# Drain by zone, one worker per zone at a time
# This preserves availability during the upgrade

Drain order:

One zone at a time, one worker at a time
Within a zone, drain the worker with the fewest workloads first
Never drain the last worker in a zone if a PDB requires it

2. Upgrade kubeadm and kubectl on the worker

Read-only / SafeUpgrade kubeadm and kubectl on the worker


sudo apt-mark unhold kubeadm kubectl
sudo apt-get update
sudo apt-get install -y kubeadm=<target-version> kubectl=<target-version>

sudo kubeadm version
kubectl version --client=true

kubeadm upgrade node only works after kubeadm has been upgraded. Doing this step before the drain avoids a node-only tool being out of sync.

3. Drain the worker

Read-only / SafeDrain the worker


kubectl drain <node>  \
--ignore-daemonsets \
--delete-emptydir-data \
--grace-period=30 \
--timeout=15m

kubectl get pods -A -o wide --field-selector spec.nodeName=<node> | grep -v kube-system || echo "drained"

4. Upgrade kubelet and restart

Read-only / SafeUpgrade kubelet and restart


sudo apt-mark unhold kubelet
sudo apt-get install -y kubelet=<target-version>
sudo systemctl restart kubelet

# Wait for the kubelet to register
sleep 30
kubectl get node <node> -o jsonpath='{.status.nodeInfo.kubeletVersion}'

5. Upgrade the container runtime (if needed)

Read-only / SafeUpgrade the container runtime (if needed)


sudo apt-get install -y containerd.io=<target-version>
sudo systemctl restart containerd
sudo crictl info | jq -r '.status.runtimeReady'

# If the kubelet cannot talk to the runtime, check the kubelet's --container-runtime-endpoint
sudo cat /var/lib/kubelet/config.yaml | grep -A2 runtime

A runtime upgrade is necessary if the target kubelet requires a newer runtime. Confirm the upgrade matrix in the kubelet release notes.

6. Uncordon and validate

Read-only / SafeUncordon and validate


kubectl get node <node> -o wide

# Wait for the node to admit a Pod
kubectl run smoketest --image=registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7 \
--restart=Never --command -- sleep 30
kubectl wait --for=condition=Ready pod/smoketest --timeout=60s
kubectl delete pod smoketest --wait=false

7. Repeat for every worker

8. Cluster-wide validation

Read-only / SafeCluster-wide validation


# Every node should report the target kubelet version

kubectl -n kube-system get pods -o wide | grep -E 'coredns|cilium|kube-proxy'
# Add-ons should still be Running

kubectl get pods -A | wc -l
# Should match the pre-upgrade count (modulo workload churn)

# A representative smoke test
curl -fsS https://app.prod.example/healthz
curl -fsS https://app.prod.example/version

9. Capture the upgrade record

Read-only / SafeCapture the upgrade record


Worker upgrade from v<previous> to v<target>""
echo "Workers upgraded: $(kubectl get nodes -o name | wc -l)"
echo "Control plane at: $(kubectl version -o yaml | grep -E 'minor|kube-apiserver' | head)"
echo "Operator: $USER"
echo "Date: $(date -u +%Y%m%dT%H%M%SZ)"
echo "Change ticket: <ticket>"
echo "Rollback plan: see kubernetes-rb-renew-cluster-certs and kubeadm downgrade"

Common pitfalls

Symptom	Cause	Action
Kubelet fails to register	Cert rotation issue or version skew with API server	See `kubernetes-rb-renew-cluster-certs`
Pods stuck in `ContainerCreating` after kubelet upgrade	Runtime mismatch with kubelet	Upgrade runtime to the version kubelet expects
Node rejoins but is unschedulable	Taints or labels missing	Inspect `kubectl describe node` and re-apply the expected taints/labels
Drain blocks on PDB	Capacity issue	Reduce concurrency; do not bypass
`kubectl get nodes` reports the new version but Pods do not start	Container runtime socket path changed	Confirm kubelet `--container-runtime-endpoint` matches the runtime

A worker upgrade is a deliberate churn of the fleet. The runbook respects PDBs, drains one node at a time, and validates between each step so the next node inherits a known-good cluster.