critical riskcluster affecting~120 min

Runbook: Upgrade the Kubernetes Control Plane

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Confirm the upgrade is within the supported version skew: control plane + kubelet + kubectl all within the documented range
· Confirm a recent etcd snapshot exists (within 24 hours): ls -lh /var/backups/etcd/
· Confirm kubeadm upgrade plan reports the target version with no blockers
· Confirm deprecated APIs in the manifests are not used in the target version: kubectl get pods -A -o json | jq
· Confirm the change ticket is peer-reviewed, change window is current, and the team is staffed
· Confirm cluster capacity can survive one control-plane node being unavailable at a time
· Capture the current kubeadm, kubelet, kubectl versions: kubeadm version; kubelet --version; kubectl version

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Upgrade kubeadm to the target version on the first control-plane node
2Run sudo kubeadm upgrade plan and capture the output to the change ticket
3Drain the first control-plane node: kubectl drain <cp1> --ignore-daemonsets
4Apply the upgrade: sudo kubeadm upgrade apply v<target>
5Upgrade kubelet and kubectl to the target version on the first control-plane node
6Restart kubelet: sudo systemctl restart kubelet
7Uncordon: kubectl uncordon <cp1>
8Confirm the first control-plane is healthy: kubectl get nodes, kubectl -n kube-system get pods -o wide
9Repeat on every additional control-plane node using kubeadm upgrade node instead of apply
10Confirm HA balance: every control-plane node reports the new version and Ready
11Upgrade cluster add-ons (CNI, CoreDNS, ingress controller) to versions compatible with the target Kubernetes version
12Validate every workload: kubectl get pods -A, smoke tests, dashboards
13Capture the upgrade record: versions, validation results, change ticket closure

4 · Verification

Confirm the procedure actually fixed the problem.

✓kubectl get nodes reports every node Ready with the target kubelet version
✓kubectl version reports both control plane and client at the target version
✓kubectl -n kube-system get pods reports every component Running with the target image tag
✓etcdctl ... endpoint health --cluster -w table reports every member healthy
✓kubectl get --raw=/healthz returns ok
✓A test workload runs end-to-end on the upgraded cluster
✓No Warning events in the last 5 minutes
✓Deprecated API usage scan is clean: kubectl get pods -A -o json | jq shows no removed APIs

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶If the first control-plane upgrade fails, kubeadm upgrade apply does not run; restore kubeadm to the previous version and re-apply
↶If the first control-plane upgrade succeeds but the cluster is unstable, restore from the pre-upgrade etcd snapshot (see kubernetes-rb-restore-etcd)
↶If the upgrade of an additional control-plane node fails, leave it on the previous version temporarily and proceed with the others only with explicit approval
↶Capture every log and the kubeadm output to the change ticket before any rollback
↶For an unrecoverable upgrade, the cluster must be rebuilt from scratch (see kubernetes-rb-recover-failed-control-plane)

6 · Escalation

When the runbook isn't enough, contact:

· Deprecated API is in use after the upgrade: identify the workloads using the removed API and migrate before retrying
· Cluster add-on (CNI, CoreDNS) is incompatible with the target version: pin to the last compatible version and escalate to platform ownership
· etcd upgrade fails on a control-plane node: restore the etcd data directory from the pre-upgrade snapshot and re-try with the correct etcd version
· API server cannot start with the new version: a flag has been removed or renamed; check the kubeadm changelog and the upgrade plan output
· Multiple control-plane nodes fail to upgrade: stop and escalate to vendor / upstream; do not proceed with a partially-upgraded control plane

A control-plane upgrade is a deliberate, multi-step operation that touches every component. The runbook walks one control-plane node at a time, with each step gated by the validation of the previous.

1. Pre-upgrade checks

Read-only / SafePre-upgrade checks


sudo kubeadm upgrade plan | tee /tmp/upgrade-plan.txt

# Confirm the cluster is healthy before any change
kubectl get nodes -o wide
kubectl -n kube-system get pods
etcdctl ... endpoint health --cluster -w table

# Confirm deprecated APIs are not used
# See https://kubernetes.io/docs/reference/using-api/deprecation-guide/
kubectl get pods -A -o json | jq -r ".items[].spec.containers[].image' | sort -u | head -30

2. Take a pre-upgrade etcd snapshot

Read-only / SafeTake a pre-upgrade etcd snapshot


sudo ETCDCTL_API=3 etcdctl \
--endpoints=https://127.0.0.1:2379 \
--cacert=/etc/kubernetes/pki/etcd/ca.crt \
--cert=/etc/kubernetes/pki/etcd/server.crt \
--key=/etc/kubernetes/pki/etcd/server.key \
snapshot save "$SNAP"
sudo etcdutl snapshot status "$SNAP" -w table
sudo rsync -a "$SNAP" backup@backup.internal:/srv/etcd/

The snapshot is the rollback path. Do not proceed without one.

3. Upgrade the first control-plane node

Read-only / SafeUpgrade the first control-plane node


kubectl drain <cp1> --ignore-daemonsets --delete-emptydir-data

# Upgrade kubeadm
sudo apt-mark unhold kubeadm && \
sudo apt-get update && sudo apt-get install -y kubeadm=<target-version>
sudo kubeadm version

# Apply the upgrade
sudo kubeadm upgrade apply v<target-version>

# Upgrade kubelet and kubectl
sudo apt-mark unhold kubelet kubectl && \
sudo apt-get install -y kubelet=<target-version> kubectl=<target-version>
sudo systemctl restart kubelet

# Uncordon
kubectl uncordon <cp1>

# Validate
kubectl get nodes -o wide
kubectl -n kube-system get pods -o wide | grep <cp1>

4. Upgrade additional control-plane nodes

Read-only / SafeUpgrade additional control-plane nodes


kubectl drain "$n" --ignore-daemonsets --delete-emptydir-data

# Upgrade kubeadm on the node
ssh "$n" -- sudo apt-mark unhold kubeadm && \
  sudo apt-get update && sudo apt-get install -y kubeadm=<target-version>

# Apply the node upgrade
ssh "$n" -- sudo kubeadm upgrade node

# Upgrade kubelet and kubectl
ssh "$n" -- sudo apt-mark unhold kubelet kubectl && \
  sudo apt-get install -y kubelet=<target-version> kubectl=<target-version>
ssh "$n" -- sudo systemctl restart kubelet

kubectl uncordon "$n"
done

kubectl get nodes -o wide

5. Upgrade cluster add-ons

Read-only / SafeUpgrade cluster add-ons


kubectl -n kube-system get configmap coredns -o yaml | grep image
helm upgrade coredns coredns/coredns --namespace kube-system --version=<target>

# CNI (Cilium example)
helm upgrade cilium cilium/cilium --namespace kube-system --version=<target>

# Ingress controller
helm upgrade ingress-nginx ingress-nginx/ingress-nginx --namespace ingress-nginx --version=<target>

# Metrics Server, etc.
helm upgrade metrics-server metrics-server/metrics-server --namespace kube-system --version=<target>

6. Validate

Read-only / SafeValidate


kubectl version
kubectl -n kube-system get pods -o wide
etcdctl ... endpoint health --cluster -w table
kubectl get --raw=/healthz; echo

# Smoke test
kubectl run smoketest --image=registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7 --restart=Never --command -- sleep 60
kubectl wait --for=condition=Ready pod/smoketest --timeout=60s
kubectl delete pod smoketest --wait=false

# Confirm no deprecated APIs
kubectl get pods -A -o json | jq -r ".items[] | select(.metadata.labels["kubernetes.io/created-by"] != null) | .metadata.namespace + "/" + .metadata.name' || echo "no deprecated APIs"

7. Capture the upgrade record

Read-only / SafeCapture the upgrade record


Control plane upgrade from v<previous> to v<target>""
echo "Cluster: $(kubectl config current-context)"
echo "Date: $(date -u +%Y%m%dT%H%M%SZ)"
echo "Operator: $USER"
echo "Change ticket: <ticket>"
echo ""
echo "Pre-upgrade checks: passed"
echo "Snapshot: $SNAP"
echo "Add-ons upgraded: <list>"
echo "Post-upgrade validation: passed"
echo ""

Common pitfalls

Symptom	Cause	Action
`kubeadm upgrade apply` refuses	A deprecated API is in use	Identify and migrate the API usage
API server fails to start after upgrade	A flag has been removed	Check the kubeadm upgrade notes
CoreDNS Pods CrashLoopBackOff after upgrade	CoreDNS version incompatible	Pin CoreDNS to the last compatible version
`kubectl get nodes` shows new version on control planes but old on workers	Workers have not been upgraded yet	Follow `kubernetes-rb-upgrade-worker-nodes`
etcd member cannot rejoin cluster after upgrade	etcd version mismatch	Restore the etcd data directory and re-try with the matching etcd version

A control-plane upgrade is one of the highest-risk operations in Kubernetes. The runbook moves one node at a time, validates after each step, and never proceeds without an etcd snapshot.