Upgrade sequence — read release notes, backup, control plane, then workers in waves

A Kubernetes upgrade is a multi-step operation that affects every component: control plane, kubelet, CNI, add-ons. The sequence is not negotiable — control plane first, then workers, validated at each step. This lesson walks the full sequence from release notes to post-upgrade validation.

The 10-step upgrade

flowchart LR
    A[1. Read release notes] --> B[2. Backup etcd]
    B --> C[3. Verify version skew]
    C --> D[4. Upgrade control plane 1]
    D --> E[5. Upgrade control plane 2]
    E --> F[6. Upgrade control plane 3]
    F --> G[7. Validate]
    G --> H[8. Upgrade workers in waves]
    H --> I[9. Update CNI and add-ons]
    I --> J[10. Post-upgrade snapshot + validate]

Each step has a defined input (the previous step’s output) and a defined output (the next step’s input). Skip a step and the next becomes risky.

Step 1: Read the release notes

Before any tool is run, the release notes are read. The notes list:

• Deprecated APIs (Part LXXIX). APIs that are removed in this release or scheduled for removal.
• Default changes. New defaults that may break existing workloads.
• Bug fixes. Issues that may affect your cluster.
• Migration guides. Steps required for features that are upgraded.

# Open the release notes
firefox https://kubernetes.io/blog/2026/08/release-1.34/

The release notes are the operator’s contract with the project. Reading them is the first step.

Step 2: Backup etcd

The snapshot is the rollback path:

sudo kubeadm upgrade plan
# Confirm the upgrade is to the desired version
sudo kubeadm upgrade apply v1.34.1 --etcd-upgrade=false
# OR
sudo kubeadm upgrade apply v1.34.1
# The snapshot is taken automatically if --etcd-upgrade is true

Or manually:

sudo ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
  --key=/etc/kubernetes/pki/etcd/healthcheck-client.key \
  --endpoints=https://127.0.0.1:2379 \
  snapshot save /var/backups/etcd-pre-upgrade-$(date +%Y%m%d).db

sudo etcdutl snapshot status /var/backups/etcd-pre-upgrade-*.db

The snapshot is uploaded off-cluster:

sudo cp /var/backups/etcd-pre-upgrade-*.db /mnt/external/etcd/

The upgrade only proceeds if the snapshot is verified.

Step 3: Verify version skew

The version skew policy (Part LXXVIII) defines which component versions are compatible. For a 1.34.0 → 1.34.1 upgrade:

• API server: 1.34.1 (target)
• kube-controller-manager, kube-scheduler, kube-proxy: 1.34.1 (target)
• kubelet: 1.34.1 (target) — workers follow the control plane
• etcd: 3.5.x (target)
• kubectl: 1.34.x (operator workstation)

The skew is small (patch version). For minor upgrades (1.33 → 1.34), the skew is also small. For multi-minor upgrades (1.31 → 1.34), the intermediate steps are required.

Step 4: Upgrade the first control plane

On the first control-plane host (the leader):

sudo kubeadm upgrade plan
# Review the output

sudo kubeadm upgrade apply v1.34.1

The command:

1. Verifies the version skew.
2. Pulls the new images.
3. Updates the static pod manifests.
4. Restarts the static pods (API server, controller-manager, scheduler, etcd).
5. Updates the cluster’s Kubernetes version annotation.
6. Updates the kubelet binary via the package manager.

The first control plane is at 1.34.1; the rest are still at 1.34.0. The cluster is in skew but functional.

Step 5: Upgrade the remaining control plane

On each remaining control-plane host (follower first):

sudo kubeadm upgrade node

The command:

1. Upgrades the kubeadm binary.
2. Upgrades the kubelet binary.
3. Restarts the kubelet service.

The control plane static pods were already updated by the leader’s kubeadm upgrade apply; the followers’ kubeadm upgrade node does not re-apply them.

After: cp-1 = 1.34.1, cp-2 = 1.34.1, cp-3 still 1.34.0. The cluster is at 2/3 of the control plane on the new version.

Step 6: Validate the control plane

After all control plane hosts are at 1.34.1:

kubectl get nodes
# All control plane nodes Ready at v1.34.1

kubectl get pods -n kube-system
# All control plane pods Running

kubectl api-resources
# API resources loaded without error

kubectl api-versions
# Versions match the expected set

The cluster is healthy on the new version.

Step 7: Upgrade workers in waves

The workers are upgraded in waves (Part LXXX). The wave strategy:

• Wave size: the wave size is the number of workers that can be unavailable simultaneously without violating the workload’s PDB.
• Drain: each worker is drained before the upgrade.
• Upgrade: kubeadm upgrade node on the worker.
• Uncordon: the worker is brought back into service after the upgrade.

# Wave 1: drain + upgrade + uncordon
for worker in worker-1 worker-2; do
  kubectl drain $worker --ignore-daemonsets --delete-emptydir-data
  ssh $worker sudo kubeadm upgrade node
  ssh $worker sudo systemctl restart kubelet
  kubectl uncordon $worker
done

# Wave 2: same for worker-3, worker-4
# Wave 3: same for worker-5

The wave size is bounded by the workload’s PDB. A maxUnavailable: 1 PDB means one worker at a time. A maxUnavailable: 25% PDB means 25% of workers at a time.

Step 8: Update CNI and add-ons

The CNI plugin must be compatible with the new Kubernetes version. Typical CNI upgrade:

# Cilium
helm upgrade cilium cilium/cilium --version 1.16.x

# Calico
kubectl apply -f calico-updated.yaml

Add-ons are similarly updated:

# CoreDNS
helm upgrade coredns coredns/coredns --version 1.11.x

# kube-proxy
kubectl -n kube-system get ds kube-proxy -o yaml | kubectl apply -f -

Step 9: Post-upgrade snapshot

After the upgrade is complete:

sudo ETCDCTL_API=3 etcdctl ... snapshot save /var/backups/etcd-post-upgrade-$(date +%Y%m%d).db

The post-upgrade snapshot is the new baseline. The pre-upgrade snapshot is the rollback path.

Step 10: Validate the cluster

Final validation:

kubectl get nodes -o wide
# All nodes Ready at v1.34.1

kubectl get pods -A
# All pods Running or Completed

kubectl get events --sort-by='.lastTimestamp' | head -30
# No critical errors

# Run a workload smoke test
kubectl run nginx-test --image=nginx --rm -it --restart=Never --command -- nginx -v

The cluster is upgraded.

Cross-course references

• The Linux course covers systemd service restart semantics.
• The Observability course covers alert validation during upgrade windows.
• The Ansible course covers idempotent package upgrades (yum, apt).

Quiz

Production discipline

• Read the release notes first. Always.
• Snapshot before upgrading. The pre-upgrade snapshot is the rollback path.
• Followers first, leader last. Contain the API server restart to the leader.
• Workers in waves. Respect the workload PDBs.
• Update CNI and add-ons after control plane. They need version-specific config.
• Take a post-upgrade snapshot. The new baseline.
• Validate at each step. Don’t proceed if validation fails.

The upgrade is a planned, ordered, validated sequence. Operating it well is following the sequence and verifying at each step.