Upgrade plan — prepare, control-plane, workers in order

A Kubernetes upgrade is a multi-step operation that affects every component: control plane, kubelet, CNI, add-ons. This lesson walks the planning, the version-slew policy, the backup-before-change gate, and the rollout order.

The upgrade plan

A Kubernetes upgrade is a structured operation:

flowchart LR
    P[Plan] --> B[Backup etcd]
    B --> V[Verify version skew]
    V --> C1[Upgrade control plane 1]
    C1 --> C2[Upgrade control plane 2]
    C2 --> C3[Upgrade control plane 3]
    C3 --> W1[Upgrade worker 1]
    W1 --> W2[Upgrade worker 2]
    W2 --> W3[Upgrade worker N]
    W3 --> V2[Validate]

The control plane upgrades first; then workers. Within the control plane, follower first, leader last.

The version skew policy

Kubernetes officially supports:

• kube-apiserver + etcd: minor mismatch within 1 version.
• kubelet: up to 3 minor versions behind API server.
• kube-controller-manager, kube-scheduler, kube-proxy: up to 1 minor version ahead of or behind API server.

Examples (valid):
  API server 1.34, kubelet 1.31-1.34 → ok
  API server 1.34, kube-proxy 1.33-1.35 → ok
  API server 1.34, etcd 3.6.x → ok

Examples (invalid):
  API server 1.34, kubelet 1.30 → invalid (4 minor mismatch)
  API server 1.34, etcd 3.4.x → invalid (version mismatch)

A simple upgrade (1.33 → 1.34) is well within skew. A jump (1.31 → 1.34) requires intermediate upgrades: 1.31 → 1.32 → 1.33 → 1.34.

The kubeadm upgrade plan

# Plan the upgrade
sudo kubeadm upgrade plan

[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with:
[upgrade/config] $ kubectl -n kube-system get cm kubeadm-config -o yaml

[preflight] Running preflight checks
[preflight] Pulling images required for this upgrade
[preflight] This might take a minute or two depending on the speed of your internet connection.
[preflight] Note: Some of the preflight checks may take a few seconds to complete.

[upgrade] Running cluster health checks

Upgrade to version 1.34.1 is available; applied to:
  - kubeadm
  - kubelet
  - control plane

The plan shows:

• The Kubernetes version available.
• The components affected.
• The health checks (preflight).

sudo kubeadm upgrade apply v1.34.1

This is the actual upgrade on the control plane host.

The kubeadm upgrade apply sequence

On the first control-plane host (cp-1 / leader):

sudo kubeadm upgrade apply v1.34.1

The command:

1. Verifies kubeadm and kubelet versions.
2. Reads the kubeadm-config ConfigMap.
3. Pulls new images.
4. Updates the static pod manifests (API server, etc).
5. Restarts the static pods (kubelet observes the manifest change).
6. Updates the cluster’s Kubernetes version.
7. Updates the kubelet binary via package manager.

After the command, cp-1 is at 1.34.1; cp-2 and cp-3 are still at 1.34.0. The cluster is in skew but functional.

$ sudo kubeadm upgrade plan

...

The follower control plane upgrade

On each additional control-plane host (follower first):

sudo kubeadm upgrade node

This:

1. Upgrades the kubeadm binary.
2. Upgrades the kubelet binary.
3. Restarts the kubelet service.

The control plane static pods are already at the new version (via the leader’s kubeadm upgrade apply); the follower’s kubeadm upgrade node does not change those.

After: cp-1 = 1.34.1, cp-2 = 1.34.1, cp-3 still 1.34.0. The cluster has 2/3 of control plane at the new version.

The worker upgrade

Workers are upgraded with kubeadm upgrade node:

sudo kubeadm upgrade node

Before running:

# Substitute your own value before running:
WORKER=worker-03

# Drain the worker first
kubectl drain "$WORKER" --ignore-daemonsets --delete-emptydir-data

Then on the worker:

sudo kubeadm upgrade node

After:

# The worker drained above:
WORKER=worker-03

# Bring back
kubectl uncordon "$WORKER"

The worker is at 1.34.1; the cluster’s control plane is at 1.34.1; the rest is still at 1.34.0.

The CNI upgrade

The CNI plugin must be compatible with the new Kubernetes version. Check:

• Cilium: supports N-1 to N+1 minor versions.
• Calico: same.
• Flannel: same.

Upgrade the CNI separately, after the control plane upgrade:

# Cilium
helm upgrade cilium cilium/cilium --version 1.16.x ...

# Calico
kubectl apply -f calico-updated.yaml

Compatibility with the control plane’s API version is the key.

The add-on upgrades

Add-ons (CoreDNS, kube-proxy, ingress controllers) must be compatible:

# Check kube-proxy version
kubectl -n kube-system get ds kube-proxy -o jsonpath='{.spec.template.spec.containers[0].image}'

# Update kube-proxy via kubeadm
sudo kubeadm upgrade apply v1.34.1
# ... this updates the kube-proxy static manifest

CoreDNS upgrades separately:

helm upgrade coredns coredns/coredns --version x.y.z

The verification at each step

# After control plane upgrade
kubectl get nodes
# Expect: control plane nodes Ready, version reported as 1.34.1

kubectl get pods -n kube-system
# Expect: kube-apiserver, etcd, controller-manager, scheduler all Running

# After worker upgrade
kubectl get nodes -o wide
# Expect: worker at 1.34.1; older workers still at 1.34.0

# Final validation
kubectl api-resources
# Expect: API resources loaded without error

The rollback path

If the upgrade fails:

1. Snapshot exists. Pre-upgrade snapshot was taken.
2. Restore etcd from snapshot (Part LXIX).
3. Restart cluster on the pre-upgrade version.
4. Investigate. What broke; what needs to change before retrying.

A rollback is heavy; the goal is to avoid needing one. The snapshot is the safety net.

The upgrade plan template

UPGRADE PLAN: 1.34.0 → 1.34.1
================================

Team contacts:
  - Lead: <name>
  - Backup: <name>
  - Vendor support: <info>

Window:
  - Start: <timestamp>
  - End: <timestamp>

Snapshot:
  - Pre-upgrade: <timestamp>
  - Post-upgrade: <timestamp>

Steps:
  1. Take etcd snapshot
  2. Verify snapshot integrity
  3. Run sudo kubeadm upgrade plan (review outputs)
  4. Run sudo kubeadm upgrade apply on cp-1 (leader)
  5. Wait for cp-1 to be Ready at v1.34.1
  6. Run sudo kubeadm upgrade node on cp-2, cp-3 (followers)
  7. Verify each follower is at v1.34.1
  8. Upgrade workers (sequentially, drain before upgrade)
  9. Update CNI
  10. Update add-ons (coredns, ingress, etc.)
  11. Final validation
  12. Take post-upgrade snapshot
  13. Document

Rollback:
  - Restore etcd from pre-upgrade snapshot
  - Re-bootstrap on the prior version

Approvals:
  - Change advisory board: <signed>
  - Operations team: <signed>

The discipline

• Backup before every upgrade. The snapshot is the rollback path.
• Validate version skew. A multi-minor jump is a recipe for incidents.
• Plan every step. Written plans capture the order and the verification.
• Test on staging first. Production upgrades mirror staging.
• Verify at each step. kubectl / crictl / alert state confirms the change.

Quiz

Production discipline

• Backup before upgrade. Always.
• One minor at a time. Validate at each step.
• Followers first, leader last. Both in control plane and the API server upgrade.
• Update CNI and add-ons. They need version-specific configuration.
• Verify at each step. Don’t proceed if validation fails.
• Document the plan and the outcome. The plan is the runbook; the outcome is the post-mortem input.

The upgrade is a planned, ordered, validated sequence. Operating it well is following the plan.