Upgrade responsibilities — who upgrades what and when

Upgrades are one of the most consequential operational events in Kubernetes. This lesson walks the responsibility split, the upgrade sequence, version skew rules, and the operational discipline.

The responsibility split

flowchart LR
    A[Managed upgrades] --> A1["Control plane: provider"]
    A --> A2["Worker nodes: provider (managed node groups) or operator"]
    A --> A3["Node OS patching: provider"]
    B[Self-managed upgrades] --> B1["Control plane: operator"]
    B --> B2["Worker nodes: operator"]
    B --> B3["Node OS patching: operator"]

The split:

Managed (EKS, AKS, GKE):

• Control plane upgrade: provider (button or API).
• Worker node upgrade: provider (managed node groups) or operator (self-managed node groups).
• Node OS patching: provider (managed node groups).
• Kubernetes version selection: limited (provider’s supported versions).

Self-managed (kubeadm):

• Control plane upgrade: operator (kubeadm upgrade).
• Worker node upgrade: operator (kubeadm upgrade + kubelet).
• Node OS patching: operator.
• Kubernetes version selection: any (the operator chooses).

The upgrade sequence

flowchart LR
    A[Cluster on N] --> B[Upgrade control plane to N+1]
    B --> C[Verify control plane on N+1]
    C --> D[Drain worker 1]
    D --> E[Upgrade worker 1 to N+1]
    E --> F[Uncordon worker 1]
    F --> G[Repeat for each worker]
    G --> H[Cluster on N+1]

The sequence:

1. Upgrade the control plane. kubeadm upgrade apply on the first control-plane node; verify API server is on N+1.
2. Verify control plane. kubectl get nodes shows control-plane nodes on N+1.
3. Drain worker 1. Move workloads to other workers.
4. Upgrade worker 1. Update kubeadm, kubelet, kubectl; restart kubelet.
5. Uncordon worker 1. Allow workloads to schedule.
6. Repeat for each worker.
7. Verify cluster. kubectl get nodes shows all nodes on N+1.

For a managed cluster, the control plane upgrade is automated; the worker upgrade is automated by managed node groups.

Version skew rules

flowchart LR
    A["API server: N"] --> B["kubelet: N, N-1, N-2"]
    A --> C["kube-proxy: N, N-1"]
    A --> D["client libraries: N, N-1, N-2"]

The supported version skew:

• API server and kubelet. The kubelet must be within 3 minor versions of the API server. So kubelet 1.34, 1.33, or 1.32 can talk to API server 1.34.
• API server and kube-proxy. kube-proxy must be within 2 minor versions. So kube-proxy 1.34 or 1.33 can talk to API server 1.34.
• API server and client libraries. kubectl and other client libraries must be within 1 minor version (for kubectl: same version or one behind).

Skew outside these rules is unsupported; the cluster may not function correctly.

The operational trade-offs

The trade-offs:

• Managed. Faster upgrades (provider handles the heavy lifting); less flexibility in version selection; cost.
• Self-managed. Slower upgrades (operator handles every step); full flexibility in version selection; more operational burden.

The discipline of testing upgrades

flowchart LR
    A[New version released] --> B[Upgrade staging cluster]
    B --> C[Run test suite]
    C --> D{All pass?}
    D -->|No| E["Investigate, fix"]
    D -->|Yes| F[Schedule production upgrade]
    F --> G[Upgrade production]
    G --> H[Validate]

The discipline:

1. Test in staging first. Upgrade a staging cluster to the new version. Run the full test suite.
2. Verify the skew. Confirm kubelet and kube-proxy versions are within skew.
3. Schedule the production upgrade. Coordinate with stakeholders; plan for maintenance window.
4. Upgrade production. Follow the sequence: control plane first, workers second.
5. Validate. Run smoke tests, monitor metrics, verify workloads are functional.

The operational failure modes

Upgrades fail for predictable reasons:

• Wrong upgrade order. Workers upgraded before control plane; cluster is non-functional.
• Skew exceeded. kubelet too far behind API server; kubelet cannot authenticate.
• APIs deprecated. Workloads using APIs removed in the new version fail to start.
• CNI / CSI not upgraded. Network plugins incompatible with new version; Pods stay Pending.
• Backup taken before upgrade. An upgrade that breaks the cluster requires restore; without a recent backup, recovery is impossible.

Quiz

Production discipline

Upgrades in production rest on five non-negotiable elements:

• Control plane first, workers second. The upgrade order is non-negotiable.
• Version skew within bounds. kubelet within 3, kube-proxy within 2 of API server.
• Test in staging first. A production upgrade that has never been tested in staging is a gamble.
• Take an etcd snapshot before upgrade. The recovery path for a failed upgrade is restore from snapshot.
• Audit deprecated APIs. The target version may have removed APIs that workloads use.

Upgrades are the most consequential operational event. The discipline is to test, follow the order, and have a recovery path.