kubeadm upgrade apply — the leader's upgrade

kubeadm upgrade apply is the command that runs on the leader of the control plane. It performs the full upgrade of the cluster’s control plane components: API server, controller-manager, scheduler, etcd. This lesson walks the command, the phase-by-phase execution, and the verification.

The command

sudo kubeadm upgrade apply v1.34.1

The argument is the target Kubernetes version. The version must be a valid semver with the major, minor, and patch components. The command does not support multi-minor upgrades (e.g., v1.31.0 → v1.34.0) in a single call.

The phases

The command runs several phases:

flowchart LR
    A[Validate config] --> B[Pull images]
    B --> C[Update ControlPlane static pod manifests]
    C --> D[Update ClusterConfiguration]
    D --> E[Update kubelet config]
    E --> F[Update kubelet binary]
    F --> G[Update node labels/annotations]

Each phase has a defined output. If any phase fails, the command aborts and the cluster is in a partially-upgraded state.

Phase 1: Validate config

The command reads the cluster’s kubeadm config from the kubeadm-config ConfigMap in kube-system:

kubectl -n kube-system get cm kubeadm-config -o yaml

The config includes the current version, the ClusterConfiguration, the InitConfiguration, and the KubeletConfiguration. The validate phase verifies the config is consistent and the upgrade target is supported.

Phase 2: Pull images

The new images for the target version are pulled:

Pulling images required for this upgrade.
This might take a minute or two depending on the speed of your internet connection.

The images are:

• registry.k8s.io/kube-apiserver:v1.34.1
• registry.k8s.io/kube-controller-manager:v1.34.1
• registry.k8s.io/kube-scheduler:v1.34.1
• registry.k8s.io/kube-proxy:v1.34.1
• registry.k8s.io/etcd:3.5.x
• registry.k8s.io/pause:3.9
• registry.k8s.io/coredns/coredns:v1.11.x

The pull is parallelized; the duration is bounded by the slowest image.

Phase 3: Update ControlPlane static pod manifests

The static pod manifests in /etc/kubernetes/manifests/ are updated:

ls /etc/kubernetes/manifests/

kube-apiserver.yaml
kube-controller-manager.yaml
kube-scheduler.yaml
etcd.yaml

Each manifest’s image field is updated to the new version. The kubelet sees the manifest change via the hash; it terminates the existing pod and starts a new one with the new image.

sequenceDiagram
    participant K as kubeadm
    participant FS as /etc/kubernetes/manifests/
    participant KL as kubelet
    participant AS as API server pod
    K->>FS: write new kube-apiserver.yaml (v1.34.1)
    K->>KL: notify via file watch
    KL->>AS: terminate old pod
    KL->>AS: start new pod (v1.34.1)
    AS-->>K: pod started

The restart is brief (seconds). The LB routes around; clients see a 503 and retry.

Phase 4: Update ClusterConfiguration

The cluster’s ClusterConfiguration is updated to reflect the new version. The KubeadmConfig ConfigMap is updated:

apiVersion: v1
data:
  ClusterConfiguration: |
    apiServer:
      extraArgs:
        authorization-mode: Node,RBAC
      timeoutForControlPlane: 4m0s
    apiVersion: kubeadm.k8s.io/v1beta4
    certificatesDir: /etc/kubernetes/pki
    clusterName: kubernetes
    ...
kind: ConfigMap
metadata:
  name: kubeadm-config
  namespace: kube-system

The version is recorded in the ConfigMap.

Phase 5: Update kubelet config

The KubeletConfiguration is updated to match the new version’s defaults. The kubelet’s config file is rewritten at /var/lib/kubelet/config.yaml.

Phase 6: Update kubelet binary

The kubelet binary is updated via the package manager:

# Debian/Ubuntu
sudo apt-get update
sudo apt-get install -y kubelet=1.34.1-1.34

# RHEL/Fedora
sudo yum install -y kubelet-1.34.1-1.34

The kubelet service is restarted:

sudo systemctl restart kubelet

The kubelet starts with the new version.

Phase 7: Update node labels/annotations

The cluster’s node labels and annotations are updated:

metadata:
  annotations:
    kubeadm.kubernetes.io/cluster-id: 1a2b3c4d
    kubeadm.kubernetes.io/control-plane-endpoint: lb:6443
  labels:
    node-role.kubernetes.io/control-plane: ""
    node.kubernetes.io/exclude-from-external-load-balancers: ""

The version label is implicit (it is reported by the kubelet, not stored as a label).

The output

The full output of kubeadm upgrade apply:

[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with:
[upgrade/config] $ kubectl -n kube-system get cm kubeadm-config -o yaml

[preflight] Running preflight checks
[preflight] Pulling images required for this upgrade
[preflight] This might take a minute or two depending on the speed of your internet connection.
[preflight] Note: Some of the preflight checks may take a few seconds to complete.

[upgrade] Running cluster health checks

[upgrade/health] Checking API server health...
[upgrade/health] Checking controller-manager health...
[upgrade/health] Checking scheduler health...
[upgrade/health] Checking etcd health...

Upgrade to version 1.34.1 is available; applied to:
  - kubeadm
  - kubelet
  - control plane

[upgrade/apply] Applying the new version to the cluster

[upgrade/apply] Pulling images: ...
[upgrade/apply] Updating static pod manifests
[upgrade/apply] Updating KubeletConfiguration
[upgrade/apply] Updating kube-apiserver static pod
[upgrade/apply] Updating kube-controller-manager static pod
[upgrade/apply] Updating kube-scheduler static pod
[upgrade/apply] Updating etcd static pod
[upgrade/apply] Restarting kubelet

[upgrade/apply] Waiting for control plane to be ready
[upgrade/apply] Waiting for API server to be healthy
[upgrade/apply] Waiting for API server to be ready
[upgrade/apply] Updating configuration
[upgrade/apply] Updating cluster information

[upgrade] Successful upgrade to version 1.34.1

The command is silent on success; the final line is the success indicator.

The verification

# Cluster version
kubectl version
# Server Version: v1.34.1

# Node version
kubectl get nodes
# All control plane nodes Ready, version v1.34.1

# Static pod state
kubectl get pods -n kube-system
# kube-apiserver, controller-manager, scheduler, etcd all Running

# API server endpoint
kubectl get endpointslices -n default

The cluster is at the new version.

Cross-course references

• The Linux course covers package manager-based binary upgrades.
• The Observability course covers alerting during upgrade windows.
• The Ansible course covers the equivalent of kubeadm upgrade apply for non-kubeadm clusters.

Quiz

Production discipline

• Run from the leader. The leader’s upgrade is the cluster’s upgrade.
• Snapshot before. The pre-upgrade snapshot is the rollback path.
• Watch the static pod restart. The brief unavailability is bounded by the LB.
• Verify at each phase. The output walks the phases; review each line.
• Don’t proceed on partial upgrade. If a phase fails, investigate before retrying.
• Document the upgrade. Step-by-step notes in the runbook.

The leader’s upgrade is the most disruptive step in the Kubernetes upgrade sequence. Operating it well is understanding the phases and the static pod restart.