kubelet-only upgrade — the patch-level maintenance

A kubelet-only upgrade updates the kubelet binary on a worker without changing the cluster version. The pattern is used for patch-level maintenance (e.g., 1.34.0 → 1.34.1 on the kubelet while the control plane is at 1.34.0). This lesson walks the pattern, the use cases, and the verification.

The pattern

flowchart LR
    A[Cluster at v1.34.0] --> B[Cluster version unchanged]
    B --> C[Upgrade kubelet binary on worker]
    C --> D[Restart kubelet service]
    D --> E[Worker joins at new kubelet]

The cluster version (kubectl version) is unchanged. The kubelet binary is updated on the worker.

The use cases

• Security patch. A CVE is discovered in the kubelet; the operator patches the kubelet binary without upgrading the entire cluster.
• Bug fix. A kubelet bug is fixed in a patch release; the operator updates the kubelet.
• Staged upgrade. The control plane is at v1.34.0; the workers need to be at v1.34.1 to align with a forthcoming cluster upgrade.

The pattern is common in production.

The procedure

# On the worker
ssh worker-1

# Drain
kubectl drain worker-1 --ignore-daemonsets --delete-emptydir-data

# Update the kubelet package
sudo apt-get update
sudo apt-get install -y kubelet=1.34.1-1.34

# Restart kubelet
sudo systemctl restart kubelet

# Verify
kubectl uncordon worker-1

# Confirm
kubectl get nodes worker-1 -o wide

The kubelet binary is updated; the kubelet service is restarted with the new version.

The kubeadm upgrade node

The kubeadm equivalent:

# On the worker
sudo kubeadm upgrade node

The command upgrades the kubeadm and kubelet binaries. The kubelet service is restarted.

The kubeadm upgrade node command is the canonical command for kubelet-only upgrades when the cluster is managed by kubeadm.

The cluster version

The cluster version is unchanged:

kubectl version

Server Version: v1.34.0

The kubelet version is updated:

kubectl get nodes worker-1 -o jsonpath='{.status.nodeInfo.kubeletVersion}'

v1.34.1

The cluster version is at 1.34.0; the kubelet is at 1.34.1. The cluster is in skew (kubelet is 1 patch ahead of the cluster version), but the skew policy allows this (the kubelet is forward-compatible with the apiserver).

The validators

# On the worker
kubelet --version

Kubernetes v1.34.1

The kubelet version is reported.

journalctl -u kubelet -n 100

The kubelet logs confirm the new version.

The verify with kubectl

kubectl get nodes worker-1 -o wide

NAME       STATUS   ROLES    AGE   VERSION   INTERNAL-IP
worker-1   Ready    <none>   30d   v1.34.1   10.0.1.20

The VERSION column shows v1.34.1 (the kubelet version).

The cluster version

The cluster version is reported separately:

kubectl version

Server Version: v1.34.0

The server version is v1.34.0 (the apiserver version).

kubectl get nodes -o jsonpath='{.items[0].status.nodeInfo.kubeletVersion}'

v1.34.1

The kubelet version is v1.34.1.

The two versions differ; the cluster is in skew.

The cluster upgrade

After the kubelet is updated, the cluster can be upgraded (control plane to v1.34.1) to bring the cluster version in line:

# On the leader
sudo kubeadm upgrade apply v1.34.1

The cluster version is now v1.34.1; the kubelet is at v1.34.1. The skew is resolved.

The kubelet cert rotation

The kubelet’s client cert (rotated at 75% of validity) is unchanged by the kubelet-only upgrade. The kubelet’s cert is rotated by the kubelet itself, not by the upgrade.

The discipline

• Document the kubelet version. The cluster version is v1.34.0; the kubelet is at v1.34.1. The skew is recorded.
• Plan the cluster upgrade. The kubelet-only upgrade is a stepping stone; the cluster upgrade follows.
• Validate the worker. The kubelet version matches the binary version.
• Run smoke tests. The worker resumes normal operation.

Cross-course references

• The Linux course covers package management.
• The Observability course covers kubelet metrics.
• The Ansible course covers idempotent package upgrades.

Quiz

Production discipline

• Document the skew. Cluster version vs kubelet version.
• Stage the cluster upgrade. The kubelet-only upgrade is a stepping stone.
• Validate the worker. The kubelet version matches the binary.
• Track the CVE. The CVSS score, the patched version, the rollout.
• Plan the cluster upgrade. The kubelet-only upgrade is the patch; the cluster upgrade is the planned event.

The kubelet-only upgrade is patch-level maintenance. Operating it well is documenting the skew and planning the cluster upgrade.