kube-apiserver vs kubelet — the asymmetric skew

The asymmetric skew between kube-apiserver and kubelet is the foundation of rolling worker upgrades. The kubelet can be up to 3 minor behind the apiserver, but not 1 minor ahead. This lesson walks the asymmetry, why it exists, and how it enables rolling upgrades.

The asymmetry

The skew policy:

kube-apiserver: 1.34.0
kubelet: 1.31.x, 1.32.x, 1.33.x, 1.34.x → all OK
kubelet: 1.35.x → INVALID (kubelet ahead of apiserver)

The kubelet can be 3 minor behind, but not 1 minor ahead.

flowchart LR
    A[apiserver 1.34] --> B[kubelet 1.31]
    A --> C[kubelet 1.32]
    A --> D[kubelet 1.33]
    A --> E[kubelet 1.34]
    A -.-> F[kubelet 1.35 INVALID]

Why the asymmetry

The Kubernetes API server guarantees backward compatibility for older clients. The guarantee:

• A new apiserver tolerates old kubelet calls (older API versions, deprecated fields).
• The old apiserver does not necessarily understand new kubelet calls (new fields, new API endpoints).

The asymmetry is what enables rolling worker upgrades:

sequenceDiagram
    participant CP as Control plane
    participant W1 as Worker 1 (old kubelet)
    participant W2 as Worker 2 (new kubelet)
    Note over CP: Upgrade to 1.34
    CP->>W1: api call (1.31 kubelet, 1.34 apiserver)
    Note over W1: Old kubelet talks to new apiserver
    W1-->>CP: ok (backward compatible)
    Note over CP,W2: Worker 2 upgraded
    W2->>CP: api call (1.34 kubelet, 1.34 apiserver)
    W2-->>CP: ok (current version)

The kubelet on Worker 1 is at 1.31; the apiserver is at 1.34. The kubelet’s API calls are still supported because the 1.34 apiserver understands the 1.31 kubelet’s API.

The asymmetry’s limits

The asymmetry is bounded:

• 3 minor behind: the kubelet must be at most 3 minor versions behind the apiserver.
• Not ahead: the kubelet cannot be ahead of the apiserver.

The 3-minor behind limit is a maintenance burden: the project keeps API versions stable for at least 3 minor releases after they are deprecated. Beyond 3 minors, the deprecated API may be removed.

The asymmetry in a rolling upgrade

Step 1: All at 1.33.0 (cluster)
Step 2: apiserver at 1.34.0, kubelets at 1.33.0 (workers not yet upgraded)
Step 3: Some kubelets at 1.34.0, others at 1.33.0 (rolling)
Step 4: All at 1.34.0 (complete)

The skew is maintained at each step. The workers can be upgraded in waves or as a surge; the apiserver is indifferent to the workers’ versions until they exceed the 3-minor behind limit.

The asymmetry and the kubelet’s CSR

The kubelet’s CSR (Certificate Signing Request) is signed by the kube-controller-manager. The kubelet’s cert (following the cluster CA) is rotated at 75% of validity. The CSR flow is part of the kubelet’s bootstrap; the asymmetry does not affect it.

The validation

# Get the cluster's apiserver version
kubectl version
# Server Version: v1.34.0

# Get the kubelet version on each node
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.nodeInfo.kubeletVersion}{"\n"}{end}'

cp-1       v1.34.0
cp-2       v1.34.0
cp-3       v1.34.0
worker-1   v1.31.0   # 3 minor behind, OK
worker-2   v1.32.0   # 2 minor behind, OK
worker-3   v1.33.0   # 1 minor behind, OK
worker-4   v1.34.0   # current, OK
worker-5   v1.34.0   # current, OK

The output confirms the skew is within policy.

The failure mode

A kubelet that is 4+ minor behind is not supported. The apiserver may reject the kubelet’s calls (e.g., the API versions the kubelet uses are removed). The cluster loses the node.

INVALID for 1.34.0 apiserver:
  kubelet: 1.30.x → kubelet 4 minor behind
  The kubelet cannot register or run pods.

The operator must upgrade the kubelet to bring it within skew.

The kubeadm upgrade flow

The kubeadm upgrade (Part LXXVII) follows the skew:

# On the leader (control plane)
sudo kubeadm upgrade apply v1.34.1
# Apiserver is now at 1.34.1; kubelets are still at 1.33.0.

# On each worker (in waves)
sudo kubeadm upgrade node
# Kubelet is now at 1.34.1.

The skew is maintained throughout the upgrade.

Cross-course references

• The Linux course covers binary compatibility concepts.
• The Terraform course covers Kubernetes provider version pinning.
• The Ansible course covers module version pinning.

Quiz

Production discipline

• Track the kubelet version on each node. kubectl get nodes -o jsonpath.
• Stay within 3 minor behind. A 4 minor behind kubelet is unsupported.
• Never upgrade the kubelet ahead of the apiserver. The asymmetry is one-way.
• Plan rolling upgrades in waves. Respect the workload PDBs.
• Verify the skew after every upgrade. A pre-flight check.
• Document the skew in the runbook. The kubelet versions, the apiserver version, the policy.

The asymmetry is the foundation of rolling worker upgrades. Operating it well is staying within the policy and planning upgrades around it.