Controller-manager and scheduler — the symmetric skew

kube-controller-manager and kube-scheduler are the control plane’s other two stateless components. The skew policy treats them symmetrically: within 1 minor ahead or behind the apiserver. This lesson walks the symmetric skew, the rationale, and the upgrade implications.

The symmetric skew

The skew policy:

kube-apiserver: 1.34.0
kube-controller-manager: 1.33.x, 1.34.x, 1.35.x → all OK
kube-scheduler: 1.33.x, 1.34.x, 1.35.x → all OK

Both components can be 1 minor ahead or behind the apiserver. The skew is symmetric.

flowchart LR
    A[apiserver 1.34] --> B[controller-manager 1.33]
    A --> C[controller-manager 1.34]
    A --> D[controller-manager 1.35]
    A --> E[scheduler 1.33]
    A --> F[scheduler 1.34]
    A --> G[scheduler 1.35]

Why the symmetric skew

The reason for the symmetric skew is that these components read the apiserver’s API and write back to etcd via the apiserver:

controller-manager → apiserver (GET state)
                    → apiserver (PATCH state)
apiserver → etcd (write)

A controller-manager at 1.35 calling an apiserver at 1.34 is a forward-compatible call (the controller-manager uses APIs that the 1.34 apiserver supports, because the project keeps API versions available for 1 minor back).

A controller-manager at 1.33 calling an apiserver at 1.34 is a backward-compatible call (the apiserver still supports the 1.33 API versions).

Both directions are within the project’s compatibility guarantee.

The components in the upgrade

In a kubeadm upgrade, the controller-manager and scheduler are updated together with the apiserver:

sudo kubeadm upgrade apply v1.34.1

The command updates:

• kube-apiserver (manifest)
• kube-controller-manager (manifest)
• kube-scheduler (manifest)
• etcd (manifest, if --etcd-upgrade=true)

The static pod manifests are rewritten; the kubelet restarts the corresponding static pods.

sequenceDiagram
    participant K as kubeadm
    participant FS as /etc/kubernetes/manifests/
    participant KL as kubelet
    participant AS as API server
    participant CM as controller-manager
    participant SC as scheduler
    K->>FS: write all three manifests at v1.34.1
    K->>KL: notify via file watch
    KL->>AS: restart pod
    KL->>CM: restart pod
    KL->>SC: restart pod

The three components are restarted in sequence. The overall restart is brief (seconds); the LB routes around.

The validation

# Get the controller-manager version
kubectl -n kube-system get pod -l component=kube-controller-manager -o jsonpath='{.items[0].spec.containers[0].image}'
# registry.k8s.io/kube-controller-manager:v1.34.1

# Get the scheduler version
kubectl -n kube-system get pod -l component=kube-scheduler -o jsonpath='{.items[0].spec.containers[0].image}'
# registry.k8s.io/kube-scheduler:v1.34.1

The images confirm the versions.

The skew during a multi-minor upgrade

During a multi-minor upgrade (1.31 → 1.34), the upgrade sequence is:

1.31 → 1.32: apiserver, controller-manager, scheduler all at 1.32
1.32 → 1.33: apiserver, controller-manager, scheduler all at 1.33
1.33 → 1.34: apiserver, controller-manager, scheduler all at 1.34

At each step, the three components are upgraded together and the skew is maintained.

The failure mode

A controller-manager or scheduler at 2+ minor away from the apiserver is unsupported. The component may fail to authenticate, may use API versions that are removed, or may produce incorrect behavior.

INVALID for 1.34.0 apiserver:
  controller-manager: 1.32.x → 2 minor behind, INVALID
  scheduler: 1.36.x → 2 minor ahead, INVALID

The recovery is to upgrade the offending component to within 1 minor of the apiserver.

The kubeadm skew enforcement

kubeadm upgrade apply enforces the skew policy. The command refuses to upgrade if the target version would violate the skew against the existing components:

sudo kubeadm upgrade apply v1.34.1

[upgrade/versions] Cluster version: v1.33.0
[upgrade/versions] Target version: v1.34.1
[upgrade/versions] Kubelet version: v1.33.0
[upgrade/versions] Control plane version: v1.33.0
[upgrade/versions] Latest stable version: v1.34.1

The output confirms the cluster version, the target version, and the kubelet version. The upgrade proceeds because the target is within skew.

Cross-course references

• The Linux course covers tight cluster upgrade semantics.
• The Terraform course covers Kubernetes provider version pinning.
• The Ansible course covers module version pinning.

Quiz

Production discipline

• Keep the controller-manager and scheduler within 1 minor. The skew policy is symmetric.
• Upgrade the three control plane components together. kubeadm upgrade apply does this.
• Verify the skew after every upgrade. Inspect the static pod images.
• Investigate skew violations. A controller-manager at 2+ minor behind is a sign of an aborted upgrade.
• Document the skew in the runbook. The components, the versions, the policy.

The symmetric skew is the contract for the control plane’s stateless components. Operating it well is keeping them within 1 minor and upgrading them together.