Skew validation tooling — verifying the cluster is in policy

Validating the skew is the pre-flight check before any upgrade. The validation inspects each component’s version, compares it against the policy, and flags violations. This lesson walks the inspection tools and a pre-flight script that automates the validation.

The kubectl version command

kubectl version

Client Version: v1.34.1
Kustomize Version: v5.7.1
Server Version: v1.34.1

The client and server versions are reported. The output confirms the kubectl and apiserver versions. This short form used to need --short; that flag was deprecated in 1.24 and removed in 1.28, and the short form is now what kubectl version prints on its own.

kubectl version -o yaml

The structured output includes the git version, the platform, the build date, and the go version for both client and server. -o json gives the same fields, which is the form to parse in a pre-flight script.

The kubectl get nodes command

kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.nodeInfo.kubeletVersion}{"\n"}{end}'

cp-1       v1.34.1
cp-2       v1.34.1
cp-3       v1.34.1
worker-1   v1.34.1
worker-2   v1.34.1
worker-3   v1.34.1
worker-4   v1.34.1
worker-5   v1.34.1

The kubelet versions on each node are reported.

The kubeadm upgrade plan

sudo kubeadm upgrade plan

[upgrade/config] Reading configuration from the cluster...
[upgrade/config] FYI: You can look at this config file with:
[upgrade/config] $ kubectl -n kube-system get cm kubeadm-config -o yaml

[preflight] Running preflight checks
[preflight] Pulling images required for this upgrade
[preflight] This might take a minute or two depending on the speed of your internet connection.

[upgrade] Running cluster health checks

Components that will be upgraded after you run 'kubeadm upgrade apply':
COMPONENT   CURRENT    TARGET
kubelet     7 x v1.34.0   v1.34.1
kube-apiserver, kube-controller-manager, kube-scheduler, kube-proxy   1.34.0   v1.34.1

Upgrade to version 1.34.1 is available; applied to:
  - kubeadm
  - kubelet
  - control plane

The output shows the current and target versions of the components. The upgrade plan is the validation: the target version is within the skew policy.

The kubeadm upgrade plan for multi-minor

A multi-minor upgrade:

sudo kubeadm upgrade plan v1.34.1

[preflight] Running preflight checks
[upgrade] Running cluster health checks

External components that should be upgraded manually before you upgrade the control plane:
  - etcd (3.5.x → 3.5.x; no upgrade required)

Components that must be upgraded first:
  - N/A

Components that will be upgraded after you run 'kubeadm upgrade apply':
COMPONENT   CURRENT    TARGET
kubelet     7 x v1.31.0   v1.34.1

The kubelet line shows the current version (1.31.0) and the target version (1.34.1). The upgrade validation:

WARNING: kubelet version difference between nodes:
  worker-1: v1.31.0
  worker-2: v1.31.0
  ...
  worker-5: v1.34.0

The warning is informational; the upgrade is allowed.

The control plane components

kubectl -n kube-system get pod -l component=kube-apiserver -o jsonpath='{.items[0].spec.containers[0].image}'
# registry.k8s.io/kube-apiserver:v1.34.1

kubectl -n kube-system get pod -l component=kube-controller-manager -o jsonpath='{.items[0].spec.containers[0].image}'
# registry.k8s.io/kube-controller-manager:v1.34.1

kubectl -n kube-system get pod -l component=kube-scheduler -o jsonpath='{.items[0].spec.containers[0].image}'
# registry.k8s.io/kube-scheduler:v1.34.1

The images confirm the versions.

The etcd version

sudo ETCDCTL_API=3 etcdctl version

etcd Version: 3.5.x
Git SHA: 1a2b3c4d
Go Version: go1.22

The etcd version is reported.

The pre-flight script

A pre-flight script validates the entire cluster skew:

#!/bin/bash
# /usr/local/bin/k8s-skew-validate.sh
# Validates the cluster's version skew against the policy.

set -e

# Get the apiserver version
APISERVER_VERSION=$(kubectl version -o json | jq -r '.serverVersion.minor')
echo "Apiserver minor: $APISERVER_VERSION"

# Get the kubelet versions
echo "Kubelet versions:"
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.nodeInfo.kubeletVersion}{"\n"}{end}'

# Get the controller-manager version
CM_VERSION=$(kubectl -n kube-system get pod -l component=kube-controller-manager -o jsonpath='{.items[0].spec.containers[0].image}' | grep -oP 'v\d+\.\d+')
echo "Controller-manager minor: $CM_VERSION"

# Get the scheduler version
SCHED_VERSION=$(kubectl -n kube-system get pod -l component=kube-scheduler -o jsonpath='{.items[0].spec.containers[0].image}' | grep -oP 'v\d+\.\d+')
echo "Scheduler minor: $SCHED_VERSION"

# Get the etcd version
ETCD_VERSION=$(sudo ETCDCTL_API=3 etcdctl version | grep '^etcd Version' | awk '{print $3}')
echo "Etcd version: $ETCD_VERSION"

# Validate the skew
echo "=== Skew validation ==="
# (Production logic: compare each component's minor against the apiserver's minor,
#  flag violations of the policy)

The output is a complete report of the cluster’s version state.

The validation rules

The script encodes the policy:

The skew is the difference between the component’s minor and the apiserver’s minor.

flowchart LR
    A[Apiserver: 1.34] --> B[Kubelet: 1.31-1.34 OK]
    A --> C[Controller-manager: 1.33-1.35 OK]
    A --> D[Scheduler: 1.33-1.35 OK]
    A --> E[Etcd: 3.5.x OK]

The CI/CD integration

The pre-flight script is run in CI/CD before any upgrade:

# .github/workflows/upgrade-cluster.yml
- name: Validate skew
  run: /usr/local/bin/k8s-skew-validate.sh

The script’s exit code is non-zero if the skew is violated; the CI/CD pipeline fails.

Cross-course references

• The Terraform course covers Kubernetes provider version pinning.
• The Ansible course covers module version pinning.
• The Linux course covers binary version management.

Quiz

Production discipline

• Run the pre-flight script before every upgrade. A cluster with skew violations is not supported.
• Automate the validation in CI/CD. The pipeline fails on skew violation.
• Document the skew in the runbook. The component versions, the policy, the upgrade plan.
• Use kubeadm upgrade plan as the validation. The plan reports the skew state.
• Update the validation script with the policy. The policy is the source of truth.

The skew validation is the cluster’s pre-flight check. Operating it well is automating the validation and acting on the violations.