Worker validation after upgrade — confirming the cluster is healthy

The worker validation is the cluster’s go/no-go signal for the next wave. The validation confirms the worker is at the new version, the pods are running, the workload is healthy, and the cluster is functional. This lesson walks the validation steps and the document.

The validation order

flowchart LR
    A[Worker upgrade complete] --> B[kubectl get nodes]
    B --> C[kubectl get pods]
    C --> D[Smoke tests]
    D --> E[Cluster health]
    E --> F[Document]

Each step has a defined output. If any step fails, the wave is not validated; the operator investigates.

Step 1: Node version

kubectl get nodes worker-1 -o wide

NAME       STATUS   ROLES    AGE   VERSION   INTERNAL-IP
worker-1   Ready    <none>   30d   v1.34.1   10.0.1.20

The worker is at v1.34.1. The status is Ready.

kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.nodeInfo.kubeletVersion}{"\n"}{end}'

cp-1       v1.34.1
cp-2       v1.34.1
cp-3       v1.34.1
worker-1   v1.34.1   # the upgraded worker
worker-2   v1.34.0   # not yet upgraded
worker-3   v1.34.0
worker-4   v1.34.0
worker-5   v1.34.0

The output confirms the worker is at v1.34.1; the others are still at v1.34.0 (in skew).

Step 2: Pod health

kubectl get pods -A -o wide | grep -E 'worker-1|^NAMESPACE'

NAMESPACE     NAME                              READY   STATUS    NODE
kube-system   kube-proxy-xyz                    1/1     Running   worker-1
kube-system   cilium-xyz                        1/1     Running   worker-1
default       nginx-1-abc                       1/1     Running   worker-1
default       nginx-2-def                       1/1     Running   worker-1

The pods are running on the worker. The DaemonSet pods (kube-proxy, cilium) are scheduled on the worker.

kubectl get pods -A -o json | jq -r '.items[] | select(.status.phase != "Running") | "\(.metadata.namespace)/\(.metadata.name) is \(.status.phase)"'

The output should be empty. No pods are in a non-Running phase.

Step 3: Workload smoke tests

# Test pod scheduling
kubectl run nginx-test --image=nginx --rm -it --restart=Never --command -- nginx -v

# Test DNS
kubectl run dns-test --image=busybox:1.36 --rm -it --restart=Never -- nslookup kubernetes.default

# Test Service
kubectl run nginx-test-svc --image=nginx --port=80
kubectl expose pod nginx-test-svc --port=80
kubectl run curl-test --image=curlimages/curl --rm -it --restart=Never -- curl -s http://nginx-test-svc

Each smoke test confirms a different cluster path.

Step 4: Cluster health

kubectl get --raw='/healthz'

ok

The API server is responsive.

kubectl get --raw='/readyz'

{"health":"ok"}

The API server is ready.

kubectl api-versions

The list of API versions matches the expected set for v1.34.1.

Step 5: Watch the node conditions

kubectl get nodes worker-1 -o json | jq '.status.conditions'

[
  {
    "type": "Ready",
    "status": "True",
    "lastTransitionTime": "2026-08-16T10:00:00Z"
  }
]

The Ready condition is True. Watch for 24 hours after the upgrade for any latent issues.

Step 6: Document the upgrade

WORKER UPGRADE RESULT: v1.34.0 → v1.34.1 (worker-1)
================================================
Date: 2026-08-16
Operator: <name>

Worker version: v1.34.1
Pods: All Running
Smoke tests: PASSED
Cluster health: OK

Issues:
  - None

Next steps:
  - Watch worker-1 conditions for 24 hours
  - Proceed to wave 2 (worker-2)

The document is the audit trail.

The crictl inspection

For deeper validation:

# On the worker
sudo crictl ps

The crictl output lists the running containers on the worker. All expected containers are running.

# Take a container ID from the `crictl ps` output above:
CONTAINER_ID=9c2f4b8e1a7d3

sudo crictl logs "$CONTAINER_ID"

The container logs are inspected for errors.

Cross-course references

• The Linux course covers systemd service restart semantics.
• The Observability course covers node metrics.
• The Ansible course covers automated validation.

Quiz

Production discipline

• Validate after each wave. The validation is the go/no-go signal.
• Watch for 24 hours. Latent issues surface late.
• Document the upgrade. The audit trail and the input for the next.
• Run smoke tests. Scheduling, DNS, Services, Ingress, RBAC.
• Investigate failed validations. Don’t skip a failed validation.
• Run staging first. Catch the validation issue before production.

The worker validation is the cluster’s acceptance test. Operating it well is validating each wave and documenting the outcome.