Post-upgrade validation — confirming the cluster is healthy

The post-upgrade validation is the cluster’s acceptance test. The upgrade is not complete until the validation passes. This lesson walks the validation steps and the post-upgrade snapshot.

The validation order

flowchart LR
    A[Node health] --> B[Pod health]
    B --> C[API server health]
    C --> D[etcd health]
    D --> E[Workload smoke tests]
    E --> F[Post-upgrade snapshot]
    F --> G[Document]

Each step has a defined output. If any step fails, the upgrade is not complete; the operator investigates.

Step 1: Node health

kubectl get nodes -o wide

NAME       STATUS   ROLES           AGE   VERSION   INTERNAL-IP   ...
cp-1       Ready    control-plane   30d   v1.34.1   10.0.1.10
cp-2       Ready    control-plane   30d   v1.34.1   10.0.1.11
cp-3       Ready    control-plane   30d   v1.34.1   10.0.1.12
worker-1   Ready    <none>          30d   v1.34.1   10.0.1.20
worker-2   Ready    <none>          30d   v1.34.1   10.0.1.21
worker-3   Ready    <none>          30d   v1.34.1   10.0.1.22
worker-4   Ready    <none>          30d   v1.34.1   10.0.1.23
worker-5   Ready    <none>          30d   v1.34.1   10.0.1.24

All nodes Ready at v1.34.1. No NotReady nodes.

kubectl get nodes -o json | jq -r '.items[] | .status.conditions[] | select(.type != "Ready") | "\(.type) on \(.status) for node \(.nodeName)"'

The output should be empty. Any non-Ready condition is a signal.

Step 2: Pod health

kubectl get pods -A

NAMESPACE     NAME                              READY   STATUS    RESTARTS   AGE
kube-system   coredns-xyz                       1/1     Running   0          1m
kube-system   kube-apiserver-cp-1               1/1     Running   0          1m
kube-system   kube-controller-manager-cp-1      1/1     Running   0          1m
kube-system   kube-proxy-xyz                    1/1     Running   0          1m
kube-system   kube-scheduler-cp-1               1/1     Running   0          1m
kube-system   etcd-cp-1                         1/1     Running   0          1m
default       nginx-test                        1/1     Running   0          2m

All pods Running. No CrashLoopBackOff, no ImagePullBackOff, no Pending.

kubectl get pods -A -o json | jq -r '.items[] | select(.status.phase != "Running") | "\(.metadata.namespace)/\(.metadata.name) is \(.status.phase)"'

The output should be empty.

Step 3: API server health

kubectl get --raw='/healthz'

ok

The API server is responsive.

kubectl get --raw='/readyz'

{"health":"ok"}

The API server is ready.

kubectl api-versions

The list of API versions should match the expected set for v1.34.1.

Step 4: etcd health

sudo ETCDCTL_API=3 etcdctl \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
  --key=/etc/kubernetes/pki/etcd/healthcheck-client.key \
  --endpoints=https://127.0.0.1:2379 \
  endpoint health

https://127.0.0.1:2379 is healthy: successfully committed proposal: took = 2ms

The etcd member is healthy.

sudo ETCDCTL_API=3 etcdctl \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
  --key=/etc/kubernetes/pki/etcd/healthcheck-client.key \
  --endpoints=https://127.0.0.1:2379 \
  member list

All etcd members are listed.

Step 5: Workload smoke tests

Run a workload that exercises the cluster’s core paths:

# Test pod scheduling
kubectl run nginx-test --image=nginx --rm -it --restart=Never --command -- nginx -v

# Test DNS
kubectl run dns-test --image=busybox:1.36 --rm -it --restart=Never -- nslookup kubernetes.default

# Test Service
kubectl run nginx-test-svc --image=nginx --port=80
kubectl expose pod nginx-test-svc --port=80
kubectl run curl-test --image=curlimages/curl --rm -it --restart=Never -- curl -s http://nginx-test-svc

# Test Ingress (if applicable)
kubectl apply -f ingress-test.yaml
curl -H "Host: test.example.com" http://ingress-controller/

# Test RBAC
kubectl auth can-i get pods --as=system:serviceaccount:default:default

Each smoke test confirms a different cluster path: scheduling, DNS, Services, Ingress, RBAC.

Step 6: Post-upgrade snapshot

After the validation passes:

sudo ETCDCTL_API=3 etcdctl \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/healthcheck-client.crt \
  --key=/etc/kubernetes/pki/etcd/healthcheck-client.key \
  --endpoints=https://127.0.0.1:2379 \
  snapshot save /var/backups/etcd-post-upgrade-$(date +%Y%m%d-%H%M%S).db

sudo aws s3 cp /var/backups/etcd-post-upgrade-*.db s3://k8s-backups/etcd/

The post-upgrade snapshot is the new baseline. The pre-upgrade snapshot is the rollback path.

Step 7: Document

The upgrade is documented:

UPGRADE RESULT: 1.34.0 → 1.34.1
================================
Date: 2026-08-16
Operators: <name>, <backup>

Timeline:
  - 09:00 - Release notes read
  - 09:30 - Pre-upgrade snapshot
  - 10:00 - Control plane upgrade (cp-2, cp-3, cp-1)
  - 11:00 - Workers in waves
  - 12:00 - CNI upgrade
  - 12:30 - CoreDNS upgrade
  - 13:00 - kube-proxy upgrade
  - 13:30 - Ingress upgrade
  - 14:00 - Validation
  - 14:30 - Post-upgrade snapshot

Issues encountered:
  - None

Next steps:
  - Watch node conditions for 24 hours
  - Calendar the next upgrade (1.34.1 → 1.34.2)

The document is the input for the next upgrade.

Cross-course references

• The Observability course covers alerting during upgrade windows.
• The Linux course covers service restart semantics.
• The Ansible course covers automated upgrade validation.

Quiz

Production discipline

• Validate at each step. Don’t proceed on partial validation.
• Watch node conditions for 24 hours. Some issues surface late.
• Run workload smoke tests. Scheduling, DNS, Services, Ingress, RBAC.
• Take a post-upgrade snapshot. The new baseline.
• Retain the pre-upgrade snapshot for 30 days. The rollback window.
• Document the upgrade. Timeline, issues, next steps.

The post-upgrade validation is the cluster’s acceptance test. Operating it well is validating each step and documenting the outcome.