Workers in waves — the PDB-driven scheduling

Workers in waves is the canonical pattern for upgrading workers in stages. The wave size is bounded by the workload’s PDB; the wave order is sequenced by the workload’s availability requirements. This lesson walks the wave plan, the sequencing, and the validation.

The wave plan

flowchart LR
    A[5 workers at v1.33.0] --> B[Wave 1: 1 worker]
    B --> C[Validate]
    C --> D[Wave 2: 1 worker]
    D --> E[Validate]
    E --> F[Wave 3: 1 worker]
    F --> G[Validate]
    G --> H[Wave 4: 1 worker]
    H --> I[Validate]
    I --> J[Wave 5: 1 worker]
    J --> K[Validate]

The waves are sequential; each wave is validated before the next begins.

The PDB inspection

kubectl get pdb -A

NAMESPACE   NAME                MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS
default     nginx-pdb           3               -                  3
default     api-pdb             -               1                  1
default     batch-pdb           -               25%                1
kube-system coredns-pdb         -               1                  1

The ALLOWED DISRUPTIONS column shows the wave size per workload.

The wave size calculation

The wave size is the smallest floor of the workload’s maxUnavailable:

For 5 workers:
  nginx-pdb: maxUnavailable implicit (5 - 3 = 2) → 2
  api-pdb: maxUnavailable: 1 → 1
  batch-pdb: maxUnavailable: 25% → floor(0.25 * 5) = 1
  coredns-pdb: maxUnavailable: 1 → 1

  Wave size: min(2, 1, 1, 1) = 1 worker per wave

The wave size is 1 worker per wave.

The wave order

The wave order is by node name (default) or by custom ordering:

# Alphabetical order
worker-1, worker-2, worker-3, worker-4, worker-5

# Custom order (e.g., starting from the least-loaded)
kubectl get nodes -o json | jq -r '.items[] | "\(.metadata.name) \(.status.allocatable.cpu)"' | sort -k2 -n

The custom order is for advanced use cases (e.g., shifting load gradually). The default order is alphabetical.

The wave execution

# Wave 1
kubectl drain worker-1 --ignore-daemonsets --delete-emptydir-data
ssh worker-1 sudo kubeadm upgrade node
ssh worker-1 sudo systemctl restart kubelet
kubectl uncordon worker-1

# Validate
kubectl get nodes worker-1 -o wide
kubectl get pods -A -o wide | grep worker-1
# Run a smoke test if applicable

# Wave 2
kubectl drain worker-2 --ignore-daemonsets --delete-emptydir-data
ssh worker-2 sudo kubeadm upgrade node
ssh worker-2 sudo systemctl restart kubelet
kubectl uncordon worker-2

# Validate
kubectl get nodes worker-2 -o wide
# ... etc

Each wave is validated before the next begins.

The validation

# Node version
kubectl get nodes -o wide

# Pod health
kubectl get pods -A

# Workload smoke tests
kubectl run nginx-test --image=nginx --rm -it --restart=Never --command -- nginx -v

# Cluster health
kubectl get --raw='/healthz'

The cluster is healthy if all nodes are Ready, the pods are Running, and the smoke tests pass.

The failure mode

A failed wave is the cluster’s stop signal:

# Wave 1 fails (e.g., kubelet does not restart)
ssh worker-1 sudo journalctl -u kubelet -n 100

The wave is rolled back:

# Drain the worker at the new version (if it's broken)
kubectl drain worker-1 --ignore-daemonsets --delete-emptydir-data

# Restore the kubelet binary
ssh worker-1 sudo apt-get install -y kubelet=1.34.0-1.34
ssh worker-1 sudo systemctl restart kubelet

# Uncordon
kubectl uncordon worker-1

The cluster is back to the prior state. The operator investigates the failure before retrying.

The wave plan document

A wave plan is a document that captures the order, the wave size, and the validation:

WORKER UPGRADE WAVE PLAN: v1.34.0 → v1.34.1
================================================
Cluster: 5 workers
PDB constraint: maxUnavailable: 1 (api-pdb)
Wave size: 1 worker per wave
Total waves: 5

Wave 1: worker-1
  - Drain
  - Upgrade kubelet
  - Restart kubelet
  - Uncordon
  - Validate: kubectl get nodes worker-1 shows v1.34.1
  - Validate: smoke test
  - Status: PASSED

Wave 2: worker-2
  ...

The document is the input for the upgrade and the audit trail.

Cross-course references

• The Linux course covers systemd service restart semantics.
• The Helm course covers PDB-aware Helm hooks.
• The Ansible course covers idempotent worker configuration.

Quiz

Production discipline

• Calculate the wave size from the PDBs. The smallest floor of maxUnavailable.
• Sequence the waves. Alphabetical or custom order.
• Validate after each wave. Node version, pod health, smoke tests.
• Document the wave plan. The wave order, the validation, the rollback.
• Investigate failed waves. Don’t skip a failed wave.
• Run staging first. Catch the wave-size issue before production.

The workers in waves is the canonical staged upgrade. Operating it well is calculating the wave size, sequencing the waves, and validating at each step.