Drain before upgrade — the rule of patience

Draining is the rule of patience before any worker upgrade. The drain evicts the pods from the worker, respects the workload’s PDB, and ensures the cluster is healthy during the upgrade. This lesson walks the drain command, the flags, the wave size, and the failure modes.

The drain command

kubectl drain worker-1 \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --force

The flags:

• --ignore-daemonsets: DaemonSet pods are not evicted (they are system pods; they survive on every node).
• --delete-emptydir-data: pods with emptyDir volumes are evicted (their data is lost on the new pod).
• --force: pods that are not managed by a controller (RS, RC, Deployment, etc.) are forcefully deleted.

The drain evicts the pods, waits for the workload to reschedule, and marks the worker as unschedulable.

flowchart LR
    A[Worker Ready] --> B[kubectl drain]
    B --> C[Pods evicted]
    C --> D[Workload reschedules]
    D --> E[Worker empty + cordoned]

The PDB-bound wave size

The wave size is the largest number of workers that can be drained and upgraded simultaneously without violating any workload’s PDB.

For 5 workers with maxUnavailable: 1:
  Wave size: 1 worker per wave
  Total waves: 5

For 5 workers with maxUnavailable: 25%:
  Wave size: floor(0.25 * 5) = 1 worker per wave
  Total waves: 5

For 5 workers with maxUnavailable: 50%:
  Wave size: floor(0.50 * 5) = 2 workers per wave
  Total waves: 3

The wave size is the smallest floor of the workload PDBs.

# Inspect the PDBs
kubectl get pdb -A

NAMESPACE   NAME                MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS
default     nginx-pdb           3               -                  3
default     api-pdb             -               1                  1
default     batch-pdb           -               25%                1

The MAX UNAVAILABLE column shows the wave size per workload.

The drain flow

sequenceDiagram
    participant O as Operator
    participant K as kubectl
    participant AS as API server
    participant W as Worker
    O->>K: kubectl drain worker-1
    K->>AS: list pods on worker-1
    K->>AS: mark worker-1 as unschedulable (cordon)
    loop for each pod
        K->>AS: create eviction
        AS->>AS: check PDB
        AS->>W: terminate pod
        AS->>K: pod terminated
    end
    K-->>O: drain complete

The drain iterates over the pods, evicting each one. The API server checks the PDB before each eviction.

The flags in detail

--ignore-daemonsets

DaemonSet pods are not evicted. They are system pods that run on every node; evicting them would defeat the purpose of the DaemonSet.

--delete-emptydir-data

Pods with emptyDir volumes have their data lost on the new pod. The flag deletes the pod; the new pod starts with an empty emptyDir.

Without the flag, the drain refuses to evict pods with emptyDir volumes (data loss risk).

--force

Pods that are not managed by a controller (e.g., a bare Pod without a ReplicaSet) are not evicted by default. The --force flag evicts them.

Without the flag, the drain refuses to evict unmanaged pods.

--grace-period

The grace period for the pod termination:

kubectl drain worker-1 --grace-period=30

The default is 30 seconds. The pod is evicted; after the grace period, it is forcefully terminated.

--timeout

The drain’s total timeout:

kubectl drain worker-1 --timeout=5m

The default is 5 minutes. The drain fails if the workload does not reschedule within the timeout.

The cordoning

The drain cordons the worker (marks it as unschedulable) before evicting the pods. The cordoning is automatic.

# Manual cordon
kubectl cordon worker-1

# The worker is now unschedulable
kubectl get nodes worker-1
# STATUS: Ready,SchedulingDisabled

After the upgrade, the worker is uncordoned:

kubectl uncordon worker-1

The drain failure

The drain fails if:

• A pod is not managed by a controller (without --force).
• A pod has an emptyDir volume (without --delete-emptydir-data).
• A PDB would be violated.
• A pod does not terminate within the timeout.

The failure stops the drain; the operator must investigate before retrying.

Cross-course references

• The Linux course covers systemd service restart semantics.
• The Observability course covers drain-related alerts.
• The Helm course covers chart hooks during drain.

Quiz

Production discipline

• Drain before upgrade. Always.
• Respect the PDB. The wave size is bounded by the PDB.
• Use —ignore-daemonsets. DaemonSet pods are system pods.
• Use —delete-emptydir-data when appropriate. Acknowledge the data loss.
• Do not use —force unless necessary. Force is a known anti-pattern.
• Document the drain procedure. The flag, the wave size, the PDB constraint.

The drain is the rule of patience. Operating it well is respecting the PDB and following the procedure.