high riskservice affecting~35 min

Runbook: Drain a Production Node Safely

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Confirm a change ticket is open and the change window current
· Confirm cluster capacity can absorb the drained node workloads: kubectl describe nodes | grep -E "Allocatable|Allocated"
· Confirm every PDB allows the drain: kubectl get pdb -A -o jsonpath='{.items[*].status}' | jq
· Confirm the maintenance reason is documented (OS upgrade, kernel patch, hardware swap, decommission)
· Capture the current workload census: kubectl get pods -A -o wide --field-selector spec.nodeName=<node> | tee /tmp/node-pods.txt
· Capture the node role and any taints: kubectl get node <node> -o jsonpath='{.metadata.labels}{"\n"}{.spec.taints}' | jq

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Cordon the node to stop new scheduling: kubectl cordon <node>
2Plan the drain: kubectl drain <node> --dry-run=client -o yaml | tee /tmp/drain.yaml to see what would be evicted
3Drain with appropriate flags: kubectl drain <node> --ignore-daemonsets --delete-emptydir-data --grace-period=<n> --timeout=10m
4If any PDB blocks the drain, stop; do not bypass with --disable-eviction
5Monitor Pod termination: kubectl get pods -A -o wide --field-selector spec.nodeName=<node>
6Verify the node has no workload Pods (DaemonSets are expected to remain): kubectl get pods -A -o wide --field-selector spec.nodeName=<node> | grep -v -E "kube-system|fluentd|cni"
7Perform the node operation (upgrade, hardware swap, etc.)
8After the operation, confirm the node is Ready and the kubelet is healthy
9Uncordon: kubectl uncordon <node>
10Re-admit workloads if required by re-deploying or letting the scheduler place them

4 · Verification

Confirm the procedure actually fixed the problem.

✓kubectl get node <node> reports Ready with all pressures False
✓kubectl get pods -A -o wide --field-selector spec.nodeName=<node> shows only DaemonSet and system Pods
✓kubectl uncordon <node> succeeds and the node becomes schedulable
✓Dashboards show the workloads now running on other nodes with no error spike
✓kubectl get events -A --sort-by=.lastTimestamp | grep -i warning | tail shows no new warnings
✓kubectl get pdb -A -o jsonpath='{.items[*].status}' | jq reports no Disallowed pods from the drain

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶If the drain blocks on a PDB, do not bypass; either delay the operation until capacity allows the PDB to be honoured, or reschedule fewer replicas before draining
↶If a workload cannot reschedule (Pending on every node), the cluster is at capacity; scale down the workload replicas temporarily with explicit approval, drain, then restore
↶If the node operation (e.g. reboot) fails to bring the node back, escalate to the kubernetes-rb-troubleshoot-node-notready runbook; do not panic and force-evict more Pods
↶If the operation corrupts the node and the node must be decommissioned, remove it from the cluster: kubectl delete node <node> and follow kubernetes-rb-recover-failed-worker
↶Capture the failing Pod termination logs before any force-delete

6 · Escalation

When the runbook isn't enough, contact:

· Drain blocks on a PDB: capacity issue, not a manifest issue; escalate to capacity planning
· Pods stuck in Terminating beyond terminationGracePeriodSeconds: finalizer or stuck handler; investigate before force-delete
· Workload cannot reschedule after drain: scheduler, capacity, or PDB issue; see kubernetes-rb-investigate-pending-pod
· Node operation fails to bring the node back: hardware failure; escalate to datacenter ownership
· Cluster-wide Pod churn after uncordon: the node is missing capacity or labels; investigate the node template before admitting workloads

A drain is a deliberate, observable eviction of every workload Pod from a node, honouring PodDisruptionBudgets. It is the right operation when a node needs work; it is the wrong operation when the operator needs the workload somewhere else and the node is fine.

1. Plan the drain

Read-only / SafePlan the drain


kubectl drain <node>  \
--dry-run=client \
--ignore-daemonsets \
--delete-emptydir-data \
--grace-period=30 \
--timeout=10m \
-o yaml | tee /tmp/drain.yaml

# What would actually be evicted?
kubectl drain <node> --dry-run=client --ignore-daemonsets -o json | \
jq -r '.items[] | select(.metadata.namespace != "kube-system") | .metadata.namespace + "/" + .metadata.name'

2. Cordon first

Read-only / SafeCordon first


kubectl cordon <node>
kubectl get node <node> -o jsonpath='{.spec.unschedulable}'
# Expect: true

3. Drain

Read-only / SafeDrain


kubectl drain <node>  \
--ignore-daemonsets \
--delete-emptydir-data \
--grace-period=30 \
--timeout=15m

If the drain returns with Cannot evict Pods ... PDB violation, the capacity or the PDB is wrong. Investigate before retrying.

4. Verify

Read-only / SafeVerify


# Expect: only kube-system and DaemonSet Pods (e.g. CNI, kube-proxy)

kubectl get pods -A -o wide --field-selector spec.nodeName=<node> | grep -v kube-system
# Expect: empty (no application workloads)

5. Perform the node operation

The drain has prepared the node. The actual operation (reboot, hardware swap, OS upgrade, kernel patch) follows.

Read-only / SafePerform the node operation


ssh <node> -- sudo reboot

# B. OS upgrade
ssh <node> -- sudo apt full-upgrade -y
ssh <node> -- sudo systemctl reboot

# C. Hardware swap
# Use BMC/IPMI to power down; physically swap; power on

# After reboot, confirm the node is reachable
ssh <node> -- uptime
ssh <node> -- sudo journalctl -u kubelet --since "2 min ago" --no-pager | tail

6. Uncordon

Read-only / SafeUncordon


kubectl get node <node> -o jsonpath='{.status.conditions}' | jq
# Expect: Ready=True, all pressures False

kubectl uncordon <node>
kubectl get node <node> -o jsonpath='{.spec.unschedulable}'
# Expect: false (or null)

7. Re-admit workloads

Read-only / SafeRe-admit workloads


# Watch the cluster rebalance
kubectl get pods -A -o wide --sort-by=.spec.nodeName | head -20
sleep 300
kubectl get pods -A -o wide --sort-by=.spec.nodeName | head -20

Common pitfalls

Symptom	Cause	Action
Drain blocks on `Cannot evict Pods ... PDB violation`	Capacity issue, PDB doing its job	Scale down or wait; do not bypass
Pods stuck in `Terminating` beyond grace	Finalizer or stuck handler	Investigate; do not force-delete without understanding
Node returns with kubelet auth errors	Clock skew or cert rotation during drain	See `kubernetes-rb-renew-cluster-certs`
Workload cannot reschedule after drain	Cluster at capacity	Scale down before drain or scale up after
DaemonSet Pod evicted by drain	Used `--ignore-daemonsets` but the DaemonSet has a custom toleration	Verify the DaemonSet’s tolerations before draining

A drain is not a node restart. It is a contract with every workload that its replicas will be honoured. The runbook respects the contract by not bypassing PDBs and not force-evicting stuck Pods.