kubectl drain — the eviction-based maintenance tool

kubectl drain is the cluster’s standard tool for node maintenance. The drain cordons the node and evicts the Pods. The flags control the eviction behaviour. This lesson walks the drain flow, the flags, and the operational patterns.

The drain operation

# Substitute your own node name:
NODE=worker-03

kubectl drain "$NODE"

The drain’s flow:

sequenceDiagram
    autonumber
    participant O as Operator
    participant API as API server
    participant K as kubelet

    O->>API: cordon (set unschedulable=true)
    API->>API: set taint
    O->>API: list Pods on node
    API-->>O: Pod list
    loop For each Pod
        O->>API: evict Pod
        API->>API: check PDB
        API->>K: send SIGTERM
        K->>K: wait for grace period
        K->>K: send SIGKILL
        K->>API: update Pod status: Failed
    end
    O->>API: verify eviction
    O->>Operator: drain complete

The drain is a two-step operation: cordon, then evict. The cordon is the soft gate; the eviction is the destructive step.

The drain’s eviction is the API server’s eviction API; the kubelet sends SIGTERM, waits for the grace period, and sends SIGKILL.

The drain’s flags

The drain’s flags:

The flags are the operator’s tuning for the drain. The default is to ignore DaemonSets (the DaemonSet’s Pods are not evicted) and to delete emptyDir data.

The --force flag is the destructive option. The flag forces the eviction of Pods that are not managed by a controller (bare Pods, static Pods). The Pod’s controllers are not running; the Pod is lost.

The drain’s behaviour with DaemonSets

The drain’s default behaviour is to reject the drain if the node has DaemonSet Pods. The drain refuses to evict the DaemonSet Pods because the Pods will be re-created on the node when the node is back.

The --ignore-daemonsets flag allows the drain to continue. The DaemonSet Pods are not evicted; the DaemonSet’s Pods are left on the node.

The drain’s behaviour with DaemonSets:

# Substitute your own node name:
NODE=worker-03

kubectl drain "$NODE" --ignore-daemonsets

The drain cordons the node and evicts the Pods that are not DaemonSet Pods. The DaemonSet Pods are left on the node.

The drain’s behaviour with emptyDir

The drain’s default behaviour is to reject the drain if the Pod has an emptyDir volume. The emptyDir volume is deleted when the Pod is evicted; the Pod’s data is lost.

The --delete-emptydir-data flag allows the drain to continue. The emptyDir volume is deleted.

The drain’s behaviour with emptyDir:

# Substitute your own node name:
NODE=worker-03

kubectl drain "$NODE" --delete-emptydir-data

The drain cordons the node and evicts the Pods. The emptyDir volumes are deleted.

The drain’s behaviour with PodDisruptionBudgets

The drain’s eviction API checks the PodDisruptionBudget. The eviction is rejected if the PDB rejects the eviction.

The drain’s behaviour with a restrictive PDB:

# Substitute your own node name:
NODE=worker-03

kubectl drain "$NODE"

error when evicting pod "billing-1": Cannot evict pod as it would violate the pod's disruption budget.

The drain fails. The fix is to:

• Wait for the PDB to allow the eviction.
• Adjust the PDB to allow the eviction.
• Use --force to bypass the PDB (not recommended).

The drain’s interaction with the PDB is the cluster’s mechanism for protecting workloads from disruption. The drain respects the PDB; the PDB must be adjusted to allow the drain.

The drain’s behaviour with priority classes

The drain’s eviction API checks the Pod’s priority class. The Pod is evicted based on the priority class and the QoS class.

The drain’s behaviour with a high-priority Pod:

# Substitute your own node name:
NODE=worker-03

kubectl drain "$NODE"

The high-priority Pod is evicted last. The drain evicts the low-priority Pods first.

The drain’s behaviour with the grace period

The drain’s default grace period is the Pod’s terminationGracePeriodSeconds (default 30s). The grace period is the time the kubelet waits for the Pod’s containers to terminate.

The --grace-period flag overrides the grace period:

# Substitute your own node name:
NODE=worker-03

kubectl drain "$NODE" --grace-period=10

The grace period is the operator’s tuning for the drain. A short grace period is faster; a long grace period is more graceful.

The drain’s verification

The drain’s verification:

# Substitute your own node name:
NODE=worker-03

kubectl get pods -o wide | grep "$NODE"

The output should be empty. The drain is complete when the node has no Pods.

The drain’s verification with DaemonSets:

# Substitute your own node name:
NODE=worker-03

kubectl get pods -o wide -l k8s-app=fluent-bit | grep "$NODE"

The output should show the DaemonSet Pods still Running. Their controller tolerates the node’s unschedulable taint, so the cordon does not touch them.

The drain’s operational patterns

The drain’s operational patterns:

• Drain during maintenance windows. The drain is disruptive; the operator should schedule the drain during a maintenance window.
• Drain before node replacement. The drain evicts the Pods; the node can be replaced.
• Drain with --ignore-daemonsets. The DaemonSet Pods are left on the node; the drain is faster.
• Drain with --delete-emptydir-data. The emptyDir volumes are deleted; the drain is faster.
• Drain with --grace-period. The grace period is the time the kubelet waits for the Pod’s containers to terminate.

The drain’s failure modes

The drain’s failure modes:

The diagnostic:

# The Pod named in the drain's error message:
POD=billing-1

kubectl describe pod "$POD" | grep -A 5 "Events"

The events show the drain’s failure. The fix is to investigate the events and the kubelet’s logs.

Quiz

Production discipline

• Drain is the cluster’s standard tool. The drain cordons the node and evicts the Pods.
• Drain with --ignore-daemonsets. The DaemonSet Pods are left on the node; the drain is faster.
• Drain with --delete-emptydir-data. The emptyDir volumes are deleted; the drain is faster.
• Drain with --force is destructive. The Pod’s controllers are not running; the Pod is lost.
• Respect the PodDisruptionBudget. The drain is rejected if the PDB rejects the eviction. The fix is to adjust the PDB.
• Verify the drain. The drain is complete when the node has no Pods.
• Audit the drain at every node repave. A new node that joins the cluster with the wrong drain is a node that is failing silently. The audit catches the failure.
• Test the drain in non-production. A staging cluster that mirrors production is the right place to test the drain.