Cordon and drain orchestration — the eviction choreography

Cordon stops new scheduling on a node. Drain evicts workloads with PDB respect. The orchestration is the sequence of cordons and drains across a failure domain, watched by an owner with a pager. A drain that is not choreographed is a drain that will discover the workload’s PDB only after the eviction has started.

The cordon / drain lifecycle

A node retirement is a four-step state machine:

1. Cordon. The node’s spec.unschedulable is set to true. The scheduler no longer places new Pods on the node. Existing Pods continue to run.
2. Drain. The node is evicted. Each Pod is removed via the eviction API. The scheduler reschedules the workload’s replicas onto other nodes, respecting the workload’s PodDisruptionBudget.
3. Maintenance. The node is patched, rebooted, or replaced. The operator does whatever the window requires.
4. Uncordon. The node’s spec.unschedulable is set to false. The scheduler resumes placing workloads on the node.

stateDiagram-v2
    [*] --> Ready
    Ready --> Cordoned: kubectl cordon
    Cordoned --> Draining: kubectl drain
    Draining --> Ready: kubectl uncordon
    Draining --> Maintenance: node patched
    Maintenance --> Ready: kubectl uncordon
    Draining --> Stalled: PDB / force
    Stalled --> Draining: force deleted

The states are observable in kubectl get nodes -o jsonpath='{.items[*].spec.unschedulable}' and in the node conditions.

The eviction API

The drain is not a delete; it is an eviction. The eviction is a POST to the eviction subresource that the API server intercepts and treats as a deletion wrapped in a PDB check.

kubectl drain node-03 --ignore-daemonsets --delete-emptydir-data

The flags tell the drain how to handle the corner cases:

• --ignore-daemonsets skips DaemonSet pods. A drain that does not ignore DaemonSets will fail to evict the kube-proxy and cni DaemonSets, which are intentional.
• --delete-emptydir-data recognises that an emptyDir is destructive to wipe. Without the flag, the drain blocks on emptyDir volumes because they are not safe to delete.
• --force is the escape hatch. The drain forces eviction even if the PDB does not permit it. The use of --force is audited; it is the line that says “I have decided the PDB does not apply.”

Why the drain is choreographed

A drain that is not choreographed across a failure domain will surprise the cluster. If the operator drains all ten nodes in domain A simultaneously, the workload’s PDB is satisfied by the nodes in domain B and C, but the cluster’s capacity may be insufficient. The orchestration is:

1. Pick a failure domain. A single domain or rack.
2. Drain one node at a time, with a pause. The pause lets the rescheduled replicas settle and the cluster’s capacity-balancing controllers move workloads.
3. Verify after each drain. Confirm the node is empty, the rescheduled replicas are Ready, and the PDB is satisfied.
4. Move to the next node. Repeat.

flowchart LR
    A[Domain A] --> B[Node 1]
    B --> C[Drain]
    C --> D{Verify}
    D -->|Pass| E[Node 2]
    D -->|Fail| F[Abort]
    E --> G[Drain]
    G --> H{Verify}
    H -->|Pass| I[Node 3]
    H -->|Fail| F
    I --> J[Drain]
    J --> K{Verify}
    K -->|Pass| L[Domain done]
    K -->|Fail| F

The “fail” branch is the abort path. The drain is stopped on the first failure that the operator cannot explain. The window is closed; the cluster is rolled back; the failure is investigated.

PDB interaction

A PodDisruptionBudget is the workload’s contract with the drain. The PDB says “at most N unavailable” or “at least N available.” The eviction API checks the PDB before allowing the eviction.

An eviction that would violate the PDB is blocked. The drain stalls. The operator sees the stall in the kubectl drain output and in the workload’s events:

Warning: evicting pod billing-7d8f-abcde
Warning: cannot evict pod as it would violate the workload's
PodDisruptionBudget

The remediation is not --force. The remediation is to either wait until the workload has fewer unavailable replicas than the PDB permits, or to scale the workload up so the drain fits inside the PDB.

Automated drain orchestrators

Production clusters rarely run kubectl drain by hand. The orchestrator is one of:

• Cluster-API. Each Machine is owned by a MachineDeployment; the controller rolls the new image in by replacing the machine. The drain is internal to the controller.
• Karpenter. A node is consolidated when a cheaper configuration is available. The consolidation includes a drain with PDB respect; the workload is rescheduled before the node is terminated.
• Custom drain controllers. A workflow that owns a queue of node retirements, drains them in order, and watches the cluster’s PDB compliance.

Each orchestrator has the same failure modes: a stuck drain that does not respect the PDB, a --force that bypasses the audit, a race between two orchestrators on the same node.

Production discipline

The drain is the cluster’s most dangerous maintenance operation. The discipline is to choreograph it across failure domains, respect the PDB, verify after each step, and never --force an eviction that contradicts the workload’s contract. The window’s success is the workload’s resilience.

• Cordon before drain. The cordon prevents new Pods from being scheduled onto the node while the drain is in progress.
• Drain one node at a time, in a single failure domain. The drain’s blast radius is contained by the choreography.
• Wait for the rescheduled replicas to be Ready before moving to the next node. The verification is the evidence that the workload has absorbed the drain.
• Never --force an eviction without a written reason. The flag is the audit trail that says “the operator has decided the PDB does not apply.”

Quiz