Cordon and uncordon — the soft scheduling gate

kubectl cordon and kubectl uncordon are the cluster’s soft scheduling gate. Cordon sets the node’s unschedulable flag and adds the built-in node.kubernetes.io/unschedulable:NoSchedule taint; uncordon clears the flag and removes the taint. New Pods are blocked; existing Pods continue running. This lesson walks the operations, the difference between the flag and the taint, and the operational patterns.

The cordon operation

sequenceDiagram
    autonumber
    participant O as Operator
    participant API as API server
    participant S as Scheduler
    O->>API: kubectl cordon <node>
    API->>API: spec.unschedulable = true
    API->>API: add unschedulable:NoSchedule taint
    S->>API: watch node
    Note over S: new Pods skip the node
    O->>API: kubectl uncordon <node>
    API->>API: spec.unschedulable = false
    API->>API: remove taint
    Note over S: new Pods can be scheduled

The cordon’s effect on the Node object:

# Substitute your own value before running:
NODE=node-1

kubectl cordon "$NODE"

node/node-1 cordoned

The cordon’s effect on the Node object:

# Substitute your own value before running:
NODE=node-1

kubectl get node "$NODE" -o jsonpath='{.spec}' | jq

{
  "podCIDR": "10.244.1.0/24",
  "podCIDRs": ["10.244.1.0/24"],
  "unschedulable": true,
  "taints": [
    {
      "key": "node.kubernetes.io/unschedulable",
      "effect": "NoSchedule"
    }
  ]
}

The cordon sets two things:

1. spec.unschedulable: true — the node’s unschedulable flag.
2. spec.taints[] — the built-in node.kubernetes.io/unschedulable:NoSchedule taint.

The two are equivalent for the scheduler: both cause the scheduler to reject the node for new Pods. The scheduler checks both; a node that is unschedulable is not a candidate for new Pods.

The unschedulable flag

The spec.unschedulable field is a boolean. The field is true when the node is cordoned; the field is false when the node is uncordoned.

The scheduler reads the field:

if node.Spec.Unschedulable {
  return nil, fmt.Errorf("node is unschedulable")
}

The scheduler skips the node for new Pods. The scheduler does not consider the node for any Pod.

The flag is the cluster’s record of the cordon’s intent. The flag is the source of truth for the scheduler.

The unschedulable taint

The node.kubernetes.io/unschedulable:NoSchedule taint is the cluster’s built-in taint. The taint is added when the node is cordoned; the taint is removed when the node is uncordoned.

The taint is equivalent to the spec.unschedulable flag. The scheduler checks both; a node that has either is unschedulable.

The taint is the cluster’s record of the cordon’s intent. The taint is the source of truth for the TaintToleration filter plugin.

The uncordon operation

# Substitute your own value before running:
NODE=node-1

kubectl uncordon "$NODE"

node/node-1 uncordoned

The uncordon’s effect on the Node object:

# Substitute your own value before running:
NODE=node-1

kubectl get node "$NODE" -o jsonpath='{.spec}' | jq

{
  "podCIDR": "10.244.1.0/24",
  "podCIDRs": ["10.244.1.0/24"],
  "unschedulable": false,
  "taints": []
}

The uncordon clears the spec.unschedulable flag and removes the node.kubernetes.io/unschedulable taint. The node is again a candidate for new Pods.

The uncordon is a single PATCH on the Node object. The PATCH sets spec.unschedulable: false and removes the taint.

The cordon’s effects

The cordon’s effects:

• New Pods: blocked. The scheduler rejects the node for any Pod that does not tolerate the node.kubernetes.io/unschedulable taint.
• Existing Pods: continue running. The kubelet does not evict the Pods.
• API server: the Node object is updated.
• Events: the cluster’s events record the cordon’s intent.

The cordon’s effect is soft. The node is not a candidate for new Pods, but the existing Pods are unaffected. A cordoned node keeps serving traffic, keeps its Service endpoints, and keeps its DaemonSet Pods.

The cordon’s diagnostic

The cordon’s diagnostic:

# Substitute your own value before running:
NODE=node-1

kubectl describe node "$NODE" | grep -A 5 "Taints"

Taints:             node.kubernetes.io/unschedulable:NoSchedule

The node’s taint indicates the cordon’s state. The production rule is to verify the taint before maintenance.

The cordon’s operational patterns

The cordon’s operational patterns:

• Cordon before maintenance. The cordon’s effect is to block new Pods. The maintenance operation is the next step.
• Cordon before drain. The drain evicts the existing Pods. The cordon precedes the drain.
• Verify the cordon. The cordon’s taint is the diagnostic. The operator should verify the taint before proceeding.
• Uncordon after maintenance. The maintenance is complete; the node is again a candidate for new Pods.

The cordon’s failure modes

The cordon’s failure modes:

The diagnostic:

# Substitute your own value before running:
NODE=node-1

kubectl describe node "$NODE" | grep -A 5 "Taints"

The taint is the source of truth. The fix is to investigate the API server’s logs and the RBAC configuration.

The cordon’s anti-patterns

The cordon’s anti-patterns:

• Cordon without immediate drain. A node that is cordoned but not drained is a node that is wasting capacity. The existing Pods continue running; the node is not a candidate for new Pods. The fix is to drain immediately.
• Cordon as a substitute for taint. A node that is cordoned with no explanation is a node that is blocked. The fix is to use a taint with a reason.
• Cordon with a long grace period. The cordon’s effect is immediate. The grace period is the drain’s concern.

Quiz

Production discipline

• Cordon is the soft gate. The cordon blocks new Pods; the existing Pods continue running.
• Cordon before drain. The cordon’s effect is immediate; the drain’s effect is graceful.
• Uncordon after maintenance. The maintenance is complete; the node is again a candidate for new Pods.
• Verify the cordon. The cordon’s taint is the diagnostic. The operator should verify the taint.
• Audit the cordon at every node repave. A new node that joins the cluster with the wrong cordon is a node that is failing silently. The audit catches the failure.
• Document the cordon’s intent. A node that is cordoned with no explanation is a node that is blocked. The fix is to document the intent.
• Test the cordon in non-production. A staging cluster that mirrors production is the right place to test the cordon.