Effect types — NoSchedule, PreferNoSchedule, NoExecute

The effect field is the only part of a taint that changes the user-visible behaviour. This lesson walks the three values, the operational differences between them, and the patterns that arise from combining them.

NoSchedule — the hard filter

NoSchedule is the default effect and the most common. A taint with effect: NoSchedule causes the scheduler’s TaintToleration filter plugin to reject the node for any Pod that does not have a matching toleration. The Pod stays in the unscheduled queue; if no other node is feasible, the Pod stays Pending with a FailedScheduling event.

apiVersion: v1
kind: Pod
metadata:
  name: web-1
spec:
  containers:
    - name: app
      image: registry.example.com/web:1.0.0
  tolerations:
    - key: dedicated
      operator: Equal
      value: prod
      effect: NoSchedule

The Pod is allowed onto nodes with the matching taint; it is also allowed onto nodes with no taint at all. The taint constrains; it does not require.

A NoSchedule taint does not affect running Pods. If a Pod is already on the node when the taint is added, the Pod continues running, the kubelet does not evict it, and the next reconciliation cycle does not move it. The Pod becomes a “tainted” running workload that no longer has matching scheduling criteria. This is the property that makes NoSchedule safe for staged maintenance.

PreferNoSchedule — the soft hint

PreferNoSchedule is the same logic as NoSchedule from the filter plugin’s perspective, except the scheduler does not reject the node for the mismatch. Instead, the scheduler decreases the node’s score by a small amount (currently NodeTaintPolicy weight, default 1 in the scoring weights).

flowchart LR
    A[Pod submitted] --> B[Filter plugins]
    B --> C[Feasible set]
    C --> D[Score plugins]
    D -->|PreferNoSchedule reduces score| E[Winning node probably not tainted]
    E --> F[Bind]

The result: the scheduler tries to avoid the tainted node, but if no other node is feasible, the tainted node wins. This is the difference between “do not schedule here” and “do not schedule here unless you must.”

When to use PreferNoSchedule:

• Soft isolation. A node pair carrying data=local:PreferNoSchedule is a hint that the cluster operator would prefer to keep general workloads off these nodes, but tolerating them is acceptable when capacity is thin.
• Canary rollouts. New nodes can be tainted with preflight=run:PreferNoSchedule so the scheduler prefers them for new Pods, but production workloads do not fail to schedule if the canary set is too small.
• Soft affinity. If you want to encourage Pods onto a node class but not block them, PreferNoSchedule is the right shape.

When not to use PreferNoSchedule:

• Compliance or security boundaries. The scheduler does not guarantee placement; it just prefers it. A pod that explicitly tolerates the taint or that has no other option lands on the node anyway.
• Cost-bound nodes. A spot-instance node tainted cloud=spot:PreferNoSchedule does not protect against a spot interruption; the taint is a placement hint, not a scheduling policy.

NoExecute — the eviction lever

NoExecute is the only effect that removes running Pods. When a NoExecute taint is added to a node, the kubelet’s node-level controller (the “node lifecycle controller” in kube-controller-manager, plus the per-node eviction logic in kubelet) walks the node’s running Pods and evicts any Pod whose tolerations do not match the taint.

The eviction is graceful by default. The Pod’s containers receive SIGTERM, the kubelet waits for the Pod’s terminationGracePeriodSeconds (default 30s), then sends SIGKILL. The Pod’s status moves to Failed with reason Evicted. The replacement Pod, if one is managed by a controller, is scheduled by the scheduler under the same constraints.

NoExecute is the most production-loaded effect. Adding NoExecute to a node with running Pods is a destructive operation; in production, the operation must be staged.

tolerationSeconds

A toleration can specify how long the Pod tolerates the taint before eviction. The order of precedence is:

1. The Pod’s tolerationSeconds for the matching taint
2. The taint’s tolerationSeconds (set on the taint itself)
3. The kubelet’s default eviction grace period (300 seconds)

tolerations:
  - key: node.kubernetes.io/unreachable
    operator: Exists
    effect: NoExecute
    tolerationSeconds: 30

This declares “if the node goes unreachable, evict me after 30 seconds”. Most operational taints are tolerated for 300 seconds by default; shorter values make the cluster more sensitive to transient network partitions, longer values ride out brief blips.

Combining effects for staged maintenance

The three effects let the operator build a progression:

stateDiagram-v2
    [*] --> Operating
    Operating --> NoSchedule: taint NoSchedule<br/>no new Pods
    NoSchedule --> NoExecute: taint NoExecute<br/>drain via tolerationSeconds
    NoExecute --> Maintenance: all pods evicted
    Maintenance --> Operating: remove taints

The pattern, in production:

1. Apply node.kubernetes.io/unschedulable:NoSchedule (via kubectl cordon) — the node is no longer a candidate for new Pods. Existing Pods continue running.
2. Schedule the maintenance. The window is now self-paced; you can wait for the existing Pods to finish naturally.
3. If you need to force the existing Pods off the node (rolling reboot, kernel upgrade), apply a second NoExecute taint with a finite tolerationSeconds so the Pods drain gracefully.

Adding PreferNoSchedule to the same node is almost never necessary; if the node is cordoned, the scheduler will not consider it for new Pods.

Diagnosing a taint-related failure

# A Pod stuck in Pending, from `kubectl get pods --field-selector status.phase=Pending`:
PENDING_POD=billing-7d8f-abcde

kubectl describe pod "$PENDING_POD" | grep -A 3 "Events:"

Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               ----
  Warning  FailedScheduling  12s   default-scheduler  0/5 nodes are available:
                                            1 node(s) had taint {dedicated=prod:NoSchedule},
                                            2 node(s) had taint {gpu=true:NoSchedule},
                                            2 node(s) didn't match Pod's node affinity.

The events list every taint that rejected every node. The fix is either to add the tolerations to the Pod, remove the taint from the node, or change the Pod’s deployment so it runs on a different node class.

Quiz

Production discipline

• Use NoSchedule for hard isolation. Infrastructure nodes, GPU nodes, and security-sensitive workloads should carry NoSchedule taints that the appropriate Pods tolerate.
• Use PreferNoSchedule sparingly. A soft hint becomes noise if it is overridden most of the time. If the Pod always tolerates the taint or always runs on the node despite the hint, the taint has no effect.
• Treat NoExecute as a write operation. Adding NoExecute to a node evicts running Pods. Stage it. Use kubectl drain, which adds NoSchedule first, then NoExecute with a tolerance, after the cluster has time to reschedule.
• Reserve tolerationSeconds for batch and stateless workloads. A long tolerationSeconds on a stateful workload means the workload tolerates a node failure for that long before eviction; this can hide a slow-fail node, prolonging a partition.
• Audit taints at every node repave. A taint set on a node that the new node does not inherit is a forgotten thing. Drive taints from the cluster bootstrap (cloud-init, kubelet flags, or a node bootstrap controller) so they survive node replacement.