Taints and tolerations — the node-repulsion model

Taints and tolerations are Kubernetes’ node-repulsion model: a node carrying a taint rejects Pods that do not carry a matching toleration. Unlike node affinity, which is a Pod attraction rule (“run on nodes like X”), taints are a node rejection rule (“do not run here unless explicitly permitted”). The two complement each other: taints exclude by default, tolerations opt-in. This lesson establishes the model and the three effect types.

What is a taint

A taint is a key=value:effect triple attached to a Node. The key and value are arbitrary labels; the effect is one of three enumerated values that the scheduler and the kubelet both interpret.

kubectl taint nodes node-1 dedicated=prod:NoSchedule

node/node-1 tainted

The taint is recorded on the Node object:

kubectl get node node-1 -o jsonpath='{.spec.taints}' | jq

[
  {
    "key": "dedicated",
    "value": "prod",
    "effect": "NoSchedule"
  }
]

A second taint on the same key replaces the first; taints are not additive on the same key. Removing a taint requires the full key=value:effect triple:

kubectl taint nodes node-1 dedicated:NoSchedule-
kubectl taint nodes node-1 dedicated=prod:NoSchedule-   # equivalent

What is a toleration

A toleration is a Pod-level declaration that says “the Pod accepts one or more taints.” A toleration matches a taint when the key, value (if specified), and effect all match. The Pod then becomes a candidate for the tainted node.

apiVersion: v1
kind: Pod
metadata:
  name: billing-api
spec:
  tolerations:
    - key: dedicated
      operator: Equal
      value: prod
      effect: NoSchedule
  containers:
    - name: app
      image: registry.example.com/billing:1.4.0

The operator field is one of Equal (default) or Exists. Equal requires the value to match; Exists matches on key alone, which is useful for tolerating “any taint with this key” (for example, node.kubernetes.io/not-ready).

The matching model

A toleration matches a taint when key, value, and effect all satisfy the operator. The scheduler treats a node as feasible for a Pod when every taint on the node is matched by at least one of the Pod’s tolerations. A single unmatched taint rejects the node for that Pod.

flowchart LR
    N[Node N1<br/>taints: dedicated=prod:NoSchedule,<br/>gpu=true:NoSchedule] --> S{Scheduler}
    P1[Pod P1<br/>tolerations: dedicated=prod:NoSchedule] --> S
    P2[Pod P2<br/>tolerations: none] --> S
    S -->|matched all taints| A[P1 feasible on N1]
    S -->|unmatched gpu=true| B[P2 rejected]

The matching is all-tolerations-all-taints: the Pod must have at least one toleration for every taint on the node. A single toleration that matches all taints is enough; a Pod with three tolerations that match two taints is rejected.

The three effect types

The effect is the most important field. It determines what happens to a Pod that does not match the taint:

The three effects have a single ordering from weakest to strongest. NoExecute is the only one that removes running Pods; the other two are scheduling decisions only.

The built-in taints

Kubernetes itself adds a small set of well-known taints to nodes in specific conditions. These are the operational taints an operator will encounter daily:

A Pod tolerates all of these by default only if every tolerations entry references the key. The control plane injects some of these tolerations into Pods that opt in via node.kubernetes.io/not-ready, unreachable, and unschedulable tolerations; without explicit tolerations, the Pods are evicted when the condition is reported.

Inspecting taints and tolerations

# Substitute your own values before running:
NODE=node-19
POD=billing-7d8f-abcde

kubectl describe node "$NODE" | grep -i taint
kubectl get pod "$POD" -o jsonpath='{.spec.tolerations}' | jq

Taints:             dedicated=prod:NoSchedule
                    node.kubernetes.io/unschedulable:NoSchedule

A scheduled Pod that is “Pending” with a FailedScheduling event often has a missing toleration. The next lesson walks the diagnostic flow.

Quiz

Production discipline

Taints and tolerations are how production clusters reserve nodes for specific workloads. The standard pattern:

• Infrastructure nodes (CNI, logging, monitoring agents) carry dedicated=infra:NoExecute so non-tolerating workloads cannot land there.
• GPU nodes carry nvidia.com/gpu=present:NoSchedule plus the nvidia.com/gpu toleration on the GPU Pod, so non-GPU Pods do not waste capacity.
• Maintenance cordoning uses kubectl cordon which sets node.kubernetes.io/unschedulable:NoSchedule; this is non-evicting, which is exactly what you want before a drain.
• Failure response uses node.kubernetes.io/not-ready (auto-set by the node controller); almost no Pod should tolerate it, because running on a node that has lost its lease is a stale-fence hazard.

The principle: taints should name a reason, not a Pod. A taint called security=privileged:NoSchedule is a reason; a taint called app=payments:NoSchedule is a Pod label and should be a label, not a taint.