Taints and tolerations preview — the counterpart to Node Affinity

Taints and tolerations are the counterpart to Node Affinity. Where Node Affinity says “this Pod wants this node,” taints say “this node rejects Pods that don’t tolerate me.” Combined, they give both sides a voice: the Pod chooses what it wants; the node chooses what it accepts. This lesson covers the interaction and the production patterns (covered in detail in Part XXVI).

The asymmetry

flowchart LR
    A[Pod] -->|affinity: I want...| B[Node]
    B -->|taint: I reject...| A
    A -->|toleration: I tolerate...| B

Node Affinity is pod-driven: the Pod specifies constraints, the scheduler filters nodes. Taints are node-driven: the node specifies constraints, the scheduler filters Pods. The two are complementary.

The standard taints

Control-plane taint

kubectl taint nodes cp-01 node-role.kubernetes.io/control-plane=:NoSchedule

Set by kubeadm on control-plane nodes. Pods without the toleration cannot land on control-plane nodes.

Dedicated taint

kubectl taint nodes gpu-pool dedicated=gpu:NoSchedule

A pool of GPU nodes is dedicated to GPU workloads. Pods without the toleration cannot land here; GPU workloads explicitly tolerate it.

Spot taints

# AWS
kubectl taint nodes spot-pool node.kubernetes.io/lifecycle=spot:PreferNoSchedule
# GCP
kubectl taint nodes spot-pool cloud.google.com/gke-spot=true:PreferNoSchedule

Cloud providers set taints on spot instances. Workloads that cannot tolerate preemption either avoid them or explicitly tolerate them.

The three effects

NoExecute is the strongest: it evicts running Pods that don’t tolerate. Used in cluster maintenance and node failure scenarios.

flowchart TB
    A["Taint: dedicated=gpu:NoSchedule"] --> B{Pod tolerates?}
    B -->|no| C[Pod not scheduled here]
    B -->|yes| D[Pod scheduled here]
    A2["Taint: dedicated=gpu:NoExecute"] --> E{Pod tolerates?}
    E -->|no| F["Existing Pod evicted<br/>new Pod not scheduled"]
    E -->|yes| G[Pod continues]

Tolerations

spec:
  tolerations:
  - key: dedicated
    operator: Equal
    value: gpu
    effect: NoSchedule

The Pod tolerates the taint dedicated=gpu:NoSchedule. The toleration must match the taint exactly (or use operator: Exists for any taint with that key/effect).

TolerationSeconds

spec:
  tolerations:
  - key: node.kubernetes.io/unreachable
    operator: Exists
    effect: NoExecute
    tolerationSeconds: 300

The Pod tolerates the taint for 300 seconds; after that, the Pod is evicted. Used for graceful handling of node failures.

Combining with Node Affinity

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.io/arch
            operator: In
            values: ["amd64"]
  tolerations:
  - key: dedicated
    operator: Equal
    value: gpu
    effect: NoSchedule

The Pod:

• Wants (affinity): amd64 nodes (hard).
• Tolerates (tolerations): the dedicated=gpu taint.

A node must satisfy both: it must be amd64 (affinity) and the Pod must tolerate the node’s taints. If the node is arm64, the affinity filter eliminates it. If the node has a taint the Pod does not tolerate, the taint filter eliminates it.

flowchart TB
    A[Pod with affinity + toleration] --> B["Filter 1: Affinity"]
    B --> C{amd64?}
    C -->|yes| D[Survives]
    C -->|no| E[Eliminated]
    D --> F["Filter 2: Taints"]
    F --> G{Pod tolerates<br/>node's taints?}
    G -->|yes| H[Survives]
    G -->|no| I[Eliminated]
    H --> J[Score phase]

Production patterns

Pattern 1: GPU pool

# Node-side: taint the GPU pool
kubectl taint nodes gpu-01 dedicated=gpu:NoSchedule

# Pod-side: tolerate the taint
spec:
  tolerations:
  - key: dedicated
    operator: Equal
    value: gpu
    effect: NoSchedule
  nodeSelector:
    kubernetes.io/arch: amd64
  containers:
  - name: training
    resources:
      limits:
        nvidia.com/gpu: 1

The GPU pool is dedicated to GPU workloads. Non-GPU workloads are blocked by the taint.

Pattern 2: spot instances

spec:
  nodeSelector:
    node.kubernetes.io/instance-type: m5.large
  tolerations:
  - key: node.kubernetes.io/lifecycle
    operator: Equal
    value: spot
    effect: PreferNoSchedule

The Pod can run on spot instances; the PreferNoSchedule effect makes it a soft preference.

Pattern 3: dedicated tenant

kubectl taint nodes tenant-a dedicated=tenant-a:NoSchedule

A node pool for tenant a. Only workloads with the toleration can land there.

Common mistakes

Mistake 1: missing toleration

# A Pod that runs on every node except tainted ones
spec:
  containers:
  - name: web
  # no tolerations

If every node has dedicated=gpu:NoSchedule, the Pod is unschedulable. Add the toleration.

Mistake 2: over-tainting

kubectl taint nodes node-01 dedicated=team-a:NoSchedule
kubectl taint nodes node-01 dedicated=team-b:NoSchedule
kubectl taint nodes node-01 dedicated=team-c:NoSchedule

A node with multiple dedicated taints requires a Pod to tolerate all of them. Most workloads tolerate one; the node is effectively reserved for a specific Pod.

The discipline: one taint per node, clearly describing the workload class.

Mistake 3: toleration without affinity

spec:
  tolerations:
  - key: dedicated
    operator: Equal
    value: gpu
    effect: NoSchedule

A Pod that tolerates the GPU taint but has no nodeSelector for GPU. The Pod may land on GPU nodes (preferred by the toleration) but also on non-GPU nodes (the toleration only says “I can be on GPU nodes” — it does not say “I must be”).

The fix: combine with nodeSelector or required Node Affinity.

Mistake 4: NoExecute without grace

kubectl taint nodes node-01 maintenance=true:NoExecute

This evicts every Pod on the node immediately. The Pod’s terminationGracePeriodSeconds is the only buffer. For graceful maintenance, use PreferNoSchedule or cordon + drain instead.

Quiz

Production discipline

• Taints are for node operators; tolerations are for workload authors. Both must agree for the workload to land on the node.
• One taint per node, with a clear key/value. Multiple taints on one node are an anti-pattern.
• Combine toleration with affinity. A toleration alone is “I can”; combined with nodeSelector, it is “I want.”
• Use NoExecute deliberately. Eviction is destructive; use it for maintenance and node failure, not for soft placement.
• Audit taints and tolerations. A dashboard that lists nodes’ taints and Pods’ tolerations catches misconfigurations.

Taints and tolerations are covered in detail in Part XXVI. This lesson is the preview for Node Affinity users. The discipline is in the combination: affinity says “I want”; toleration says “I tolerate.” Operators who combine them have workloads that land predictably.