Taints and tolerations in production — anti-patterns and discipline

Taints and tolerations are one of the easiest features to misuse in Kubernetes. The taint command is one line; the toleration is a small YAML block. The misuses accumulate silently: a taint that is not in bootstrap automation is forgotten after a node replacement; a toleration that swallows a pressure taint hides a capacity leak; a taint that names a Pod instead of a reason blocks the wrong workloads. This lesson enumerates the anti-patterns and the discipline that prevents them.

Anti-pattern 1: taints that name Pods

A taint’s key should name a reason, not a workload.

# WRONG: a taint that names a Pod
taints:
  - key: app
    value: payments
    effect: NoSchedule

# RIGHT: a taint that names a reason
taints:
  - key: dedicated
    value: prod
    effect: NoSchedule

The first taint says “the node is for payments Pods.” This is a Pod selector expressed as a taint; the operator should have written a node label and a node affinity. The second says “the node is dedicated to production workloads of all kinds.” Any Pod with the matching toleration can land there.

The rule: a taint is a capability on a node, not a target for a Pod. The taint says “this node has hardware X” or “this node belongs to class Y”; the Pod that targets the class is the one that tolerates the taint.

flowchart LR
    A[Node: dedicated=prod] -->|tolerated by| B[Pod: billing-api]
    A -->|tolerated by| C[Pod: auth-service]
    A -->|rejected| D[Pod: dev-tooling]

The same taint is consumed by many Pods. Each Pod opts in with its toleration. The Pod-name-as-taint anti-pattern limits the taint to one consumer, which is what node affinity is for.

Anti-pattern 2: tolerations that swallow pressure

A Pod that tolerates memory-pressure keeps running on a node that is starving. The taint becomes a scheduling detail layered on top of a real resource problem.

# WRONG: a general Pod tolerating memory pressure
tolerations:
  - key: node.kubernetes.io/memory-pressure
    operator: Exists
    effect: NoSchedule

The Pod will keep running on a node that is under memory pressure. The kubelet’s eviction loop may still evict the Pod if the pressure is severe enough, but the toleration prevents the scheduler from routing new Pods away. The workload is using capacity that the cluster is trying to reclaim.

The rule: tolerate pressure taints only on workloads that can run on a memory-starved node by design (a metrics agent that needs to report the pressure, a node-local log collector). General workloads should not tolerate them; the right response to memory pressure is to add capacity or fix the leak.

Anti-pattern 3: missing label-taint pair

A node with a taint but no label is rejection without identity. The Pod that wants to land on the node cannot target it; the scheduler cannot route the Pod there.

# WRONG: only a taint
kubectl taint nodes node-gpu-1 nvidia.com/gpu=present:NoSchedule

# RIGHT: a label and a taint
kubectl label nodes node-gpu-1 node-role.kubernetes.io/gpu=true
kubectl taint nodes node-gpu-1 node-role.kubernetes.io/gpu=true:NoSchedule

The right pattern always pairs a label with a taint. The label is the selector; the taint is the repulsion. Without the label, the only way to land a Pod on the node is to tolerate the taint and hope the scheduler picks the node for some other reason.

Anti-pattern 4: taints that are not in bootstrap automation

A taint set with kubectl taint is recorded on the Node object. When the node is replaced (the typical cloud lifecycle), the new node has no taints. The replacement node runs general workloads that the taint was supposed to exclude.

kubectl taint nodes node-1 dedicated=prod:NoSchedule
# Later, the node is replaced. The new node has no taint.

The fix: drive taints from the bootstrap. Three options:

# Option 1: kubelet flag
--register-with-taints=dedicated=prod:NoSchedule

# Option 2: bootstrap controller that re-applies the taint
# A controller watches the node's class label and re-applies
# the corresponding taint if it is missing.

# Option 3: node feature discovery
# NFD labels nodes based on hardware; a chained controller
# translates the label into a taint.

The right pattern in a managed environment is to let the cluster’s bootstrap pipeline handle the taint. The operator should never set a taint with kubectl taint and hope it survives.

Anti-pattern 5: tolerations on the default namespace

The default namespace has no Pod Security Standards enforcement, no quotas, and no audit. A Pod in the default namespace that tolerates a maintenance taint will continue running on a node that is being taken out of service.

kubectl get pods -n default -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName,TOLERATIONS:.spec.tolerations

The fix: do not run production workloads in the default namespace. Always name the namespace explicitly; a Pod in team-a-prod cannot be confused with a Pod in team-b-prod.

Anti-pattern 6: tolerating not-ready at all

A Pod that tolerates not-ready with tolerationSeconds: 0 (or no tolerationSeconds) keeps running on a node that the cluster has marked as dead. The Pod is invisible to the service routing layer; the Pod runs, but it does not serve traffic.

# WRONG: tolerating not-ready forever
tolerations:
  - key: node.kubernetes.io/not-ready
    operator: Exists
    effect: NoExecute

The right pattern: tolerate not-ready for a finite window, with tolerationSeconds calibrated to the cluster’s recovery time. The 300s default is the right answer for most workloads.

The taints-and-tolerations audit

A production cluster should be audited at every release and at every node replacement. The audit has four questions:

1. Which taints are set on each node? Run kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints and diff the keys against the cluster’s intended inventory.
2. Which Pods tolerate which taints? Grep every Pod manifest for tolerations: and match the keys against the cluster’s taints.
3. Are the taints reproducible from bootstrap? Diff the kubelet flags, the node bootstrap controller, or the NFD configuration against the observed taints.
4. Do the production-critical Pods tolerate the right taints? A Pod that tolerates not-ready with tolerationSeconds: 0 is a stale-fence hazard. A Pod that tolerates memory-pressure is hiding a capacity leak.

# List all taints on all nodes
kubectl get nodes -o json | jq '.items[].spec.taints'

# List all unique tolerations across all namespaces
kubectl get pods -A -o json | jq -r '.items[].spec.tolerations[].key' | sort -u

The two outputs should be consistent: the cluster’s taints should be a subset of the cluster’s tolerations, and the cluster’s tolerations should be a subset of the cluster’s intended inventory.

Quiz

Production discipline

• Taints must name a reason. A taint called class=production is a reason; a taint called app=payments is a Pod label and should be a label.
• Tolerations must be in version control. A Pod without a version-controlled toleration is a Pod that tolerates whatever taints the cluster thinks it should tolerate. Put the tolerations in the Helm chart, the Kustomize patch, or the static manifest.
• Taints must be in bootstrap automation. A taint that is not in the bootstrap is a forgotten taint. The cluster bootstrap pipeline should be the only place taints are declared for the general fleet.
• Pressure taints must be tolerated deliberately. A Pod that tolerates memory-pressure, disk-pressure, or pid-pressure should have a written justification for the toleration. The default is “no toleration.”
• Audit taints every release. A node repaved after a failure should pick up the same taints as the one it replaced. The audit is the only way to catch a forgotten taint.
• Diff tolerations across the cluster at every upgrade. A toleration that was added by a Helm chart and is no longer needed may not be removed automatically. The cleanup is the operator’s responsibility.