Dedicated nodes — infra, GPU, and workload isolation

Taints and tolerations shine in the dedicated-node pattern: a node class reserved for specific workloads by labelling the nodes (for affinity) and tainting the nodes (for repulsion). The two are complementary. A label makes the node findable by a Pod that wants it; a taint makes the node unfindable by a Pod that does not want it. This lesson walks the canonical patterns.

The label-taint pairing

The first principle: a node class should have a label and a taint with the same key. The label is for selectors (node affinity, kubectl --selector); the taint is for repulsion. Using a single key for both makes the relationship explicit and the operation greppable.

kubectl label nodes node-gpu-1 node-role.kubernetes.io/gpu=true
kubectl taint nodes node-gpu-1 node-role.kubernetes.io/gpu=true:NoSchedule

A Pod that should run on GPU nodes:

apiVersion: v1
kind: Pod
metadata:
  name: trainer
spec:
  nodeSelector:
    node-role.kubernetes.io/gpu: "true"
  tolerations:
    - key: node-role.kubernetes.io/gpu
      operator: Exists
      effect: NoSchedule
  containers:
    - name: train
      resources:
        limits:
          nvidia.com/gpu: 1

The Exists operator matches any taint with the key, which is the production-friendly choice when the taint value is not a stable identifier.

The four common dedicated node classes

Production clusters typically have four reserved node classes.

1. Infrastructure nodes

Nodes that run platform add-ons: CNI agents, kube-proxy, metrics-server, log shippers, node-local DNS cache, CNI chained plugins, and the cluster’s own observability stack.

# Substitute your own node names before running:
INFRA_NODES=(infra-01 infra-02 infra-03)

kubectl label nodes "${INFRA_NODES[@]}" node-role.kubernetes.io/infra=true
kubectl taint nodes "${INFRA_NODES[@]}" node-role.kubernetes.io/infra=true:NoSchedule

The toleration is added to the platform add-on Pods:

tolerations:
  - key: node-role.kubernetes.io/infra
    operator: Exists
    effect: NoSchedule

If the platform stack is deployed via a Helm chart or a GitOps manifest, the tolerations are baked into the chart’s values. Helm’s nodeSelector and tolerations are the standard place to put them.

2. GPU nodes

GPUs are scarce and expensive. A GPU Pod tolerates the taint and requests the GPU resource; a non-GPU Pod is rejected.

# Substitute your own node names before running:
GPU_NODES=(gpu-01 gpu-02)

kubectl label nodes "${GPU_NODES[@]}" node-role.kubernetes.io/gpu=true
kubectl taint nodes "${GPU_NODES[@]}" nvidia.com/gpu=present:NoSchedule

The taint uses the resource name (nvidia.com/gpu) as the key rather than a generic role label, because the taint’s purpose is to advertise the hardware. A Pod without the GPU also has no reason to be on the node; the resource request drives the scheduler, the taint confirms it.

3. Latency-sensitive nodes

Nodes reserved for workloads that require predictable scheduling latency: real-time, low-jitter, single-tenant tenancy. The taint is usually class=latency-sensitive:NoSchedule combined with a label that the Pod uses for affinity.

nodeSelector:
  class: latency-sensitive
tolerations:
  - key: class
    operator: Equal
    value: latency-sensitive
    effect: NoSchedule

The Pod’s priorityClassName is also typically set higher than the default so that preemption does not free the node for someone else.

4. Compliance-scoped nodes

Nodes that must run only Pods that meet a specific compliance standard (PCI, FedRAMP, healthcare). The taint is compliance=pci:NoSchedule and the Pod toleration is combined with a Pod Security Standards label on the namespace.

tolerations:
  - key: compliance
    operator: Equal
    value: pci
    effect: NoSchedule

The combination of taint + PSS label means the Pod is allowed onto the node only if the namespace is also scoped to the same compliance boundary. Operators must also enforce this with a ValidatingAdmissionPolicy that rejects Pods whose namespace does not carry the matching label; the taint alone is not a security control.

Bootstrap and reconciliation

Taints survive node reboots but not node replacement. A production cluster must drive taints from the node bootstrap so a new node picks up the taints before the kubelet registers it.

Two common patterns:

# Pattern A: kubelet flag at bootstrap
--register-with-taints=dedicated=infra:NoSchedule

# Pattern B: node bootstrap controller / operator
# The operator watches a ConfigMap and applies the taint;
# it also re-applies the taint if it is removed.

Pattern A is simpler but requires that the entire fleet share the same taint. Pattern B is more flexible — different node classes can carry different taints, and the operator can label the node based on hardware introspection.

flowchart LR
    A[Node starts] --> B[kubelet registers]
    B --> C{Node bootstrap<br/>controller}
    C -->|Label node| D[Apply taint]
    D --> E[Node joins general pool]
    E --> F{Scheduler sees<br/>taint + label}
    F -->|Tolerating Pod| G[Schedule]
    F -->|Non-tolerating Pod| H[Reject silently]

Inspecting and verifying taints

kubectl get nodes -o custom-columns=NAME:.metadata.name,TAINTS:.spec.taints

NAME           TAINTS
node-1         <none>
node-2         [node-role.kubernetes.io/infra=true:NoSchedule]
node-3         [node-role.kubernetes.io/gpu=true:NoSchedule]
node-4         [node-role.kubernetes.io/infra=true:NoSchedule,
                node.kubernetes.io/unschedulable:NoSchedule]

The combined node-4 is a cordoned infrastructure node — the operator is preparing it for maintenance. The unschedulable taint comes from kubectl cordon; the operator must remove both when bringing the node back into service.

Quiz

Production discipline

• Drive taints from bootstrap, not from hand. A node repaved after a failure should pick up the same taints as the one it replaced. Put the taint in the --register-with-taints kubelet flag, or in a node bootstrap controller that re-applies the taint after registration.
• Pair every taint with a label. A taint without a label is rejection without identity; the operator cannot write a Pod that targets the node class. A label without a taint is identity without isolation; the operator can target the node class but the cluster can also place unrelated Pods there.
• Use Exists for tolerations where the value is unstable. When the taint value encodes dynamic state (a hardware serial, a node feature), the toleration should use Exists and match on the key alone.
• Audit taints at every cluster upgrade. A kubelet upgrade that silently drops --register-with-taints is a capacity leak. Diff the kubelet flags before and after every upgrade.
• Watch the cordon-taint interaction. kubectl cordon adds node.kubernetes.io/unschedulable:NoSchedule to the node. This is a separate taint from the dedicated taint; do not assume that letting a Pod tolerate the dedicated taint means it can schedule on a cordoned node.