Node Affinity patterns — GPU, dedicated pools, latency zones

Node Affinity is a primitive; the production patterns combine it with taints, tolerations, and preferred preferences for full placement control. This lesson walks the canonical patterns: GPU pools, dedicated tenants, latency-sensitive zones, ARM workloads, and the combination patterns.

Pattern 1: GPU pool

A cluster has a pool of GPU nodes. Only GPU workloads land there; non-GPU workloads are kept off.

# Node-side: taint the GPU pool
kubectl taint nodes gpu-01 dedicated=gpu:NoSchedule
kubectl label nodes gpu-01 node.kubernetes.io/instance-type=g5.12xlarge

# Workload-side
spec:
  nodeSelector:
    node.kubernetes.io/instance-type: g5.12xlarge
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: dedicated
            operator: In
            values: ["gpu"]
  tolerations:
  - key: dedicated
    operator: Equal
    value: gpu
    effect: NoSchedule
  containers:
  - name: training
    image: training:v1
    resources:
      limits:
        nvidia.com/gpu: 1

The Pod:

• Must be on an instance type with GPU (nodeSelector).
• Must have the dedicated=gpu label (requiredDuringScheduling).
• Tolerates the dedicated=gpu:NoSchedule taint.
• Requests a GPU (nvidia.com/gpu: 1).

flowchart TB
    A[GPU Pod] --> B{nodeSelector:<br/>GPU instance type?}
    B -->|yes| C{affinity:<br/>dedicated=gpu?}
    C -->|yes| D{toleration:<br/>dedicated=gpu:NoSchedule?}
    D -->|yes| E[Scheduled]
    B -->|no| F[Eliminated]
    C -->|no| F
    D -->|no| F

Pattern 2: Dedicated tenant pool

A multi-tenant cluster has per-tenant node pools. Each tenant’s workloads land only on their pool.

kubectl taint nodes tenant-a-pool dedicated=tenant-a:NoSchedule
kubectl label nodes tenant-a-pool tenant=team-a

# Tenant A's workload
spec:
  nodeSelector:
    tenant: team-a
  tolerations:
  - key: dedicated
    operator: Equal
    value: tenant-a
    effect: NoSchedule

A non-tenant-a workload without the toleration cannot land on tenant-a nodes. The tenant-a workload tolerates the taint and is restricted to its pool.

Pattern 3: Latency-sensitive zone

A workload prefers a specific zone for latency but works elsewhere if the zone is unavailable.

spec:
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        preference:
          matchExpressions:
          - key: topology.kubernetes.io/zone
            operator: In
            values: ["us-east-1a"]

The Pod prefers us-east-1a; other zones work. The weight (100) makes the preference strong.

Combine with topology spread to spread replicas across zones:

spec:
  affinity:
    nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        preference:
          matchExpressions:
          - key: topology.kubernetes.io/zone
            operator: In
            values: ["us-east-1a"]
  topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app: web

The Pods prefer us-east-1a but spread evenly across zones (skew ≤ 1).

Pattern 4: ARM workloads

An ARM-built image can only run on ARM nodes.

spec:
  nodeSelector:
    kubernetes.io/arch: arm64

The image must also be ARM (e.g., myapp:v1-arm64 or a multi-arch image). A x86 image on an ARM node is a crashloop.

Pattern 5: Combined required + preferred

A workload has hard and soft placement constraints.

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: kubernetes.io/arch
            operator: In
            values: ["amd64"]
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        preference:
          matchExpressions:
          - key: disk
            operator: In
            values: ["ssd"]
      - weight: 50
        preference:
          matchExpressions:
          - key: topology.kubernetes.io/zone
            operator: In
            values: ["us-east-1a"]

The workload must be amd64; prefers SSD and zone us-east-1a. A node matching both preferences gets a 150 bonus; a node matching only one gets 100 or 50.

Pattern 6: Failover zones

A workload that should land in a specific zone but fail over to a backup zone if the primary is unavailable.

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: topology.kubernetes.io/zone
            operator: In
            values: ["us-east-1a"]
        - matchExpressions:
          - key: topology.kubernetes.io/zone
            operator: In
            values: ["us-east-1b"]

The Pod can land in us-east-1a or us-east-1b. If both are unavailable, the Pod is unschedulable. The OR is at the term level.

Pattern 7: Instance-type pinning

A workload requires a specific instance type (e.g., for hardware acceleration).

spec:
  nodeSelector:
    node.kubernetes.io/instance-type: m5.4xlarge

The Pod must land on an m5.4xlarge instance. The cluster must have such instances.

Anti-patterns

Anti-pattern 1: too many preferences

preferredDuringSchedulingIgnoredDuringExecution:
- weight: 50
  preference:
    matchExpressions:
    - key: zone
      operator: In
      values: ["us-east-1a"]
- weight: 50
  preference:
    matchExpressions:
    - key: arch
      operator: In
      values: ["amd64"]
- weight: 50
  preference:
    matchExpressions:
    - key: disk
      operator: In
      values: ["ssd"]
- weight: 50
  preference:
    matchExpressions:
    - key: tier
      operator: In
      values: ["production"]

Four preferences with weight 50. The score is the sum of weights of matching preferences. A node that matches all four gets 200; a node that matches one gets 50. The scheduler picks the highest, but the weights do not reflect any real priority — every preference is equal.

The discipline: use weights that reflect actual priority. A latency-sensitive preference is weight 100; a “nice to have” preference is weight 10.

Anti-pattern 2: pinning everything

spec:
  nodeSelector:
    kubernetes.io/hostname: node-01

Every Pod in the Deployment lands on node-01. The node becomes a single point of failure; the deployment has no spread. Use topology spread or anti-affinity instead.

Anti-pattern 3: toleration without nodeSelector

spec:
  tolerations:
  - key: dedicated
    operator: Equal
    value: gpu
    effect: NoSchedule

A Pod that tolerates the GPU taint but has no nodeSelector. The Pod can land on GPU nodes (the toleration says “I can be there”) but can also land on non-GPU nodes. The toleration alone is insufficient.

The fix: combine with nodeSelector or required affinity.

Anti-pattern 4: conflicting rules

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: zone
            operator: In
            values: ["us-east-1a"]
        - matchExpressions:
          - key: zone
            operator: In
            values: ["us-east-1b"]

The OR is at the term level, so this is “zone is us-east-1a OR us-east-1b.” A common mistake: writing this as AND (both terms must match), which is unsatisfiable.

flowchart TB
    A[Two terms] -->|OR| B["Term 1: zone=a"]
    A -->|OR| C["Term 2: zone=b"]
    B --> D{Match?}
    C --> D
    D -->|yes| E[Feasible]
    D -->|no| F[Eliminated]

Quiz

Production discipline

• Combine required and preferred. Required for compliance and hardware; preferred for performance and cost.
• Combine with taints and tolerations. Affinity says “I want”; taints say “I reject.” Together they place workloads precisely.
• Use well-known labels for portability. A manifest with topology.kubernetes.io/zone works across clusters with the label.
• Avoid pinning everything. Pinning all replicas to one node is a single point of failure.
• Test with the cluster’s labels. A Pod that requires a label no node has is unschedulable. Validate in CI.

Node Affinity is a primitive; the patterns combine it with the rest of the placement toolkit. Operators who master the patterns have workloads that land predictably.