Affinity, taint, and topology failures

After resource failures, the dominant scheduling failures are affinity, taint, and topology rejections. The message is precise; the fix is to align the constraint with the cluster’s reality. This lesson walks the four common patterns and the diagnostic moves for each.

Node affinity failures

The NodeAffinity filter plugin rejects a node when the Pod’s spec.affinity.nodeAffinity does not match any node in the cluster. The failure message:

0/5 nodes are available: 5 node(s) didn't match Pod's node affinity.

The Pod’s spec:

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: workload
                operator: In
                values: [critical]

The Pod is asking for a node with workload=critical; the cluster has no node with that label. The fix is to label a node or remove the constraint.

The matchExpressions shape is the most common source of silent failures. A typo in the key, a missing label, or a label that does not match the value is enough to reject every node.

kubectl get nodes -L workload

NAME      WORKLOAD
node-1    critical
node-2    critical
node-3    standard
node-4    standard
node-5    standard

A Pod with operator: In, values: [critical] is feasible on node-1 and node-2. If the cluster has been scaled down to only node-3, node-4, node-5, the Pod is rejected for the same spec. The fix is to either re-label the nodes or relax the affinity.

Pod affinity and anti-affinity failures

The PodAffinity and PodAntiAffinity filter plugins require a co-located (or anti-colocated) Pod. The failure message:

0/5 nodes are available: 3 node(s) didn't match Pod's anti-affinity rules.

The Pod’s spec:

spec:
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - topologyKey: kubernetes.io/hostname
          labelSelector:
            matchLabels:
              app: cache

The Pod is asking “do not put me on a node with another Pod with app=cache.” If the cluster has every node running a cache Pod, the new Pod is rejected.

The fix is either:

• Ensure the cluster has enough nodes to satisfy the rule. The rule requires a node without an app=cache Pod; if every node has one, there is no feasible node.
• Relax the rule to preferred instead of required. The Pod will land on a node that may have a co-located Pod, but the rule is best-effort.
• Reduce the spread of the existing Pods. If the cache is concentrated on two nodes, the other three are feasible.

The topologyKey is critical. kubernetes.io/hostname restricts the rule to a single node; topology.kubernetes.io/zone restricts the rule to a zone; topology.kubernetes.io/region restricts the rule to a region. A topologyKey that does not label any node is a failure mode (the constraint is unsatisfiable).

Taint failures

The TaintToleration filter plugin rejects a node that carries a taint the Pod does not tolerate. The message:

0/5 nodes are available: 3 node(s) had taint {dedicated=prod:NoSchedule}, 2 node(s) had taint {gpu=true:NoSchedule}.

The fix is to add the toleration to the Pod or remove the taint from the node. The lesson on dedicated nodes (Part XXVI) covers the patterns.

A common production bug: a Pod is tolerating the dedicated taint but the cluster has a new node class with a different taint. The Pod is scheduled on the old node class only; the new class is rejected.

Topology failures

The NodeAffinity filter plugin also handles topology constraints expressed via spec.topologySpreadConstraints. The failure message:

0/5 nodes are available: 2 node(s) didn't match topology spread constraints.

The Pod’s spec:

spec:
  topologySpreadConstraints:
    - maxSkew: 1
      topologyKey: kubernetes.io/hostname
      whenUnsatisfiable: DoNotSchedule
      labelSelector:
        matchLabels:
          app: api

The constraint is “spread app=api across nodes with at most 1 Pod difference.” If every node has 2 Pods and the new Pod would make it 3, the rule is violated and the Pod is rejected.

The whenUnsatisfiable: DoNotSchedule setting is the hard-rejection mode. The ScheduleAnyway mode is a soft scoring hint; the rule is preferred but not required.

The filter interaction

The scheduler runs every filter plugin on every node. A node is feasible for the Pod only if every filter says feasible. The FailedScheduling event lists every filter that rejected every node, but the ordering of the messages is not always the order of the rejection.

flowchart TD
    A[Pod under scheduling] --> B[NodeResourcesFit]
    B --> C[NodeName]
    C --> D[NodeAffinity]
    D --> E[TaintToleration]
    E --> F[NodePorts]
    F --> G[PodAffinity]
    G --> H[NodeAffinity]
    H --> I[VolumeBinding]
    I --> J[PodTopologySpread]
    J --> K[All filters passed?<br/>feasible nodes]

A node that fails multiple filters is reported under each filter name. The operator reads the message and decodes which filters are rejecting the cluster.

The impossible-constraint failure mode

The most dangerous failure mode is a constraint that no node in the cluster can satisfy. The Pod is Pending forever; the operator checks the message, sees a “didn’t match” reason, and tries to fix the wrong thing.

# A Pod that requires a label no node has
spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: cluster-version
                operator: In
                values: [v2.0]

0/5 nodes are available: 5 node(s) didn't match Pod's node affinity.

The fix is to label a node with cluster-version=v2.0 or remove the constraint. The lesson is to validate the constraint against the cluster’s actual labels before deploying the Pod.

kubectl get nodes -L cluster-version

If no node has the label, the constraint is impossible. The operator should fail the CI build that introduced the constraint.

The diagnostic workflow

1. Read the message. The filter plugins each have a distinct format. Insufficient memory is the resource plugin; didn't match Pod's node affinity is the affinity plugin; had taint is the taint plugin.
2. Look at the cluster’s labels. A node affinity rejection means the cluster does not have the right labels. A taint rejection means the cluster has taints the Pod does not tolerate.
3. Look at the cluster’s existing Pods. A pod affinity rejection means the cluster’s existing Pods are spread in a way that makes the rule unsatisfiable.
4. Look at the cluster’s topology. A topology rejection means the cluster’s node distribution is too tight to satisfy the rule.
5. Apply the fix. The fix is usually one of:
   • Label a node.
   • Remove the constraint.
   • Tolerate the taint.
   • Reduce the rule’s spread.

Quiz

Production discipline

• Validate constraints in CI. A Pod that requires a label the cluster does not have is a Pod that will be Pending forever. The CI pipeline should pull the cluster’s labels and validate every constraint against the actual labels.
• Prefer preferred over required for soft rules. A requiredDuringSchedulingIgnoredDuringExecution rule is a hard rejection. The same intent expressed as preferredDuringSchedulingIgnoredDuringExecution is a scoring hint. Use the soft form unless the rule is business-critical.
• A cluster with a sustained “didn’t match” message is a cluster with a wrong constraint. The Pod’s spec is wrong, or the cluster’s labels are wrong. The fix is to fix the spec, not to add nodes.
• Audit affinity rules at every release. A new workload that requires a label the cluster does not have will fail at deployment. The CI should catch it; the operator should still audit.
• Topology constraints are difficult to test. A Pod spread across nodes is hard to test in a CI cluster with few nodes. The preferred mode is the safer default; the required mode is for production where the cluster’s topology is known.