Filter phase — predicates that eliminate impossible nodes

The scheduler’s filter phase is a set of predicates that eliminate nodes that cannot run the Pod. If all nodes are eliminated, the Pod is Pending with a FailedScheduling event. The event is the operator’s diagnostic; this lesson covers the standard filters, the events they produce, and the common production failure modes.

The filter pipeline

flowchart LR
    A[All nodes in cluster] --> B["Filter 1<br/>NodeUnschedulable"]
    B --> C["Filter 2<br/>NodeName / NodeSelector"]
    C --> D["Filter 3<br/>NodeAffinity (required)"]
    D --> E["Filter 4<br/>Taints"]
    E --> F["Filter 5<br/>PodFitsResources"]
    F --> G["Filter 6<br/>PodFitsHostPorts"]
    G --> H["Filter 7<br/>Volume binding"]
    H --> I["Filter 8<br/>Pod affinity (required)"]
    I --> J[Feasible nodes]

Each filter is binary: the node survives or is eliminated. The order is implementation-defined; all filters are run on all nodes.

Standard filters

NodeUnschedulable

flowchart TB
    A["Node spec.unschedulable=true"] -->|filter| B[Eliminated]
    C["Node spec.unschedulable=false"] -->|filter| D[Survives]

The node is cordoned (kubectl cordon or spec.unschedulable: true). New Pods are not scheduled. Existing Pods continue; the cordoning is for drain.

NodeName

spec:
  nodeName: node-01

The Pod is restricted to one node. If node-01 is not in the cluster or does not match other constraints, the Pod is unschedulable.

NodeSelector

spec:
  nodeSelector:
    disk: ssd
    tier: production

The Pod’s nodeSelector must match all labels on the node. Nodes without both labels are eliminated.

NodeAffinity (required)

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        - key: topology.kubernetes.io/zone
          operator: In
          values: ["us-east-1a"]

Hard constraint: the Pod must land on a node in zone us-east-1a. Nodes outside the zone are eliminated.

Taints

spec:
  tolerations:
  - key: dedicated
    operator: Equal
    value: gpu
    effect: NoSchedule

A node with taint dedicated=gpu:NoSchedule is eliminated unless the Pod tolerates it. Pods without the toleration cannot land on the tainted node.

PodFitsResources

flowchart TB
    A["Pod requests<br/>500m CPU, 1Gi memory"] --> B["Node has<br/>1000m CPU, 2Gi memory"]
    A --> C["Node has<br/>200m CPU, 4Gi memory"]
    B --> D["Survives<br/>500m ≤ 1000m<br/>1Gi ≤ 2Gi"]
    C --> E["Eliminated<br/>500m > 200m"]

The scheduler sums the Pod’s resources.requests against the node’s allocatable (which is the capacity minus the reserved). If the request fits, the node survives.

The scheduler does not consider the node’s actual utilisation; only the requests. A node with 1000m CPU allocatable but currently using 900m CPU is considered “has 1000m CPU” for scheduling purposes.

PodFitsHostPorts

spec:
  containers:
  - name: web
    ports:
    - hostPort: 8080

If hostPort: 8080 is already in use on the node, the Pod is eliminated. Most production Pods use containerPort without hostPort; this filter rarely fires.

Volume binding

flowchart TB
    A["Pod references<br/>PVC"] --> B{PVC bound?}
    B -->|no| C["Filter fails<br/>Pod Pending"]
    B -->|yes| D{PV zone matches<br/>node zone?}
    D -->|no| E["NoVolumeZoneConflict<br/>eliminated"]
    D -->|yes| F{Single-Pod RWO<br/>on different node?}
    F -->|yes| G["NoVolumeConflict<br/>eliminated"]
    F -->|no| H[Survives]

A Pod referencing a PVC with WaitForFirstConsumer binding mode is scheduled only after the PVC is bound. A Pod referencing a PV with a specific zone is eliminated on nodes outside the zone.

For a ReadWriteOnce PVC, only one Pod on one node can mount it. If the PVC is already bound to a Pod on a different node, the new Pod is eliminated.

Pod affinity / anti-affinity (required)

affinity:
  podAntiAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
    - topologyKey: kubernetes.io/hostname
      labelSelector:
        matchLabels:
          app: web

The Pod must not co-locate with a Pod labelled app: web. Nodes hosting such Pods are eliminated.

The FailedScheduling event

When all nodes are eliminated, the scheduler creates a FailedScheduling event:

kubectl describe pod web-7c8d9b1f8-abcd
# Events:
#   Type     Reason            Age    From               Message
#   ----     ------            ----   ----               -------
#   Warning  FailedScheduling  4m     default-scheduler  0/6 nodes are available: 1 Node(s) didn't match Pod's node affinity, 2 Node(s) didn't match Pod's node selector, 3 Insufficient cpu.

The message lists every filter that failed and how many nodes each filter eliminated.

flowchart TB
    A[FailedScheduling event] --> B[Read each filter's message]
    B --> C["Identify which filter<br/>eliminated all nodes"]
    C --> D{Fix}
    D -->|NodeAffinity| E[Adjust affinity rules]
    D -->|NodeSelector| F[Adjust selector]
    D -->|Taints| G[Add toleration]
    D -->|Resources| H["Reduce requests<br/>or add nodes"]
    D -->|Volume| I["Fix PV binding mode<br/>or zone"]

The most common production failures

1. NodeAffinity that no node matches

A Pod requires a node with topology.kubernetes.io/zone: us-east-1c, but the cluster has nodes only in us-east-1a and us-east-1b. The Pod is unschedulable.

Diagnostic: FailedScheduling: 0/6 nodes are available: 3 Node(s) didn't match Pod's node affinity.

Fix: correct the affinity rule, add nodes to the zone, or relax the constraint.

2. NodeSelector with no matches

A Pod’s nodeSelector: gpu: true but no nodes have the label. The Pod is unschedulable.

Diagnostic: FailedScheduling: 0/6 nodes are available: 6 Node(s) didn't match Pod's node selector.

Fix: add the label to the nodes, or remove the selector.

3. Insufficient resources

A Pod’s request is 32Gi memory but no node has that much allocatable. The Pod is unschedulable.

Diagnostic: FailedScheduling: 0/6 nodes are available: 6 Insufficient memory.

Fix: reduce the request, add a node with more memory, or add a capacity-scaled node pool.

4. Taints without tolerations

A Pod has no toleration for the cluster’s taint dedicated=ml:NoSchedule. Every node has the taint. The Pod is unschedulable.

Diagnostic: FailedScheduling: 0/6 nodes are available: 6 Node(s) had taints that the pod didn't tolerate.

Fix: add the toleration to the Pod, or untaint the nodes.

5. PVC Pending

A Pod’s PVC is Pending (no PV bound). The Pod waits for the PVC; the scheduler holds the Pod until the PVC is bound.

Diagnostic: kubectl get pvc shows Pending. The Pod’s events are quiet; the PVC’s events are not.

Fix: investigate the PVC’s binding (StorageClass, provisioner, access mode).

6. Image pull in progress

The kubelet cannot pull the image yet (registry unreachable, credentials missing). The Pod is in ContainerCreating, not Pending. The scheduler has bound the Pod; the kubelet cannot start it.

Diagnostic: kubectl describe pod shows ImagePullBackOff.

Fix: verify the image name, registry credentials, and network connectivity.

The hard-vs-soft distinction

required* filters eliminate nodes; preferred* do not.

A requiredDuringSchedulingIgnoredDuringExecution Pod’s node is checked at scheduling time, ignored at execution time. If the node’s labels change after scheduling, the Pod continues to run.

A requiredDuringSchedulingRequiredDuringExecution (rare) checks the constraint at execution time too. If the node’s labels change, the Pod is evicted. This is rarely used.

Quiz

Production discipline

• Read the FailedScheduling event. Every reason and every node count is in the event; the fix is one of the standard patterns.
• Use preferred for soft constraints. A preferredDuringScheduling rule does not block; the Pod schedules even if the preference is not satisfied.
• Validate nodeSelector and affinity in CI. A manifest with nodeSelector: tier: production should be checked against the cluster’s nodes before applying.
• Set resource requests correctly. A Pod with requests: 32Gi may not fit a small node; the operator must size requests to the cluster’s nodes.
• Audit Pending Pods. A dashboard that surfaces Pending Pods and their events catches scheduling failures before they become user-visible.

The filter phase is deterministic. Every FailedScheduling event has a reason; the operator’s job is to read the reason and apply the right fix.