Filtering — feasibility across nodes

The filter phase of the scheduling cycle answers one question: which nodes can run this Pod? Multiple filter plugins run in sequence, each eliminating some nodes. The output is a subset of nodes (the “feasibleNodes”). If the set is empty, the Pod cannot be scheduled; it stays Pending with events explaining why. This lesson walks the filter plugins and how to interpret Pending Pods.

The filter concept

flowchart TB
    N[All nodes] -->|NodeAffinity| NA[subset]
    NA -->|NodeSelector| NS[subset]
    NS -->|Taints| T[subset]
    T -->|Resources| R[subset]
    R -->|Volumes| V[subset]
    V -->|...other| O[subset]
    O -->|feasibleNodes| F[Feasible set]

Each plugin reduces the set. The plugins run in configured order; the configured order can affect which plugin fires first to short-circuit for a failed plugin.

The default filter plugins

NodeName

A Pod with spec.nodeName=cp-3 is filtered to only that node. This is a hard filter; if spec.nodeName is set and doesn’t match a node, the Pod is unschedulable.

NodeAffinity

spec:
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
        - matchExpressions:
          - key: gpu
            operator: Exists

The Pod requires the label gpu on the node. Any node without gpu=true is filtered out.

Taints and Tolerations

spec:
  tolerations:
  - key: dedicated
    operator: Equal
    value: prod
    effect: NoSchedule

A node with dedicated=prod:NoSchedule is filtered out unless the Pod has the corresponding toleration.

PodAffinity / PodAntiAffinity

spec:
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        topologyKey: kubernetes.io/hostname
        labelSelector:
          matchLabels:
            role: db

The Pod requires (anti-affinity) or prefers (affinity) co-location with other Pods.

NodeResourcesFit

The plugin that checks whether the node has enough CPU, memory, ephemeral storage, and (in newer versions) custom resources to host the Pod.

Node has:
  cpu: 16
  memory: 64Gi
  ephemeral-storage: 1Ti

Pod requires:
  requests:
    cpu: 1
    memory: 2Gi

Result: feasible

If the Pod’s requests exceed the node’s allocatable, the node is filtered out.

NodePorts and HostPorts

If the Pod uses hostPort, only nodes with that port free can host it. If hostNetwork, the node must be in the appropriate network namespace.

NodeCondition / NodeUnschedulable

kubectl get node cp-1 -o jsonpath='{.spec.unschedulable}' | jq

If spec.unschedulable=true (cordon), the node is filtered out. If the node has Ready=False, the node may be filtered depending on configuration.

VolumeBinding / NodeAffinity for PVs

A Pod with PVCs that have nodeAffinity requirements is filtered against nodes matching the affinity. A Pod with unbound PVCs (no PV yet) is filtered against nodes that can host the storage backend.

PodTopologySpreadConstraints

A Pod with topologySpreadConstraints may be filtered out if no node satisfies the constraints.

VolumeRestrictions / NodeLabel

Various volume-related and label-related plugins that filter nodes based on Pod-specific requirements.

The filter evaluation order

A Pod’s filter pipeline typically runs:

1. NodeName (if specified, the set is 1 node).
2. NodeAffinity.
3. NodeSelector (legacy equivalent of NodeAffinity).
4. Taints tolerations.
5. NodeResourcesFit.
6. NodePorts.
7. PodAffinity / AntiAffinity.
8. VolumeBinding.
9. PodTopologySpreadConstraints.
10. VolumeRestrictions / CSI.
11. Custom plugins.

Each plugin’s input is the previous plugin’s feasible set. The output is the next plugin’s feasible set.

Reading a Pending Pod’s events

kubectl describe pod web -n prod

Events:
  Type     Reason            Age     From               Message
  ----     ------            ----    ----               -------
  Warning  FailedScheduling  5m13s   default-scheduler  0/3 nodes are available: 3 Insufficient cpu.

The message says “Insufficient cpu on all 3 nodes”. The Pod’s requests.cpu exceeds the cluster’s free CPU.

Warning  FailedScheduling  5m13s   default-scheduler  0/3 nodes are available: 1 node(s) had taint {dedicated=prod:NoSchedule}, that taint is not tolerated.

A node is excluded by a taint; the Pod needs a toleration.

$ kubectl describe pod web -n prod | grep -A 5 Events

...

Common Pending failures

Insufficient CPU / memory

0/3 nodes are available: 3 Insufficient cpu.

The Pod’s requests.cpu is too high. Reduce requests; add capacity; or reschedule.

Taint not tolerated

1 node(s) had taint {dedicated=prod:NoSchedule}, that taint is not tolerated.

The Pod has no toleration for the node’s taint. Add a toleration or remove the taint.

Node affinity mismatch

1 node(s) didn't match node affinity.

The Pod’s NodeAffinity rules don’t match the cluster’s node labels. Adjust the rules or the labels.

Insufficient ephemeral-storage

3 Insufficient ephemeral-storage.

The Pod uses ephemeral-storage requests and the nodes are under disk pressure.

PVC binding failure

0/3 nodes are available: 3 node(s) didn't find available persistent volumes.

The Pod’s PVCs cannot bind. The PV’s storage class, node affinity, or capacity doesn’t match.

Pod affinity constraint

0/3 nodes are available: 1 node(s) didn't match pod affinity rules, 2 node(s) had taint ...

A combination of constraints.

Custom filter plugins

Custom plugins extend the scheduler. A typical pattern:

// A scoring plugin for network locality
func (pl *MyPlugin) Filter(ctx context.Context, state *framework.CycleState, pod *v1.Pod, node *framework.NodeInfo) *framework.Status {
    // Check if the node has the right network locality
    if !pl.isLocalToZone(pod, node) {
        return framework.NewStatus(framework.Unschedulable, "not local to zone")
    }
    return nil
}

The scheduler framework supports custom Filter, Score, Reserve, Permit, Bind, PreBind, PostBind plugins.

The interaction with rescheduling

When a Pod cannot be scheduled:

• It stays Pending.
• The scheduler retries when:
  • A node is added (label matches the affinity).
  • The Pod’s spec is updated (request reduced).
  • The cluster’s resource pressure decreases.

The scheduler does not continuously retry the same Pod; it waits for a state change that may make scheduling feasible.

Inspecting filter reasoning

The scheduling framework exposes a verbosity flag (--v=4) that logs the filter reasoning for debugging:

Pod web/prod: FilterNodeName -> feasible
Pod web/prod: FilterNodeAffinity -> feasible
Pod web/prod: FilterTaintToleration -> feasible
Pod web/prod: FilterNodeResourcesFit -> node-1 infeasible (insufficient memory)
Pod web/prod: FilterNodeResourcesFit -> node-2 feasible

The verbose log is the diagnostic for “why is my Pod Pending”.

The preemption fallback

If filtering yields no feasible nodes, the scheduler can preempt lower-priority Pods to make room:

flowchart LR
    N[No feasible node] --> P[Preempt lower-priority]
    P -->|free resources| R[Reserve on previously infeasible node]

Preemption is governed by PriorityClass. A high-priority Pod can preempt lower-priority Pods to schedule.

Quiz

Production discipline

• Inspect filter events. A Pending Pod’s events identify every failed filter plugin.
• Reduce constraints where possible. A Pod that requires affinity and toleration and topology is fragile.
• Avoid exclusive nodes. Dedicated nodes via taints reduce cluster capacity.
• Test resource requests. A Pod with too-high requests consumes cluster capacity without serving traffic.
• Monitor scheduling fail rate. A spike in scheduler_attempts_total{result="unschedulable"} is a cluster capacity issue.

Filtering is the cycle phase that identifies which nodes can host a Pod. Operating it well is keeping Pods feasible.