The scheduler — assigning Pods to nodes

The Kubernetes scheduler (kube-scheduler) is the component that assigns Pods to nodes. Its input is “an unscheduled Pod and a set of candidate nodes”; its output is spec.nodeName set on the Pod. This lesson walks the scheduler’s pipeline and the operational patterns that arise from it.

The scheduler in one sentence

The scheduler picks one node for each unscheduled Pod by filtering nodes that cannot run it and scoring the ones that can, then binds the Pod to the chosen node.

flowchart LR
    P[Unscheduled Pod] --> Q[Queue sort]
    Q --> F[Filter plugins<br/>feasible nodes?]
    F -->|feasible set| S[Score plugins<br/>rank nodes]
    S -->|top score| R[Reserve]
    R --> B[Bind<br/>set spec.nodeName]
    B --> W[kubelet picks up Pod]

The scheduler is a control loop. It runs continuously, watching for unscheduled Pods and binding them. It does not start Pods; that is kubelet’s job.

The scheduling framework

Since Kubernetes 1.17, the scheduler is built on the scheduling framework: a set of extension points where plugins implement specific behaviour. The framework’s flow:

sequenceDiagram
    autonumber
    participant Q as QueueSort
    participant F as Filter
    participant P as PreFilter
    participant Post as PostFilter
    participant Sc as Score
    participant R as Reserve
    participant Pe as Permit
    participant Pr as PreBind
    participant B as Bind

    Q->>P: pre-filter (early exit?)
    P->>F: filter (which nodes can run this Pod?)
    F->>Post: if no feasible nodes, run PostFilter (preemption?)
    Post-->>F: return feasible set
    F->>Sc: score (rank the feasible nodes)
    Sc->>R: reserve (claim the chosen node)
    R->>Pe: permit (wait or reject before binding)
    Pe->>Pr: pre-bind (final checks)
    Pr->>B: bind (write spec.nodeName)

Built-in plugins implement each extension point. The most commonly engaged ones:

Filter, score, reserve, bind

Filter

The filter stage asks each candidate node: can this Pod run here? The answer is binary: feasible or not. Common reasons a node is filtered out:

• Insufficient CPU/memory (requests exceed node allocatable)
• Taints the Pod does not tolerate
• Node selector mismatch
• Required port already bound
• PVC cannot be bound to this zone (when WaitForFirstConsumer)
• Node is cordoned (spec.unschedulable: true)
• Required affinity rule cannot be satisfied
• Pod affinity rule prevents scheduling

If zero nodes are feasible, the Pod remains Pending and the scheduler records an event on the Pod explaining why.

Score

The score stage ranks the feasible nodes. Each scoring plugin contributes a number; the scheduler sums them (weighted by the plugin’s weight). The top-scoring node wins. Common score contributions:

• NodeResourcesFit: prefers nodes with the most free resources after the Pod is scheduled.
• ImageLocality: prefers nodes that already have the Pod’s image cached.
• InterPodAffinity: prefers nodes where the Pod’s affinity is satisfied.
• DefaultPodTopologySpread: prefers nodes that produce a balanced topology spread.
• NodeAffinity (preferred): soft preferences.

Reserve

The reserve stage records the chosen node as “tentatively allocated” to the Pod. If the Pod fails to bind later, the reservation is released.

Bind

The bind stage writes spec.nodeName onto the Pod’s spec and persists to etcd via the API server. The kubelet on that node will see the Pod in its watch and start it.

Scheduling profiles

The scheduler supports multiple profiles: named configurations of which plugins are enabled. Different Pods can be scheduled by different profiles via spec.schedulingGates or by the extension points that route to a profile.

In production, the default profile is usually sufficient. Custom profiles are useful for:

• Batch workloads with relaxed affinity rules
• Latency-sensitive workloads with strict placement
• GPU workloads with node selectors and resource matching

Preemption

When no node can run a high-priority Pod, the scheduler can preempt: evict lower-priority Pods to make room. The preempted Pods are given a grace period to terminate gracefully before being killed.

sequenceDiagram
    autonumber
    participant S as Scheduler
    participant API as API server
    participant P1 as Pod A (priority 100)
    participant P2 as Pod B (priority 10)

    S->>API: GET unscheduled Pod X (priority 1000)
    Note over S: No node has room for Pod X
    S->>API: Find a node where evicting lower-priority Pods frees enough resources
    S->>API: Choose Pod A and Pod B for eviction
    API-->>P1: graceful termination (30s default)
    API-->>P2: graceful termination (30s default)
    S->>API: Bind Pod X to the freed node

Preemption is disruptive by design. Production use of priority classes must be careful:

• Set priority classes deliberately; misconfigured priorities can cause high-priority Pods to preempt critical workloads.
• Bind priority to workload criticality, not business importance alone.
• Audit preemption events; sustained preemption is a signal that capacity is mis-sized.

Why Pods stay Pending

A Pending Pod is a Pod that the scheduler could not bind. The Pod event tells you why. Common reasons:

# Substitute your own value before running - the Pending Pod:
POD=web-7c8d9f4b5-qr2mn

kubectl describe pod "$POD" | grep -A 20 "Events:"

Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               -------
  Warning  FailedScheduling  4m    default-scheduler  0/5 nodes are available:
                                          3 Insufficient memory,
                                          2 node(s) didn't match Pod's node affinity.

A “0/N nodes are available” line lists every filter that rejected every node. The most common patterns:

• All nodes resource-exhausted: scale the cluster or reduce requests
• All nodes tainted: add tolerations to the Pod or remove the taints
• All nodes cordoned: uncordon or wait for maintenance
• Affinity unsatisfiable: relax the affinity rule or provision matching nodes
• PVC zone mismatch: provision storage in the right zone
• Port conflict on every node: change the hostPort or the node selection

The kube-scheduler component metrics expose scheduling latency:

• scheduler_queue_incoming_pods_total — Pods entering the queue
• scheduler_pending_pods — queue length
• scheduler_schedule_attempts_total — attempts (one filter cycle = one attempt)
• scheduler_e2e_scheduling_duration_seconds — end-to-end scheduling latency

How kubelet picks up where the scheduler left off

Once the scheduler binds a Pod, kubelet takes over:

sequenceDiagram
    autonumber
    participant API as API server
    participant K as kubelet (worker)
    participant R as Runtime

    API-->>K: watch: Pod ADDED on this node (spec.nodeName = me)
    K->>K: Validate (admission policy, security context)
    K->>R: CRI PullImage (if not cached)
    K->>K: Setup volumes (CSI)
    K->>K: Setup network (CNI)
    K->>R: CRI CreateContainer (with cgroup limits)
    R->>R: start process
    K->>API: PATCH /pods/<name>/status (Running)

The scheduler does not coordinate with kubelet; it writes the Pod’s spec.nodeName and walks away. kubelet is responsible for all of the worker-side Pod creation.

How to inspect the scheduler

# Substitute your own value before running:
POD=web-7c8d9f4b5-qr2mn

# What is the scheduler doing right now?
kubectl get events -A --field-selector reason=FailedScheduling

# What profiles are configured?
kubectl get schedulernameconfig
# (in 1.34; earlier versions expose via /config)

# Which scheduler is bound to which Pod?
kubectl get pod "$POD" -o jsonpath='{.spec.schedulerName}'

# On a control-plane node, scheduler logs
journalctl -u kube-scheduler --since "10 min ago"

I0815 12:01:01.234 scheduler.go:...] pod web-7c8 added to queue
I0815 12:01:01.235 scheduler.go:...] pod web-7c8 filtered out on node worker-01: insufficient memory
I0815 12:01:01.235 scheduler.go:...] pod web-7c8 filtered out on node worker-02: taint NoSchedule not tolerated
I0815 12:01:01.236 scheduler.go:...] pod web-7c8 bound to node worker-04

Cross-course references

• The Linux course part VI-Linux-Processes covers the process scheduling primitives; the Kubernetes scheduler sits on top of them.
• The Linux course part XIX-Linux-NetFoundations covers the network primitives the scheduler depends on (topology, zones).
• The Observability course part IX-Observability-Exporters covers the kube-state-metrics surface that includes scheduler-pending pod counts.
• The Linux course part LX-Linux-DistributedStorage covers the storage topology that the scheduler respects via PVC zone constraints.

Quiz

Production discipline

• Treat spec.nodeName as the only output of the scheduler; everything else is internal mechanism.
• Read the scheduler events on every Pending Pod — they name the filter that rejected every node.
• Size cluster capacity to scheduled workload, not to peak request. Requests are upper bounds; usage is lower. But scheduler only sees requests.
• Define a priority class taxonomy deliberately. A team that has not defined priorities has implicitly given every Pod default priority (0).
• Monitor scheduler latency (scheduler_e2e_scheduling_ duration_seconds) and queue depth. A growing queue is the early signal of capacity or filter-rule pressure.