The scheduler pipeline — observe, filter, score, bind

The kube-scheduler is the Kubernetes component responsible for selecting which node a Pod runs on. It runs a two-phase pipeline: filter (eliminate nodes that cannot run the Pod) and score (rank the remaining nodes). The chosen node is bound to the Pod. This lesson covers the pipeline, the components, and where custom schedulers fit.

The scheduler component

flowchart LR
    A[Pod created] --> B[API server]
    B --> C["Scheduler<br/>watches for unscheduled Pods"]
    C --> D[Filter phase]
    D --> E[Score phase]
    E --> F[Reserve phase]
    F --> G[Bind Pod to Node]
    G --> H["Kubelet on chosen node<br/>starts Pod"]

The kube-scheduler is a controller that watches for Pods with spec.nodeName empty. For each unscheduled Pod, the scheduler:

1. Reads the Pod spec, the cluster’s nodes, and the cluster’s state (PVs, Services, etc.).
2. Runs the filter phase: a set of predicates that eliminate nodes that cannot run the Pod.
3. Runs the score phase: a set of priorities that rank the remaining nodes.
4. Reserves the chosen node (so concurrent schedulers do not double-book).
5. Binds the Pod to the node via the API server.

The filter phase

The filter phase eliminates nodes that cannot run the Pod. Common predicates:

flowchart LR
    A[All nodes] --> B[Filter]
    B --> C{Node matches<br/>Pod constraints?}
    C -->|yes| D[Survives]
    C -->|no| E[Eliminated]
    D --> F[Score phase]
    E --> G[Not scheduled here]

After filtering, only feasible nodes remain. If no nodes remain, the Pod is Pending with a FailedScheduling event listing the predicates that eliminated all candidates.

The score phase

The score phase ranks the remaining nodes. Common scoring rules:

flowchart TB
    A["Feasible nodes<br/>after filter"] --> B[Score phase]
    B --> C["Each rule produces<br/>a score 0-100"]
    C --> D[Weighted sum]
    D --> E["Highest-scoring<br/>node chosen"]

The scheduler sums weighted scores and picks the highest. With multiple ties, the scheduler may pick any of them (non-deterministic) or use additional tiebreakers (PodHostName, NodeName lexicographic).

The bind step

flowchart LR
    A[Scheduler chooses node] --> B[Reserve phase]
    B --> C["Create Binding object<br/>in API server"]
    C --> D["API server updates<br/>Pod's nodeName"]
    D --> E["Kubelet on chosen node<br/>observes Pod"]
    E --> F[Kubelet starts Pod]

The scheduler creates a Binding object (a special API object) that sets the Pod’s spec.nodeName. The API server updates the Pod; the kubelet on the chosen node observes the Pod and starts it.

The bind step is the bridge between scheduling and execution. Before bind, the Pod is Pending; after bind, the Pod is Scheduled (still Pending until the kubelet starts the container).

Custom schedulers and scheduler profiles

A cluster can have multiple schedulers. A Pod specifies which scheduler via spec.schedulerName:

spec:
  schedulerName: my-custom-scheduler

The kube-scheduler ignores the Pod; my-custom-scheduler processes it.

flowchart LR
    A["Pod with schedulerName: default"] --> B[Default scheduler]
    A2["Pod with schedulerName: custom"] --> C[Custom scheduler]
    B --> D[Node X]
    C --> E[Node Y]

Scheduler profiles (Kubernetes 1.18+)

A single kube-scheduler can run multiple profiles:

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
profiles:
- schedulerName: default
  plugins:
    score:
      enabled:
      - name: LeastAllocated
- schedulerName: gpu-scheduler
  plugins:
    filter:
      enabled:
      - name: NodeAffinity

Different profiles implement different scoring rules. A Pod with schedulerName: gpu-scheduler is filtered by the GPU profile; a Pod with schedulerName: default uses the default profile.

The scheduler cache

The scheduler maintains an in-memory cache of cluster state: nodes, Pods, PVs, Services. The cache is updated by watch events from the API server. Scheduling reads from the cache, not the API server, to keep latency low.

flowchart LR
    A[API server] -->|watch events| B[Scheduler cache]
    B --> C[Filter phase]
    B --> D[Score phase]
    C --> E[Bind]
    D --> E

The cache is eventually consistent. A scheduler decision may be based on slightly stale state; if the state changes between filter and bind, the bind may fail (e.g., the node was deleted). The scheduler retries.

Where scheduling goes wrong

flowchart TB
    A[Pod Pending] --> B["Filter eliminates<br/>all nodes"]
    A --> C["Score produces<br/>no winner"]
    A --> D[Reserve fails]
    A --> E[Bind fails]
    B --> F[FailedScheduling event]
    C --> F
    D --> F
    E --> F

A FailedScheduling event is the scheduler’s signal that the Pod cannot be scheduled. The event lists the predicates that failed; the operator reads the event to understand the constraint.

Inspecting the scheduler

# Recent scheduling events
kubectl get events -A --field-selector reason=FailedScheduling
# Warning  FailedScheduling  4m  default-scheduler  0/6 nodes are available: insufficient cpu, 2 nodes are unschedulable.

# Pod's nodeName (set by scheduler)
kubectl get pod web-7c8d9b1f8-abcd -o jsonpath='{.spec.nodeName}'
# node-01

$ kubectl get pods -A -o custom-columns=NAME:.metadata.name,NODE:.spec.nodeName,SCHEDULER:.spec.schedulerName

NAME                  NODE       SCHEDULER
web-7c8d9b1f8-abcd    node-01    default-scheduler
...

Quiz

Production discipline

• Treat Pending Pods as a signal. A Pending Pod is the scheduler’s way of saying “I cannot schedule this.” Read the events; do not delete the Pod and re-apply.
• Custom schedulers are rare. Most production needs are met by nodeSelector, affinity, taints, and topology spread. Custom schedulers add operational complexity; reach for them only when the standard tools are insufficient.
• Scheduler profiles are the multi-tenant pattern. Different profiles for different workload classes (GPU, batch, latency-sensitive) on the same kube-scheduler.
• Audit scheduler decisions. A dashboard of kube_pod_status_phase{phase="Pending"} and the FailedScheduling events is the alert source for scheduling failures.

The scheduler is a two-phase pipeline with a binding step. Operators who understand the pipeline have Pods that schedule correctly; operators who do not have Pending Pod tickets that are hard to diagnose.