Scheduler architecture — informer, queue, scheduling cycle

The kube-scheduler is the component that decides which node a Pod runs on. It does so by filtering, scoring, reserving, and binding — a pipeline that runs once per Pod. This lesson walks the scheduler’s architecture, the path of a Pod through the cycle, and the metrics that operator should monitor.

The scheduler in one sentence

The kube-scheduler watches the API for Pods with spec.nodeName unset (unscheduled Pods), chooses a node through a filter-score cycle, and writes spec.nodeName back to the API server (the binding).

flowchart LR
    AS[API server] -->|watch Pods| IS[Informer cache]
    IS --> Q[Priority queue]
    Q --> SC[Scheduling cycle]
    SC -->|filter| F[Filter plugins]
    SC -->|score| S[Score plugins]
    SC -->|reserve| R[Reserve plugin]
    SC -->|permit| P[Permit plugins]
    SC -->|bind| B[Bind plugin]
    B -->|POST /api/v1/.../binding| AS
    AS -->|notify| K[kubelet on chosen node]
    K -->|start Pod| N[Node]

The scheduler’s input is the cluster state (Nodes, Pods, PVCs, Services). Its output is spec.nodeName for each Pod. The chosen node’s kubelet starts the Pod.

The informer cache and priority queue

The scheduler’s client watches the API server for two things:

• Pods with no spec.nodeName (the work queue).
• Nodes (the cache of node state).
• PVCs, Services, CSINodes (the cache of storage and network state).

flowchart LR
    AS[API server] -->|watch Pods| LI[Pod informer]
    AS -->|watch Nodes| NI[Node informer]
    AS -->|watch PVCs| PI[PVC informer]
    LI --> Q[Priority queue]
    Q --> SC[Scheduling cycle]
    NI --> SC
    PI --> SC

The Pods in the queue are sorted by priority and creation time. The scheduler processes them serially (with some parallelism; see below).

The scheduling cycle

For each Pod in the queue, the scheduler runs:

flowchart LR
    P[Pod from queue] --> F[Filter phase]
    F -->|FeasibleNodes| S[Score phase]
    S -->|RankedNodes| R[Reserve phase]
    R -->|ReservedNode| PM[Permit phase]
    PM -->|Approved| B[Bind phase]
    B -->|nodeName written| DONE[Bound]

The four phases:

1. Filter: which nodes can run this Pod?
2. Score: among feasible nodes, which is best?
3. Reserve: tentatively claim the resources for the Pod.
4. Permit: webhooks and framework plugins approve (or wait).
5. Bind: write spec.nodeName to the API.

Each phase can fail. A Pod that fails filter is “Pending” forever unless its constraints change.

Parallel scheduling

Modern schedulers (1.18+) run scheduling cycles in parallel across multiple Pods:

Scheduling threads: 8
Bindings processed in 1 second: ~150 Pods

The parallelism is controlled by the --parallelism flag; 8-16 is typical. The parallelism multiplies throughput but reduces per-pod latency.

The scheduler’s cache and snapshot

The scheduler maintains a snapshot of all cluster state:

• Each Node’s capacity and allocatable resources.
• Each Node’s labels, taints, conditions.
• Each Node’s existing Pods’ resource requests.
• Each Node’s CSI / network state (basic).

flowchart LR
    NC[Node cache] --> F[Filter reads NC]
    SC[Score reads NC] --> F2[Filter]
    NC --> S[Score plugins read NC]

The cache is updated via the watch feed from the API server. When a Pod is added or removed, or a Node’s capacity changes, the cache updates.

The bind step

After the cycle, the chosen node is “bound” to the Pod by writing spec.nodeName:

POST /api/v1/namespaces/<ns>/pods/<name>/binding
{
  "apiVersion": "v1",
  "kind": "Binding",
  "metadata": {"name": "..."},
  "target": {"apiVersion": "v1", "kind": "Node", "name": "cp-3"}
}

The API server validates and writes the binding. The kubelet on the chosen node observes the new spec.nodeName and starts the Pod.

Scheduler health metrics

The scheduler exposes Prometheus metrics:

The scheduler’s throughput (Pod decisions per second) and the cycle latency (time from queue to bind) are the core health signals.

$ kubectl get componentstatuses scheduler

scheduler  Healthy   ok

Scheduling throughput

A scheduler instance can typically bind dozens of Pods per second; in a busy cluster:

Average scheduling latency: 50-200 ms
Throughput: 100+ Pods/sec on modern hardware

A scheduler that drops below 10 Pods/sec has a problem. The diagnosis walks through:

• Cache freshness (watch feed OK?).
• Filter and score plugin latency (any custom plugin?).
• Permit wait timeout (a webhook waiting too long?).

Scheduling on restart

On restart, the scheduler’s informer cache is empty; it needs to re-list. During the warm-up:

• New Pods scheduled, but the cache is rebuilding.
• Existing Pods are reconciled post-cache-population.

The warm-up is typically a few seconds; the scheduler becomes fully functional within ~1 minute.

Scheduler failure modes

The reservation cycle

Modern schedulers use a reservation approach to avoid double-booking:

sequenceDiagram
    autonumber
    participant SC1 as Scheduler thread 1
    participant NC as Node cache
    participant SC2 as Scheduler thread 2
    SC1->>NC: filter (Pod A)
    NC-->>SC1: 5 feasible nodes
    SC1->>NC: reserve resources on node X
    SC2->>NC: filter (Pod B)
    NC-->>SC2: 4 feasible nodes (X excluded)
    SC1->>NC: bind Pod A on node X
    NC-->>SC1: success
    SC1->>NC: release reservation

The reservation mechanism prevents two parallel threads from racing on the same resource.

Quiz

Production discipline

• Monitor scheduling latency. A growing scheduler_pod_scheduling_duration_seconds p99 indicates a slow filter or score plugin.
• Watch queue depth. A growing scheduler_pending_pods indicates the scheduler is falling behind.
• Profile custom plugins. Custom scheduling plugins are the most common cause of scheduler regressions.
• Watch leader election. The scheduler’s leader changes should be rare; many changes indicate instability.
• HA across the control plane. Multiple scheduler instances with leader election ensure one failure does not stop scheduling.

The scheduler’s health is the cluster’s ability to place workloads. Operating it well is operating the cluster.