Scheduler integration — how priority affects scheduling decisions

The scheduler uses priority to order the scheduling queue and to trigger preemption. This lesson walks the scheduler queue, the priority sorting, the preemption hooks, and the failure modes.

The scheduling queue

flowchart LR
    A[Pending Pods] --> B[Scheduler queue]
    B --> C["activeQ: ready"]
    B --> D["backoffQ: delayed retry"]
    B --> E["unschedulableQ: cannot schedule"]
    C --> F[Scheduler picks highest priority]
    F --> G{Node has room?}
    G -->|Yes| H[Bind]
    G -->|No| I[Preempt or wait]

The scheduler’s queues:

• activeQ. Pods ready to be scheduled. Sorted by priority (highest first).
• backoffQ. Pods that failed to schedule and are waiting to retry (e.g., due to a backoff period).
• unschedulableQ. Pods that cannot be scheduled (no node fits). Re-tried periodically.

The scheduler picks from activeQ first, ordered by priority. Higher priority Pods are scheduled first.

The priority sort

flowchart LR
    A["Pod A: priority 1000"] --> E[Front of queue]
    B["Pod B: priority 100"] --> F[Back of queue]
    C["Pod C: priority 10000"] --> G[Very front]
    G --> H[Scheduled first]
    E --> I[Scheduled second]
    F --> J[Scheduled third]

Higher priority Pods are at the front. When two Pods have the same priority, FIFO order applies.

The preemption hook

flowchart LR
    A[Scheduler picks Pod] --> B{Node has room?}
    B -->|Yes| C[Schedule normally]
    B -->|No| D{PreemptionPolicy?}
    D -->|PreemptLowerPriority| E[Find lower-priority Pods]
    D -->|Never| F[Pod stays Pending]
    E --> G{Found candidates?}
    G -->|Yes| H[Schedule + graceful evict]
    G -->|No| I[Pod stays Pending]

The preemption hook runs when:

1. The scheduler picks a Pod from activeQ.
2. No node has room.
3. The Pod’s preemptionPolicy allows preemption.
4. The scheduler searches for lower-priority Pods that can be evicted.

If candidates are found, the Pod is scheduled and the lower-priority Pods are evicted. If no candidates, the Pod stays Pending.

The failure modes

flowchart LR
    A[Failure modes] --> B[No candidates]
    A --> C[Infinite preemption loop]
    A --> D[Scheduler saturation]
    A --> E[PDB blocking]

The failure modes:

• No candidates. No lower-priority Pods can be evicted (PDBs too restrictive, no lower-priority Pods).
• Infinite preemption loop. Pod A preempts Pod B; Pod B’s replacement preempts Pod A. The cluster thrashes.
• Scheduler saturation. Many Pods pending; the scheduler cannot keep up.
• PDB blocking. All lower-priority Pods are protected by PDBs; preemption cannot proceed.

Monitoring the scheduler

kubectl get pods -A --field-selector=status.phase=Pending

NAMESPACE   NAME                READY   STATUS    RESTARTS   AGE
prod-app    myapp-xxxxx-yyyyy   0/1     Pending   0          5m
prod-app    myapp-xxxxx-zzzzz   0/1     Pending   0          5m

Pending Pods indicate scheduling failures. The diagnostic is kubectl describe pod to see the events (FailedScheduling, Preempted, etc.).

kube_pod_status_phase{phase="Pending"} > 10

Alert on more than 10 Pending Pods for more than 5 minutes.

Quiz

The operational discipline

Scheduler integration in production rests on five non-negotiable elements:

• Monitor the scheduling queue. Alert on Pending Pods.
• Investigate FailedScheduling events. They indicate capacity or preemption issues.
• Configure PDBs with headroom. Allow preemption.
• Test scheduler under pressure. Quarterly: simulate a cluster under pressure; verify behaviour.
• Document the priority scheme. The runbook lists priority classes and their semantics.

The scheduler’s priority integration is the operational signal for what matters. The discipline is to monitor the queue, investigate failures, and configure the system to allow preemption when needed.