Scheduling gates, profiles, and the extended scheduler

Beyond the built-in filter and scoring plugins, the scheduler has three advanced mechanisms: PodSchedulingGates that hold a Pod until released, scheduling profiles that configure the scheduler per workload, and scheduler extenders that delegate decisions to an external service. This lesson walks all three and the operational discipline that prevents them from becoming silent blockers.

PodSchedulingGates

A Pod can be created with spec.schedulingGates[]. A gate is a name; the Pod is held by the scheduler until every gate is removed.

apiVersion: v1
kind: Pod
metadata:
  name: gated
spec:
  schedulingGates:
    - name: example.com/gate-1
  containers:
    - name: app
      image: registry.example.com/app:1.0.0

The Pod is created with the gate set. The scheduler adds the Pod to the scheduling queue but does not consider it for binding. The Pod’s Status is Pending with no FailedScheduling event.

The gate is removed by patching the Pod:

kubectl patch pod gated --type=json -p='[{"op": "remove", "path": "/spec/schedulingGates"}]'

The scheduler picks up the change and begins the filter cycle.

The use cases:

• Templating controllers. A controller that creates a Pod before all its dependencies are ready (a config issuer, a database migrator) can hold the Pod until the dependencies are available.
• External approval. A Pod that requires a human approval before scheduling (a workload that needs a change ticket) can be gated until the approval is recorded.
• Coordination. A Pod that is part of a multi-Pod workflow (a leader-follower config) can be gated until the leader is ready.

The diagnostic for a gate:

# Substitute your own Pod name before running:
POD=migrator-6c8d7f4b59-tq2vl

kubectl get pod "$POD" -o jsonpath='{.spec.schedulingGates}' | jq

[
  { "name": "example.com/gate-1" }
]

The Pod is Pending because the gate is set. The fix is to remove the gate.

Scheduling profiles

A scheduling profile is a set of plugins configured for a specific workload shape. The scheduler can run multiple profiles simultaneously; a Pod selects a profile by name via spec.schedulerName.

spec:
  schedulerName: low-latency

The scheduler’s configuration (KubeSchedulerConfiguration) declares the profiles:

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
profiles:
  - schedulerName: default-scheduler
    plugins:
      score:
        enabled:
          - name: NodeResourcesFit
          - name: NodeAffinity
  - schedulerName: low-latency
    plugins:
      score:
        enabled:
          - name: NodeResourcesFit
          - name: NodeAffinity
        disabled:
          - name: NodeResourcesBalancedAllocation

The low-latency profile disables the NodeResourcesBalancedAllocation scoring plugin (which penalises nodes with imbalanced CPU/memory) and replaces it with a custom scoret that prefers nodes with the workload’s preferred ratio.

Profiles are powerful because they let the operator configure different scoring strategies for different workloads. The cost is operational complexity: the scheduler runs multiple profiles, each with its own queue, and the operator must understand which workloads use which profile.

Scheduler extenders

A scheduler extender is an external HTTP service that the scheduler calls to make a scheduling decision. The extender can:

• Filter nodes (reject some that the built-in filters would accept).
• Score nodes (modify the built-in score).
• Preempt (override the built-in preemption decision).
• Bind the Pod (override the built-in binding).

The typical use case is a workload that needs to place Pods based on external state:

• A batch scheduler that places Pods based on a queue system (SGE, Slurm).
• A security scanner that places Pods based on a threat model.
• A custom hardware allocator that places Pods based on hardware availability.

The scheduler configuration declares the extender:

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
extenders:
  - urlPrefix: https://extender.example.com/scheduler
    filterVerb: predicate
    scoreVerb: prioritize
    enableHTTPS: true
    nodeCacheCapable: true
    weight: 1
    managedResources:
      - name: example.com/special-resource
        ignoredByScheduler: true

The scheduler calls the extender for every filter and score cycle. The extenders run in parallel with the built-in plugins; the result is the intersection.

The cost of extenders:

• Latency. A scheduler that calls an external service for every Pod pays the network round-trip. The scheduler’s scheduling_e2e_scheduling_duration_seconds rises.
• Single point of failure. An extender that is down blocks every Pod. The scheduler logs failed to call extender and the Pod is Pending.
• Debugging complexity. A Pod that is rejected by an extender has no event in the cluster’s events; the extender logs are the only signal.

Diagnostic workflow for advanced scheduling failures

A Pod that is Pending with no FailedScheduling event is the most common sign of an advanced scheduling failure.

flowchart TD
    A[Pod Pending] --> B{FailedScheduling event?}
    B -->|Yes| C[Filter rejection:<br/>resources, affinity, taint, pvc]
    B -->|No| D{Pod has schedulingGates?}
    D -->|Yes| E[Pod is gated;<br/>remove the gate]
    D -->|No| F{Pod has schedulerName?}
    F -->|Yes| G[Pod uses a profile;<br/>check profile config]
    F -->|No| H[Check pod-scheduling-readiness]
    H --> I{Readiness gates set?}
    I -->|Yes| J[Pod is gated on readiness];
    I -->|No| K[Check scheduler logs]
    K --> L{Extender calls failing?}
    L -->|Yes| M[Extender is down;<br/>check external service]
    L -->|No| N[Check scheduler metrics]

The diagnostic moves:

1. Run kubectl get pod -o yaml. Look for schedulingGates, schedulerName, and preemptionPolicy.
2. Run kubectl describe pod. Look for events.
3. Run kubectl get events. Look for scheduling events.
4. Run kubectl logs -n kube-system kube-scheduler-<name>. Look for extender failures and scheduling cycle errors.
5. Run kubectl get --raw /metrics | grep scheduler_preemption. Look at the scheduler’s metrics.

The pod-scheduling-readiness gate

The PodSchedulingReadiness field is a newer alternative to schedulingGates. It uses a label selector to gate the Pod:

spec:
  schedulingGates:
    - name: example.com/gate-1

The gate is removed by patching the Pod. The pod-scheduling-readiness field is a controller-side mechanism: the controller that creates the Pod sets the gate via the field, and the controller removes the gate when ready.

The diagnostic is the same: the Pod is Pending with no event; the gate is set in the spec.

Quiz

Production discipline

• Use scheduling gates sparingly. A gate that is not removed by the owning controller is a Pod that will never schedule. The controller must be reliable; the CI must validate the gate’s removal.
• Document the scheduler profiles. A cluster with multiple profiles must have a registry of which workloads use which profile. The audit at every release is the only way to keep this in sync.
• Audit extenders at every release. A scheduler extender that fails silently is a cluster that schedules Pods but lands them on the wrong node. Alert on scheduler_extender_* metrics.
• Watch the scheduler’s metrics. The scheduler’s pending_pods, scheduling_attempts_total, and e2e_scheduling_duration_seconds are the primary signals. A rising pending_pods is a cluster with scheduling failures.
• Test the scheduler in non-production. The scheduler’s behaviour is hard to test in CI. A staging cluster that mirrors production is the right place.