Scheduling framework — plugins, profiles, extensibility

The Kubernetes scheduling framework is the API through which the scheduler’s behaviour is customised. Production clusters rarely extend the framework directly (most use the default plugins), but understanding the framework helps diagnose custom-plugin issues and reason about scheduling profiles. This lesson walks the extension points and the profile mechanism.

The 12 extension points

flowchart LR
    PF[PreFilter] --> F[Filter]
    F --> POF[PostFilter]
    POF --> PS[PreScore]
    PS --> S[Score]
    S --> NS[NormaliseScore]
    NS --> R[Reserve]
    R --> PM[Permit]
    PM --> PB[PreBind]
    PB --> B[Bind]
    B --> PB2[PostBind]
    PB2 --> U[Unreserve]

Each extension point runs at a specific phase of the cycle. A plugin can implement one or more.

A plugin is a Plugin interface in k8s.io/kubernetes/pkg/scheduler/framework.

The plugin interface

A plugin implements the framework’s interfaces:

type Plugin interface {
    Name() string
}

type FilterPlugin interface {
    Plugin
    Filter(ctx context.Context, state *CycleState, pod *v1.Pod, node *NodeInfo) *Status
}

type ScorePlugin interface {
    Plugin
    Score(ctx context.Context, state *CycleState, pod *v1.Pod, nodeName string) (int64, *Status)
    ScoreExtensions() ScoreExtensions
}

type BindPlugin interface {
    Plugin
    Bind(ctx context.Context, state *CycleState, pod *v1.Pod, nodeName string) *Status
}

A single plugin can implement multiple interfaces (e.g., Filter and Score).

The PreFilter pattern

A PreFilter plugin can do work before the filter phase, like computing a Pod summary that filters don’t have to recompute:

func (pl *MyPlugin) PreFilter(ctx context.Context, state *framework.CycleState, pod *v1.Pod) (*framework.PreFilterResult, *framework.Status, error) {
    // Compute the Pod's "compatibility fingerprint"
    summary := computeFingerprint(pod)
    state.Write("summary", summary)
    return nil, nil, nil
}

The summary is then available to Filter and Score plugins.

The PostFilter pattern

The PostFilter runs when no feasible nodes remain. It can trigger preemption:

func (pl *MyPlugin) PostFilter(ctx context.Context, state *framework.CycleState, pod *v1.Pod, filteredNodesStatus []framework.NodeStatus) (*framework.PostFilterResult, *framework.Status, error) {
    // Find lower-priority Pods to evict
    victims := pl.findVictims(pod, filteredNodesStatus)
    if len(victims) == 0 {
        return nil, framework.NewStatus(framework.Unschedulable), nil
    }
    return &framework.PostFilterResult{Victims: victims}, nil
}

Preemption is a feature of the framework itself; custom PostFilter plugins can do project-specific victim selection.

The ScoreExtensions interface

Custom Score plugins can implement ScoreExtensions:

type ScoreExtensions interface {
    NormalizeScore(ctx context.Context, state *CycleState, p *v1.Pod, scores NodeScoreList) *Status
}

NormaliseScore modifies the raw scores to ensure consistency across plugins (e.g., to handle inconsistent-weight issues).

The Permit and WaitForFirstConsumer

The Permit interface is special:

type PermitPlugin interface {
    Plugin
    Permit(ctx context.Context, state *CycleState, pod *v1.Pod, nodeName string) (*Status, time.Duration)
}

type WaitPlugin interface {
    PermitPlugin
    Reserve(ctx context.Context, state *CycleState, pod *v1.Pod, nodeName string) *Status
    Unreserve(ctx context.Context, state *CycleState, pod *v1.Pod, nodeName string)
    Allow(ctx context.Context, state *CycleState, pod *v1.Pod, nodeName string) *Status
}

A WaitPlugin implementation can hold the binding indefinitely until it explicitly allows.

The VolumeBinding plugin in the standard scheduler is a WaitPlugin: it waits for the CSI provisioner to create the volume on the chosen node, then allows the bind.

Profiles

A profile bundles a set of plugin configurations:

apiVersion: kubescheduler.config.k8s.io/v1beta2
kind: KubeSchedulerConfiguration
profiles:
- schedulerName: default-scheduler
  plugins:
    score:
      enabled:
      - name: LeastAllocated
        weight: 1
      - name: TopologySpread
        weight: 2
    preFilter:
      disabled:
      - name: NodeAffinity
- schedulerName: low-latency-scheduler
  plugins:
    score:
      enabled:
      - name: ImageLocality
        weight: 5
      - name: LeastAllocated
        weight: 1

A Pod opts into a profile via spec.schedulerName:

spec:
  schedulerName: low-latency-scheduler

The default scheduler runs every Pod; the custom profile runs only Pods that explicitly opt in.

Common custom plugin patterns

Each is a plugin implementing some subset of the framework’s extension points.

The example: a network-locality plugin

A scheduler plugin that scores nodes by network locality to the Pod’s owner:

type NetworkLocality struct {
    handle framework.Handle
}

func (pl *NetworkLocality) Name() string {
    return "NetworkLocality"
}

func (pl *NetworkLocality) Score(ctx context.Context, state *framework.CycleState, pod *v1.Pod, nodeName string) (int64, *framework.Status) {
    nodeLocality := pl.getNodeLocality(nodeName)
    podLocality := pl.getPodLocality(pod)
    if nodeLocality == podLocality {
        return 100, nil  // same zone: high score
    }
    if nodeLocality.IsAdjacent(podLocality) {
        return 50, nil   // adjacent zone: medium
    }
    return 0, nil  // far zone: low
}

A Pod’s network locality might come from its labels or annotations.

The discipline of custom plugins

The performance profile

A modern scheduler can sustain:

• ~150 Pods/sec scheduling throughput.
• Filter plugin duration < 1 ms.
• Score plugin duration < 5 ms.
• Permit phase < 30 seconds (typically < 1 second).
• Bind phase < 100 ms.

A custom plugin that exceeds these budgets degrades the scheduler.

The configuration file

The scheduler is configured via /etc/kubernetes/kube-scheduler.yaml or as args to the kube-scheduler binary:

apiVersion: kubescheduler.config.k8s.io/v1beta2
kind: KubeSchedulerConfiguration
leaderElection:
  leaderElect: true
profiles:
- schedulerName: default-scheduler
  plugins:
    score:
      enabled:
      - name: TopologySpread
        weight: 3

For a kubeadm cluster, the configuration is set via the kube-system/kubeadm-config ConfigMap’s SchedulerConfiguration section.

The plugin lifecycle

sequenceDiagram
    autonumber
    participant Pod
    participant Sched as Scheduler
    participant F1 as Filter plugin A
    participant F2 as Filter plugin B
    participant S1 as Score plugin X
    participant S2 as Score plugin Y
    Pod->>Sched: queue
    Sched->>F1: Filter(pod, node)
    F1-->>Sched: status
    Sched->>F2: Filter(pod, node)
    F2-->>Sched: status
    Sched->>S1: Score(pod, node)
    S1-->>Sched: 60
    Sched->>S2: Score(pod, node)
    S2-->>Sched: 80
    Sched->>Sched: total = weighted sum
    Sched->>Pod: bind

Quiz

Production discipline

• Default plugins are the right answer for most clusters. Custom extension is the exception, not the rule.
• Custom plugins need profiling. A 1 ms filter over 100 nodes × 1000 Pods/sec is 100 ms/cycle; a 100 ms filter is 100 seconds/cycle.
• Reserve, Permit, Bind have specific failure modes. Each is documented; understand them before writing a custom plugin.
• Profiles bundle configurations. A Pod opts in via schedulerName; don’t break the default profile for custom behaviour.
• Test at production scale. A custom plugin should be tested at the cluster’s actual scheduling rate, not at low load.

The framework is the API through which scheduling behaviour is customised. Operating it well is keeping the default profile healthy and adding custom only when needed.