Multiple schedulers and profiles — custom and workload-specific scheduling

The default kube-scheduler handles most scheduling needs. For workload-specific requirements (GPU, batch, latency), the cluster can run multiple schedulers — either as separate binaries or as multiple profiles within a single kube-scheduler. This lesson covers when each is the right answer and how to configure it.

Two ways to extend scheduling

flowchart TB
    A[Multiple schedulers needed?] --> B{Same binary,<br/>different config?}
    B -->|yes| C[Scheduler profiles]
    B -->|no, custom logic| D[Custom scheduler binary]
    C --> E["Single kube-scheduler<br/>multiple profiles"]
    D --> F["Multiple kube-scheduler<br/>deployments"]

Scheduler profiles (preferred)

A single kube-scheduler deployment can serve multiple profiles:

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
profiles:
- schedulerName: default
- schedulerName: gpu-scheduler
- schedulerName: batch-scheduler

A Pod with spec.schedulerName: gpu-scheduler is processed by the GPU profile. The profile can have different filter and scoring plugins.

Custom scheduler binary

A separate scheduler deployment (custom binary or custom-built kube-scheduler) handles specific Pods:

kube-scheduler \
  --scheduler-name=custom-scheduler \
  --config=/etc/kubernetes/custom-scheduler-config.yaml \
  --leader-elect=true \
  --leader-elect-resource-lock=endpointsleases \
  --lock-object-namespace=kube-system

The custom scheduler watches Pods with schedulerName: custom-scheduler and ignores others.

Scheduler profiles in detail

apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
profiles:
- schedulerName: default
  plugins:
    score:
      enabled:
      - name: LeastAllocated
        weight: 1
      - name: NodeAffinity
        weight: 2
- schedulerName: gpu-scheduler
  plugins:
    filter:
      enabled:
      - name: NodeAffinity
      - name: NodeName
      - name: NodeUnschedulable
      - name: NodeSelector
      - name: TaintToleration
      - name: PodFitsResources
    score:
      enabled:
      - name: LeastAllocated
        weight: 1
      - name: NodeAffinity
        weight: 5
      disabled:
      - name: InterPodAffinity

The gpu-scheduler profile:

• Weights NodeAffinity heavily (5x) — favours nodes with GPU labels.
• Disables InterPodAffinity — GPUs are scarce; no co-location preference.

The default profile uses the standard configuration.

When profiles are enough

The default configuration plus scheduler profiles handles:

• Workload-specific scoring (GPU nodes, latency-sensitive nodes).
• Custom filter order (e.g., check resource fits before affinity).
• Plugin selection (enable/disable specific plugins per profile).

If the workload’s needs can be expressed as a different combination of standard plugins, profiles are sufficient.

When a custom scheduler is needed

A custom scheduler binary is the right answer when:

• Custom scheduling logic (e.g., a scheduler that considers network topology or storage affinity in ways the standard plugins do not).
• External data sources (e.g., a scheduler that reads from a custom metrics service).
• Tight coupling with a workload (e.g., Volcano for batch workloads with gang scheduling).

The trade-off: operational complexity. A custom scheduler is a separate deployment with its own HA, monitoring, and upgrade path. Most production needs are met by profiles.

Leader election

Both the default kube-scheduler and any custom schedulers use leader election:

flowchart LR
    A[scheduler-0] -->|lease| B[etcd]
    C[scheduler-1] -->|lease| B
    B -->|current leader| D[API server]

Only one scheduler instance holds the lease and processes events. Other instances are passive. If the leader fails, another instance takes the lease within the lease duration (default 15s).

This allows running multiple replicas of a custom scheduler for HA without duplicate scheduling decisions.

Volcano for batch workloads

Volcano is a popular batch-scheduler for Kubernetes. It adds:

• Gang scheduling — all Pods of a job start together (or none).
• Queue management — fair-share queues with priorities.
• Preemption — high-priority jobs can preempt low-priority ones.

apiVersion: scheduling.volcano.sh/v1beta1
kind: Job
metadata:
  name: batch-job
spec:
  schedulerName: volcano
  tasks:
  - replicas: 8
    template:
      spec:
        containers:
        - name: worker
          image: worker:v1

Volcano’s scheduler handles the Job’s gang scheduling; the default kube-scheduler ignores it.

Descheduler

The opposite of multiple schedulers: the descheduler moves Pods that violate cluster policies. Common scenarios:

• A node becomes over-utilised (Pods’ actual usage exceeds requests); the descheduler evicts them.
• A node has a taint added; Pods that do not tolerate it are evicted.
• Pods co-located on the same node should be spread; the descheduler evicts and lets them reschedule.

apiVersion: descheduler/v1alpha2
kind: DeschedulerPolicy
strategies:
- name: RemoveDuplicates
  params:
    nodeDuplicated:
      threshold: 0.2
- name: LowNodeUtilization
  params:
    thresholds:
      cpu: 20
      memory: 20

The descheduler runs periodically; it evicts Pods that violate the policy; the controller (Deployment, StatefulSet) recreates them.

Production patterns

Pattern 1: GPU nodes

# Scheduler profile that prefers GPU nodes
apiVersion: kubescheduler.config.k8s.io/v1
kind: KubeSchedulerConfiguration
profiles:
- schedulerName: gpu-scheduler
  plugins:
    score:
      enabled:
      - name: LeastAllocated
        weight: 1
      - name: NodeAffinity
        weight: 5

# A GPU workload
spec:
  schedulerName: gpu-scheduler
  nodeSelector:
    gpu: true
  containers:
  - name: training
    resources:
      limits:
        nvidia.com/gpu: 1

The scheduler profile strongly prefers GPU nodes; the workload’s nodeSelector requires them.

Pattern 2: batch workloads with Volcano

apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  name: low-priority
spec:
  weight: 1
  reclaimable: true
  capability:
    cpu: "100"
    memory: "200Gi"

A reclaimable queue for low-priority batch. High-priority jobs can preempt; the cluster’s resources are shared.

Pattern 3: latency-sensitive workloads

plugins:
  score:
    enabled:
    - name: NodeAffinity
      weight: 5
    - name: LeastAllocated
      weight: 1

A scheduler profile that strongly weights affinity (latency- sensitive workloads must land on specific nodes) over load balancing.

Quiz

Production discipline

• Default scheduler + profiles is enough for most workloads. Reach for a custom scheduler only when the standard plugins cannot express the requirement.
• Document the scheduler choice. A Pod with schedulerName: gpu-scheduler must reference a profile or binary that exists; the runbook is incomplete otherwise.
• Run scheduler HA. A single scheduler instance is a single point of failure. Three replicas with leader election is the production standard.
• Monitor scheduler lag. The scheduler’s kube_scheduler_e2e_scheduling_duration_seconds metric tracks the time from Pod creation to bind; alert on outliers.
• Validate in CI. A Pod with schedulerName: gpu should be checked against the cluster’s profiles before applying.

Multiple schedulers are a powerful pattern but operational overhead. Operators who reach for them only when the default is insufficient have clusters that scale.