Multiple schedulers and profiles — coexistence and opt-in

The default scheduler handles most workloads. For specialised cases (low-latency, GPU-priority, batch workloads), operators run additional scheduler instances with their own profiles. Pods opt in via schedulerName. This lesson walks the coexistence patterns and the production discipline of running multiple schedulers.

The model

flowchart LR
    subgraph schedulers["Scheduler instances"]
        SD["default-scheduler"]
        SL["low-latency-scheduler"]
        SB["batch-scheduler"]
    end
    AS["API server"] -->|"watch Pods"| schedulers
    Pod1["Pod default"] -->|sched by| SD
    Pod2["Pod low-latency"] -->|sched by| SL
    Pod3["Pod batch"] -->|sched by| SB

Each scheduler runs independently. Each has its own leader election, own configuration, own cache. The API server’s watch feed is shared; each scheduler watches the same Pod events.

Why multiple schedulers

Three reasons:

• Workload separation. A low-latency scheduler may prioritise pods that need fast placement; a batch scheduler may take its time but optimise for throughput.
• Custom plugins for specific workloads. A GPU scheduler with specialised plugins; a batch scheduler with gang scheduling.
• Operational isolation. A misbehaving scheduler (custom plugin causing latency) doesn’t affect the default’s workloads.

The configuration

A second scheduler is configured via a separate config file and a separate static pod:

# /etc/kubernetes/manifests/kube-scheduler-low-latency.yaml
apiVersion: v1
kind: Pod
metadata:
  name: kube-scheduler-low-latency
  namespace: kube-system
spec:
  containers:
  - name: kube-scheduler
    image: registry.k8s.io/kube-scheduler:v1.34.x
    command:
    - kube-scheduler
    - --config=/etc/kubernetes/kube-scheduler-low-latency.yaml
    - --leader-elect=true
    - --leader-elect-resource-name=low-latency-scheduler

The --leader-elect-resource-name is critical: it ensures the leader election lease is distinct from the default scheduler’s lease.

# /etc/kubernetes/kube-scheduler-low-latency.yaml
apiVersion: kubescheduler.config.k8s.io/v1beta2
kind: KubeSchedulerConfiguration
profiles:
- schedulerName: low-latency-scheduler
  plugins:
    score:
      enabled:
      - name: ImageLocality
        weight: 5
      - name: LeastAllocated
        weight: 1

The Pod opt-in

A Pod opts into the alternative scheduler via spec.schedulerName:

apiVersion: v1
kind: Pod
metadata:
  name: web-frontend
  namespace: prod
spec:
  schedulerName: low-latency-scheduler
  containers:
  - name: web
    image: nginx:1.27

The Pod is watched only by the scheduler whose schedulerName matches. The default scheduler ignores it.

The leader election

Multiple schedulers use Kubernetes’ lease objects for leader election:

kube-system/kube-scheduler-leader  (default scheduler)
kube-system/low-latency-scheduler-leader  (low-latency scheduler)

Only the leader processes scheduling cycles; the others wait. On leader failure, a new leader is elected.

# Check the leader
kubectl get leases -n kube-system

Profiles vs separate schedulers

Two approaches for specialised scheduling:

• Profiles inside one scheduler. Multiple profiles in one scheduler config; Pods opt into a profile via schedulerName. Single process, single set of leaders, shared cache.

# Same scheduler process; multiple profiles
schedulerNames: ["default-scheduler", "low-latency-scheduler"]

• Separate scheduler instances. Each scheduler is its own process with its own leader election. Independent.

# Two schedulers
default-scheduler (leader-elected)
low-latency-scheduler (leader-elected)

Profiles are simpler. Separate instances are more isolated.

The trade-offs

For 2-3 specialised workloads, profiles are sufficient. For 5+ workloads with very different scheduler profiles, separate instances are clearer.

The resources and capacity

A scheduler instance consumes:

• CPU: tens of milli-cores per Pod scheduled.
• Memory: a few hundred MB.
• Network: watch feed from API server.
• Cache: in-memory; scales with cluster size.

Multiple scheduler instances multiply the resources. The default scheduler on a busy cluster may use 1-2 vCPU; the second instance adds ~1 vCPU.

The common patterns

Low-latency scheduler

plugins:
  score:
    enabled:
    - name: ImageLocality
      weight: 5
    - name: LeastAllocated
      weight: 1
    - name: TopologySpread
      weight: 0
  preFilter:
    disabled:
    - name: TopologySpread

The low-latency scheduler uses ImageLocality to avoid image pulls and skips topology spread. Web-frontend Pods opt into this profile.

Batch scheduler

plugins:
  score:
    enabled:
    - name: MostAllocated
      weight: 5
    - name: NodeResourcesFit
      weight: 3
  preFilter:
    disabled:
    - name: ImageLocality

The batch scheduler uses MostAllocated to pack jobs densely and tolerates network latency. Jobs opt into this profile.

GPU scheduler

A custom scheduler with a custom plugin that knows the GPU types available on each node. ML training Pods opt in.

The operational discipline

• Document each scheduler in the runbook. Each scheduler’s name, configuration, and target workload.
• Monitor each scheduler separately. Prometheus metrics include the scheduler name as a label; alert on per-scheduler fail rates.
• Test scheduler upgrades. A scheduler upgrade can break a specialised profile; the upgrade plan must validate.
• One cache rebuild = one scheduler restarts. Plan for the cache rebuild time.

$ kubectl get leases -n kube-system | grep scheduler

kube-scheduler-leader                  ...  kube-scheduler-...

The “single scheduler” recommendation

For most production clusters, one scheduler (default) is correct. Multiple schedulers are for specialised workloads that cannot fit the default profile.

Add a second scheduler when:

• The default scheduler’s profile is incompatible with a specialised workload.
• The specialised workload has measurable cost in the default profile.
• The team is willing to operate the additional scheduler.

For most clusters, the answer is: stay with the default. Customisation comes through Pod-level features (affinity, tolerations, topology) and CRD-driven controllers, not through additional schedulers.

Quiz

Production discipline

• Default scheduler is the right answer for most clusters. Don’t add complexity you don’t need.
• When multiple, document each. Schedulers, profiles, target workloads in the runbook.
• Separate leases prevent conflict. --leader-elect-resource-name is critical.
• Monitor per-scheduler metrics. A metric label distinguishes the schedulers.
• Validate upgrades for each scheduler. Custom profiles can break with version changes.

Multiple schedulers are tools for specialised workloads. Operating them well is keeping them aligned with the workloads they target.