Scheduler backlogs — the cluster's placement engine

When the scheduler stops binding Pods, nothing crashes. Nodes stay Ready, running Pods keep serving, and the only symptom is a growing list of Pods stuck in Pending — which looks identical whether the scheduler process is dead, has lost leader election, or is running perfectly and rejecting every node on a filter. This lesson separates those cases using the pending Pods’ FailedScheduling events and the scheduler’s own log.

The scheduler’s role

The scheduler watches for Pods that are not yet scheduled (i.e., spec.nodeName is empty) and assigns them to nodes based on the filters and scoring.

flowchart LR
    A[Pod] --> B{Unscheduled?}
    B -->|Yes| C[Filter]
    C --> D[Score]
    D --> E[Bind]
    B -->|No| F[Skip]

The scheduler is the cluster’s placement engine.

The diagnostic

The canonical diagnostic:

# 1. Check the pending Pods
kubectl get pods -A --field-selector=status.phase=Pending

# 2. Check the scheduler's health
kubectl get --raw /api/v1/componentstatuses kube-scheduler

# 3. Check the scheduler's logs
kubectl logs -n kube-system -l component=kube-scheduler --tail=200

# 4. Check the scheduler's events
kubectl get events -A --field-selector reason=FailedScheduling

# 5. Check the scheduler's profile
kubectl get schedulerconfiguration -o yaml

The diagnostic is the pending Pods, the scheduler’s logs, and the events.

Common failures

• Scheduler is down. The scheduler container has crashed. The remediation is to restart the static Pod by moving its manifest aside and back.
• Leader election failing. The scheduler cannot become the leader. The remediation is to fix the election.
• Filter timeout. The scheduler’s filters are timing out. The remediation is to simplify the filter.
• Score timeout. The scheduler’s scoring is timing out. The remediation is to simplify the scoring.

flowchart TD
    A[Scheduler failing] --> B{Process running?}
    B -->|No| C[Restart the scheduler]
    B -->|Yes| D{Leader election OK?}
    D---|No| E[Fix the election]
    D---|Yes| F{Filter timeout?}
    F -->|Yes| G[Simplify the filter]
    F -->|No| H{Score timeout?}
    H -->|Yes| I[Simplify the scoring]
    H -->|No| J[Unknown]

The scheduler backlog

A scheduler backlog is the cluster’s hypothesis. The diagnostic is the pending Pods and the scheduler’s logs.

# Check the pending Pods
kubectl get pods -A --field-selector=status.phase=Pending

# Check the scheduler's latency
kubectl logs -n kube-system -l component=kube-scheduler | grep -i "scheduling"

The scheduler backlog is the cluster’s pending Pods.

The remediation

The remediation depends on the cause:

# Option 1: Restart the scheduler.
# kubeadm runs it as a static Pod, so move the manifest out of
# the kubelet's watch directory and back again.
mv /etc/kubernetes/manifests/kube-scheduler.yaml /tmp/
# wait for the kubelet's fileCheckFrequency, 20s by default
mv /tmp/kube-scheduler.yaml /etc/kubernetes/manifests/

# Option 2: Investigate the failing filter
kubectl logs -n kube-system -l component=kube-scheduler | grep -i "filter"

# Option 3: Simplify the filter
# Edit the scheduler's profile
# Disable the slow plugins

# Option 4: Add capacity
# Add nodes to the cluster

The remediation is the scheduler recovery.

Production discipline

A scheduler backlog is the cluster’s hypothesis. The discipline is to walk the 11-step methodology applied to the control plane, identify the cause, apply the remediation. The scheduler is the cluster’s placement engine; the remediation is the scheduler recovery.

• Check the pending Pods. The pending Pods are the cluster’s queue.
• Check the scheduler’s logs. The logs are the scheduler’s view.
• Simplify the filter. The simplification is the scheduler’s recovery.

Quiz