Preemption mechanics — how the scheduler evicts lower-priority Pods

Preemption is the scheduler’s mechanism for evicting lower-priority Pods to make room for higher-priority Pods. This lesson walks the mechanics, the failure modes, and the discipline.

When preemption happens

flowchart LR
    A[Higher-priority Pod pending] --> B{Cluster has room?}
    B -->|Yes| C[Schedule normally]
    B -->|No| D{Lower-priority Pods on a node?}
    D -->|Yes| E["Preempt: evict lower-priority Pods"]
    D -->|No| F[Pod stays Pending]

Preemption happens when:

1. A higher-priority Pod is pending.
2. The scheduler cannot find a node where the Pod fits.
3. There are lower-priority Pods on some node.
4. The scheduler evicts the lower-priority Pods to make room.

The n+1 search

flowchart TD
    A[Higher-priority Pod cannot fit] --> B[Search for a node]
    B --> C{Node has lower-priority Pods?}
    C -->|No| D[Try next node]
    C -->|Yes| E[Can higher-priority Pod fit after eviction?]
    E -->|No| D
    E -->|Yes| F[Nominate candidates for removal]
    F --> G[Schedule higher-priority Pod]
    G --> H[Gracefully terminate lower-priority Pods]

The n+1 search:

1. The scheduler iterates over nodes.
2. For each node, it checks if the higher-priority Pod could fit after removing some lower-priority Pods.
3. If yes, it nominates the lower-priority Pods for removal.
4. The scheduler schedules the higher-priority Pod.
5. The kubelet gracefully terminates the lower- priority Pods.

PodList ordering

flowchart TD
    A[Candidates for removal] --> B{Sort by...}
    B --> C[Highest priority first to remove]
    B --> D[Lowest priority first to remove]
    B --> E[Most requested first to remove]
    B --> F[Least requested first to remove]

The scheduler’s nomination strategy:

• Prefer removing fewer Pods. Remove the largest Pods that free the most resources.
• Prefer removing lower-priority Pods. Lower priority first.
• Respect PDBs. Don’t remove Pods that would violate a PDB.

The scheduler optimises for minimal disruption: the fewest Pods removed to fit the new Pod.

Graceful termination

flowchart LR
    A[Scheduler evicts Pod] --> B{Send SIGTERM}
    B --> C[Container performs graceful shutdown]
    C --> D{terminationGracePeriodSeconds elapsed?}
    D -->|No| E[Wait]
    E --> D
    D -->|Yes| F[Send SIGKILL]
    F --> G[Pod removed]

Graceful termination:

1. The kubelet sends SIGTERM to the Pod’s containers.
2. The containers perform graceful shutdown (close connections, flush state).
3. After terminationGracePeriodSeconds (default 30s), the kubelet sends SIGKILL.
4. The Pod is removed.

A long-running graceful shutdown (e.g., database flush) may need a longer terminationGracePeriodSeconds.

The failure modes

flowchart LR
    A[Failure modes] --> B[No preemption candidates]
    A --> C[PDB blocking]
    A --> D[Graceful termination timeout]
    A --> E[Cascading preemptions]

The failure modes:

• No preemption candidates. All lower-priority Pods are protected by PDBs; the higher-priority Pod stays Pending.
• PDB blocking. A PDB is too restrictive; the scheduler cannot evict without violating it.
• Graceful termination timeout. A container does not shut down within terminationGracePeriodSeconds; SIGKILL is sent. Data loss possible.
• Cascading preemptions. Preempting one Pod triggers another preemption; the cluster thrashes.

Quiz

The operational discipline

Preemption in production rests on five non-negotiable elements:

• Configure PDBs with headroom. Allow preemption to evict Pods.
• Set sensible terminationGracePeriodSeconds. Long enough for graceful shutdown.
• Monitor preemption events. Alert on frequent preemption; the cluster is under pressure.
• Test preemption. Quarterly: simulate a cluster under pressure; verify higher-priority Pods are scheduled.
• Document the preemption policy. The runbook lists the priority classes and their preemption behaviour.

Preemption is a powerful tool, but it requires discipline. PDBs that are too restrictive prevent preemption; PDBs that are too loose allow eviction of critical Pods. The balance is operational.