Preemption and priority — when a Pod evicts another

When a high-priority Pod cannot be scheduled because every node is full, the scheduler can preempt a lower-priority Pod to make room. Preemption is the scheduler’s last resort; it evicts a running Pod, which is a destructive operation. This lesson walks the preemption algorithm, the priority classes that control it, and the operational discipline that prevents preemption from becoming a denial-of-service vector.

Priority classes

The PriorityClass is a cluster-scoped object that assigns a numeric priority to Pods that reference it.

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: high-priority
value: 1000000
globalDefault: false
description: 'Pods that must always run; they preempt lower-priority Pods.'

A Pod references the priority class:

spec:
  priorityClassName: high-priority

The priority value is a 32-bit integer. The scheduler sorts the scheduling queue by priority, highest first. The admission control admits a Pod only if its priority is higher than the Pods already at the cluster’s quota boundary (using the system-cluster-critical and system-node-critical priorities for the cluster’s own workloads).

The globalDefault field controls which PriorityClass a Pod gets if it does not specify one. Setting globalDefault: true on a PriorityClass is the production default; a Pod without priorityClassName then gets the default priority.

The preemption algorithm

When a Pod is unschedulable, the scheduler’s DefaultPreemption post-filter plugin runs. The algorithm:

1. Find the highest-priority Pod in the scheduling queue that is unschedulable.
2. Walk the cluster’s nodes and find the nodes that would be feasible if some Pods were evicted.
3. For each candidate node, find the set of lower- priority Pods whose combined resource requests would free enough capacity.
4. Pick the candidate with the smallest set of victims (minimising disruption) and the highest-priority victim (minimising the lost priority).
5. Record the preemption in the scheduler’s internal state. The victims are marked for graceful termination.
6. The scheduler attempts to bind the Pod to the chosen node. The victims are evicted through the API server’s normal termination path.

sequenceDiagram
    autonumber
    participant S as Scheduler
    participant API as API server
    participant V1 as Pod A (priority 10)
    participant V2 as Pod B (priority 10)
    participant HP as Pod Z (priority 1000)

    HP->>API: created
    S->>API: scheduling cycle
    Note over S: No node has room for Pod Z
    S->>API: find victim Pods
    API-->>S: Pod A and Pod B on node-1 (priority 10)
    S->>API: preempt Pod A and Pod B
    API-->>V1: graceful termination
    API-->>V2: graceful termination
    S->>API: bind Pod Z to node-1

The scheduler does not directly delete the victims. It records the preemption, and the API server’s API machinery delivers the eviction to the kubelet. The kubelet then sends SIGTERM to the victims and waits for the terminationGracePeriodSeconds.

The PreemptionPolicy

The PreemptionPolicy field on the Pod controls the Pod’s behaviour as a victim:

spec:
  priorityClassName: critical
  preemptionPolicy: Never

Two values:

• PreemptLowerPriority (default): the Pod can be preempted by a higher-priority Pod.
• Never: the Pod is never preempted. The scheduler will not pick it as a victim, even if a higher-priority Pod is waiting.

PreemptionPolicy: Never is the right choice for the cluster’s own critical workloads (the CNI, the metrics server, the API server’s local Pods). These workloads should not be evicted to make room for a user Pod.

A Pod with PreemptionPolicy: Never is “sticky”: the scheduler will skip it as a victim. The Pod effectively has a hard reservation. The operational cost is that the cluster cannot get rid of the Pod even if it is no longer needed.

The preemption cost

Preemption is disruptive. The costs:

• The victim Pod loses its state. A stateless Pod tolerates preemption; a stateful Pod may lose the in-memory state and cannot serve requests during start-up.
• The victim’s replicas are reduced. If the Deployment has 3 replicas and one is preempted, the controller creates a replacement. The replacement is scheduled elsewhere or back on the same node after the high- priority Pod is gone.
• The cluster’s capacity is reduced. The high-priority Pod takes the place of the victim; the cluster has the same number of Pods but less variety.
• The Pod’s startup time is paid twice. The high- priority Pod must wait for the victim to drain (SIGTERM grace period), then the high-priority Pod starts.

The scheduler tries to minimise the cost. The preemption algorithm picks the node with the smallest set of victims. But if every node has the same lower-priority Pods, the cost is the same.

The candidate selection

The scheduler’s DefaultPreemption plugin picks the candidate node. The selection criteria:

1. Smaller victim set is better. A node where one Pod must be evicted is preferred over a node where three Pods must be evicted.
2. Higher-priority victims are better. A node with a priority-10 victim is preferred over a node with a priority-5 victim (the preemption “wastes” less of the cluster’s priority budget).
3. Namespaces with fewer victims are better. The scheduler considers namespace locality to avoid disrupting a single tenant.

The scheduler’s choice is internal. The operator can see the preemption in the events:

Events:
  Type     Reason            Age   From               Message
  ----     ------            ----  ----               ----
  Normal   Preempted         2m    kubelet             Pod billing-1 was preempted by Pod critical-1.

The Preempted reason is set on the victim Pod before its deletion. The cluster’s audit log records the preemption.

The preemption loop

The most dangerous failure mode is a preemption loop: Pod A preempts Pod B, Pod C preempts Pod A, Pod B preempts Pod C. The cluster churns, the Pods are deleted and recreated, and the actual user workloads stop.

flowchart LR
    A[Pod A<br/>priority 100] --> B[Pod B<br/>priority 200]
    B --> C[Pod C<br/>priority 300]
    C --> A

The loop is a misconfiguration: the priority classes are inverted, or the priority values are too close together, or the cluster has too little capacity.

The fix is to:

1. Audit the priority classes. A cluster with multiple priority classes at close values (e.g., 100, 200, 300) is more prone to loops than a cluster with widely separated values (10, 1000, 1000000).
2. Audit the targets. A Pod that preempts itself (priority 100 vs priority 100) is a misconfiguration.
3. Add capacity. Preemption is the cluster’s way of saying “we don’t have enough room.” Adding nodes removes the need for preemption.

The priority inversion

A subtler failure mode: a low-priority Pod is on a node that is shared with a high-priority Pod. The high-priority Pod is healthy, but the low-priority Pod is starving it of resources. The high-priority Pod’s QoS is degraded.

The fix is to use Pod-level resource isolation (separate nodes, separate node selectors) so the high-priority Pod is not on a node with the low-priority Pod. The priority class alone does not isolate resources; it controls the scheduler’s preemption decision.

Quiz

Production discipline

• Two priority classes are usually enough. A system-cluster-critical (the cluster’s own pods) and a user-priority (general workloads). Add more only when there is a clear distinction.
• PreemptionPolicy: Never for the cluster’s own workloads. The CNI, kube-proxy, and the kubelet’s static Pods must be unevictable.
• Audit preemption at every release. A rising preemption count is a cluster capacity issue. The cluster-autoscaler should be triggered.
• Use namespaced PriorityClasses carefully. A PriorityClass with globalDefault: true applies to every Pod in the cluster. A PriorityClass with globalDefault: false is opt-in. The default value should be conservative.
• Watch the preemption cost. The scheduler’s metric scheduler_preemption_attempts_total and scheduler_preemption_victims_total (or the scheduling_preemption_attempts_total in newer versions) are the right alerts.