Eviction flow — from pressure to eviction to controller replacement

The eviction flow is the cluster’s response to node pressure. This lesson walks the full flow: from the kubelet’s detection to the eviction API call to the controller’s replacement. The lesson also covers the PodDisruptionBudget’s role in limiting the eviction rate and the operational patterns for minimizing the eviction impact.

The eviction flow

sequenceDiagram
    autonumber
    participant K as kubelet
    participant API as API server
    participant C as Controller

    K->>K: detect pressure
    K->>K: identify Pods to evict
    K->>API: POST /eviction
    API->>API: validate eviction
    API->>API: check PodDisruptionBudget
    Note over API: PDB allows eviction?
    API->>K: 200 OK
    K->>K: send SIGTERM
    K->>K: wait for grace period
    K->>K: send SIGKILL
    K->>API: update Pod status: Failed
    C->>API: observe Pod deletion
    C->>API: create replacement Pod
    API->>K: assign new Pod to node

The eviction flow is a multi-step process. The kubelet detects pressure, identifies the Pods to evict, and sends the eviction API call. The API server validates the eviction against the PodDisruptionBudget, sends SIGTERM, and waits for the grace period. The controller creates a replacement Pod.

The eviction API call

The kubelet uses the API server’s eviction API to evict Pods. The API call is a POST to /api/v1/namespaces/<ns>/pods/<name>/eviction.

{
  "apiVersion": "policy/v1",
  "kind": "Eviction",
  "metadata": {
    "name": "billing-1",
    "namespace": "prod-app"
  }
}

The API server validates the eviction:

1. Authentication: the kubelet’s identity is validated.
2. Authorization: the kubelet is authorized to evict Pods.
3. PodDisruptionBudget: the eviction is checked against the PDB.
4. Webhook: the eviction is checked against the eviction webhook (if configured).

The eviction is allowed if all checks pass. The API server sends SIGTERM to the Pod’s containers.

The graceful termination

The kubelet’s eviction is graceful. The Pod’s containers receive SIGTERM; the kubelet waits for the Pod’s terminationGracePeriodSeconds (default 30s); the kubelet sends SIGKILL.

The graceful termination gives the application time to shut down cleanly. The application can flush its buffers, close its connections, and persist its state.

The graceful termination is bounded by the terminationGracePeriodSeconds. The application must shut down within the grace period.

The Pod’s status

The evicted Pod’s Status.Phase is Failed. The Status.Reason is Evicted. The Status.Message includes the reason for the eviction.

# Substitute the evicted Pod's name before running:
POD=billing-1

kubectl get pod "$POD"

NAME        READY   STATUS    RESTARTS   AGE
billing-1   0/1     Evicted   0          5m

The Pod’s Status reflects the eviction. The cluster’s controller observes the Pod’s deletion.

The controller’s replacement

The cluster’s controller (Deployment, StatefulSet, etc.) observes the Pod’s deletion. The controller creates a replacement Pod.

The replacement Pod’s lifecycle:

1. Controller observes the deletion. The controller watches the Pod’s status.
2. Controller creates a replacement Pod. The Pod’s spec is the same as the deleted Pod’s spec.
3. Scheduler binds the Pod to a node. The scheduler picks a node based on the Pod’s constraints.
4. Kubelet starts the Pod. The kubelet runs the Pod’s containers.

The total time from eviction to replacement is the Pod’s termination grace period plus the scheduler’s scheduling time plus the kubelet’s startup time.

The PodDisruptionBudget’s role

The PodDisruptionBudget (PDB) is a Kubernetes object that limits the number of Pods that can be evicted voluntarily. The PDB is checked by the eviction API.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: billing-pdb
spec:
  minAvailable: 3
  selector:
    matchLabels:
      app: billing

The PDB’s minAvailable: 3 means at least 3 Pods must be available at any time. The eviction API rejects the eviction if the eviction would violate the PDB.

The PDB does not block involuntary evictions (memory pressure, disk pressure). The PDB only blocks voluntary evictions (cluster shutdown, node drain).

The kubelet’s eviction is voluntary in the API sense — the kubelet is asking the API server to evict. The API server checks the PDB. If the PDB rejects the eviction, the kubelet cannot evict the Pod.

The eviction webhook

The API server’s eviction webhook allows the cluster operator to reject an eviction based on custom logic. The webhook is called before the eviction is allowed.

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: evict-policy
spec:
  rules:
    - operations: ["CREATE"]
      apiGroups: ["policy"]
      apiVersions: ["v1"]
      resources: ["evictions"]
  failurePolicy: Fail
  validations:
    - expression: "object.metadata.namespace == 'allowed-ns'"
      message: "Evictions are not allowed in this namespace"

The webhook is the cluster’s custom logic. The webhook can reject the eviction based on the Pod’s metadata, the namespace, or the request.

The eviction’s failure modes

The eviction’s failure modes:

The diagnostic:

# Substitute the evicted Pod's name before running:
POD=billing-1

kubectl describe pod "$POD" | grep -A 10 "Events"

The events show the eviction’s failure. The fix is to investigate the events and the kubelet’s logs.

The eviction’s operational patterns

The eviction’s operational patterns:

• Design the PDB to allow the eviction. The minAvailable should be one less than the replica count; the maxUnavailable should be one.
• Set the Pod’s termination grace period. The terminationGracePeriodSeconds should be the application’s shutdown time.
• Handle SIGTERM in the application. The application should flush its buffers, close its connections, and persist its state.
• Monitor the eviction rate. The cluster’s alerts should fire on the eviction rate.
• Test the eviction in non-production. A staging cluster that mirrors production is the right place to test the eviction.
• Audit the eviction at every release. A new workload that is not handling SIGTERM is a workload that will be evicted destructively.

The eviction’s slow churn

The eviction’s slow churn is the most common silent failure. A cluster that is evicting a few Pods per day is a cluster that is silently losing capacity.

The diagnostic:

kubectl get events --all-namespaces | grep -i "Evicted"

The events list the evicted Pods. The fix is to identify the cause:

• A workload that is over-committing memory.
• A workload that is leaking memory.
• A workload that is using more disk than expected.

The production rule is to monitor the eviction rate; the silent eviction is the most common operational failure.

Quiz

Production discipline

• The eviction flow is the cluster’s response to pressure. The kubelet detects pressure, evicts Pods, and the controller creates replacements.
• The PodDisruptionBudget can block the eviction. The fix is to design the PDB to allow the eviction.
• The graceful termination is the application’s responsibility. The application must handle SIGTERM.
• The eviction webhook is the cluster’s custom logic. The webhook can reject the eviction.
• Monitor the eviction rate. The cluster’s alerts should fire on the eviction rate.
• Test the eviction in non-production. A staging cluster that mirrors production is the right place to test the eviction.
• Audit the eviction at every release. A new workload that is not handling SIGTERM is a workload that will be evicted destructively.