PDB and eviction — what the eviction API actually checks

The PodDisruptionBudget is enforced in exactly one place: the API server’s eviction subresource. An eviction that would take the budget below its floor is answered with 429 Too Many Requests, which means ‘not now’ rather than ‘never’. Everything else that removes a Pod — node-pressure eviction, node failure, a plain DELETE — never consults the budget at all. This lesson walks the eviction API’s PDB check, the disruption controller’s part in it, and the failure modes.

The eviction API

The eviction API is a Kubernetes API (apiVersion: policy/v1, kind: Eviction) that allows an authorized entity to evict a Pod.

{
  "apiVersion": "policy/v1",
  "kind": "Eviction",
  "metadata": {
    "name": "billing-1",
    "namespace": "prod-app"
  }
}

The eviction API sends a POST to /api/v1/namespaces/<ns>/pods/<name>/eviction. The API server runs the admission checks and answers with one of three statuses:

• 200 OK — the eviction is allowed. The Eviction subresource is created and the Pod is deleted gracefully, exactly as a DELETE on the Pod URL would delete it.
• 429 Too Many Requests — the eviction is not allowed at the moment, because the matching PodDisruptionBudget has no disruptions left. The caller may retry later. API rate limiting produces the same status.
• 500 Internal Server Error — the eviction cannot be evaluated because of a misconfiguration, most often two PodDisruptionBudgets matching the same Pod.

The 429 is why kubectl drain waits rather than failing: it is being told ‘not yet’. A drain that sits quietly for twenty minutes is a budget that never recovers, not a broken call.

The eviction API is the standard mechanism for a voluntary disruption: kubectl drain, the cluster autoscaler, and any well-behaved node-maintenance controller use it. The kubelet does not — its node-pressure eviction terminates Pods directly and never touches this endpoint.

The voluntary vs involuntary eviction

Kubernetes separates two kinds of Pod loss:

• Voluntary disruption: an authorized client — an operator running kubectl drain, the cluster autoscaler, a node-maintenance controller — asks the API server to evict the Pod. The eviction subresource checks the PDB.
• Involuntary disruption: the Pod goes away without anyone calling the eviction API. The kubelet evicts it under memory, disk or PID pressure; the node fails; the kernel panics; someone issues a plain DELETE on the Pod. Nothing consults the PDB.

The voluntary path is how the cluster manages a node’s lifecycle. The involuntary path is what happens to the workload regardless of what the operator intended.

The eviction API’s PDB check

The eviction API’s PDB check:

sequenceDiagram
    autonumber
    participant O as Operator
    participant API as API server
    participant PDB as PodDisruptionBudget

    O->>API: POST pods/name/eviction
    API->>API: validate the Pod
    API->>PDB: read status.disruptionsAllowed
    PDB->>API: 0, or a positive number
    API->>O: 200 OK, or 429 when it is 0

The API server checks the PDB before allowing the eviction, and status.disruptionsAllowed is the single number it acts on.

The eviction API’s algorithm:

1. Validate the Pod's identity.
2. Find the PDB whose selector matches the Pod.
3. If more than one PDB matches, return 500.
4. Read that PDB's status.disruptionsAllowed.
5. If it is 0, return 429 Too Many Requests.
6. Otherwise decrement it and delete the Pod gracefully.

disruptionsAllowed is derived from minAvailable or maxUnavailable against the number of currently healthy Pods, and it is refilled only once the replacements report Ready. That refill is what paces a drain.

The PDB controller’s logic

The disruption controller runs in the kube-controller-manager. It does not reject anything — it keeps the PDB’s status current so the API server has a number to act on:

1. Watches the matching Pods. It resolves the PDB’s selector and observes which Pods are Ready.
2. Recomputes the status. currentHealthy, desiredHealthy, expectedPods and disruptionsAllowed.
3. Refills the budget. Once replacements report Ready, disruptionsAllowed rises again and the next eviction is permitted.

Enforcement lives in the API server, on the eviction subresource; the controller only supplies the number. A PDB whose selector matches nothing therefore has an empty status, and an operator reading ALLOWED DISRUPTIONS gets a misleading answer about a budget that is protecting no Pods at all.

The voluntary eviction’s effects

An allowed eviction is a graceful delete:

• The Pod resource is marked with a deletion timestamp and the configured grace period.
• The kubelet notices and starts the shutdown: SIGTERM to the containers.
• The control plane removes the Pod from its EndpointSlices, so traffic stops arriving.
• After terminationGracePeriodSeconds the kubelet sends SIGKILL and the Pod resource is removed.

An API-initiated eviction respects terminationGracePeriodSeconds, and the Pod is deleted rather than left behind in a Failed phase.

The involuntary eviction’s effects

A node-pressure eviction is not a delete at all:

• The kubelet ranks the Pods on the node and picks victims itself.
• On a soft eviction threshold it honours --eviction-max-pod-grace-period; on a hard threshold the grace period is 0s and the containers are killed immediately.
• The kubelet sets the Pod’s phase to Failed, with a message naming the exhausted resource.

Node-pressure eviction respects neither the PodDisruptionBudget nor the Pod’s terminationGracePeriodSeconds.

The PDB’s failure modes

The PDB’s failure modes:

The diagnostic:

# Substitute your own values before running:
NS=production
PDB=web-pdb

kubectl get pdb -n "$NS"
kubectl describe pdb "$PDB" -n "$NS"

ALLOWED DISRUPTIONS is the number the API server acts on. While it reads 0, every eviction gets a 429. If EXPECTED PODS reads 0, the selector is wrong and the budget is protecting nothing.

The PDB’s interaction with the drain

The PDB’s interaction with the drain:

flowchart TD
    A[kubectl drain] --> B[POST eviction for Pod 1]
    B --> C{"disruptionsAllowed above zero?"}
    C -->|Yes| D[200 OK, Pod 1 deleted gracefully]
    C -->|No| E[429, drain waits and retries]
    E --> C
    D --> F[POST eviction for Pod 2]
    F --> G{"disruptionsAllowed above zero?"}
    G -->|Yes| H[200 OK, Pod 2 deleted gracefully]
    G -->|No| E

A drain does not fail on a 429; it waits and retries. That is the intended behaviour — the budget paces the drain instead of aborting it — but it also means a budget that can never be satisfied produces a drain that never returns.

The PDB’s interaction with a node shutdown

Before a node is powered off, it is drained:

NODE=worker-03            # node name from `kubectl get nodes`

kubectl drain "$NODE"

The drain evicts each Pod through the eviction subresource, so the budget is honoured and the workload is rescheduled elsewhere before the node goes away. Shutting a node down without draining it first turns a planned maintenance into an involuntary disruption, which the budget cannot see.

The PDB’s interaction with the upgrade

The PDB’s interaction with the upgrade:

NODE=worker-03            # node name from `kubectl get nodes`

kubectl drain "$NODE"

The drain evicts the Pods. The PDB holds each eviction until enough replacements report Ready, which paces the upgrade one node at a time.

The upgrade is the cluster’s mechanism for upgrading the Kubelet. The drain evicts the Pods; the upgrade replaces the kubelet.

Quiz

Production discipline

• The PDB is the cluster’s protection against voluntary eviction. The PDB is the workload’s contract with the cluster.
• The PDB does not protect against involuntary disruption. Node-pressure eviction, node failure and a direct DELETE all bypass it.
• Design the PDB to allow the drain. The minAvailable should be one less than the replica count; the maxUnavailable should be one.
• Audit the PDB at every release. The PDB’s configuration should be version-controlled; the audit catches the failures.
• Monitor the PDB’s status. The PDB’s disruptionsAllowed is the operator’s primary signal.
• Document the PDB’s intent. A PDB that does not have a documented intent is a PDB that does not protect the workload.
• Test the PDB in non-production. A staging cluster that mirrors production is the right place to test the PDB.