Drain with PodDisruptionBudgets — the interruption limit

The drain’s interaction with the PodDisruptionBudget (PDB) is the cluster’s protection against the drain disrupting the workload. The eviction API checks the PDB; the drain is rejected when the PDB rejects the eviction. This lesson walks the interaction, the design patterns, and the failure modes.

The drain’s interaction with the PDB

The drain’s eviction API checks the PDB:

sequenceDiagram
    autonumber
    participant O as Operator
    participant API as API server
    participant PDB as PDB controller

    O->>API: evict Pod
    API->>PDB: check PDB
    PDB->>API: can evict?
    API->>O: 200 OK or 403 Forbidden

The eviction API checks the PDB before allowing the eviction. The PDB’s minAvailable and maxUnavailable fields determine whether the eviction is allowed.

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: billing-pdb
spec:
  minAvailable: 3
  selector:
    matchLabels:
      app: billing

The PDB’s minAvailable: 3 means at least 3 Pods must be available at any time. The eviction is rejected if the eviction would violate the PDB.

The drain’s behaviour with the PDB

The drain’s behaviour with a restrictive PDB:

NODE=node-1               # node name from `kubectl get nodes`

kubectl drain "$NODE"

node/node-1 cordoned
error when evicting pod "prod-app/billing-1": Cannot evict pod as it would violate the pod's disruption budget.

The drain is rejected. The Pod is not evicted. The drain is incomplete.

The drain’s behaviour with a permissive PDB:

NODE=node-1               # node name from `kubectl get nodes`

kubectl drain "$NODE"

node/node-1 cordoned
evicting pod "prod-app/billing-1"
pod "prod-app/billing-1" evicted
node/node-1 drained

The drain is allowed. The Pod is evicted.

The PDB’s fields

The PDB’s fields:

The fields are mutually exclusive. The PDB’s minAvailable and maxUnavailable cannot both be set.

The drain’s behaviour with minAvailable: 3:

• The drain is allowed if the eviction would not reduce the available Pods below 3.
• The drain is rejected if the eviction would reduce the available Pods below 3.

The drain’s behaviour with maxUnavailable: 1:

• The drain is allowed if the eviction would not increase the unavailable Pods above 1.
• The drain is rejected if the eviction would increase the unavailable Pods above 1.

The PDB’s design

The PDB’s design should be:

• Permissive enough to allow the drain. The minAvailable should be one less than the replica count; the maxUnavailable should be one.
• Restrictive enough to protect the workload. The minAvailable should be the minimum number of Pods that can serve the workload; the maxUnavailable should be the maximum number of Pods that can be unavailable.

The PDB’s design is the cluster’s protection against the drain disrupting the workload. The production rule is to design the PDB to allow the drain while protecting the workload.

The PDB’s failure modes

The PDB’s failure modes:

The diagnostic:

PDB=billing-pdb           # the budget from `kubectl get pdb -A`

kubectl describe pdb "$PDB"

The PDB’s status shows the current state. The fix is to investigate the PDB’s configuration.

The PDB’s anti-patterns

The PDB’s anti-patterns:

• PDB with minAvailable: 100%. The PDB that requires all Pods to be available is a PDB that blocks the drain.
• PDB with minAvailable: 0. The PDB that requires no Pods to be available is a PDB that does not protect the workload.
• PDB with no selector. The PDB that does not select any Pods is a PDB that does not protect the workload.
• PDB with a wrong selector. The PDB that selects the wrong Pods is a PDB that does not protect the workload.

The PDB’s design is the operator’s responsibility. A PDB that selects nothing, or that demands more Pods than the workload runs, fails silently; kubectl get pdb reports an ALLOWED DISRUPTIONS of 0.

The drain’s interaction with the PDB’s controller

The PDB’s controller is the eviction controller in the kube-controller-manager. The controller watches the Pod’s status and the PDB’s spec.

The controller’s logic:

flowchart TD
    A[Eviction API call] --> B{Can evict?}
    B -->|Yes| C[Allow eviction]
    B -->|No| D[Reject eviction]

The controller’s logic is the cluster’s enforcement of the PDB. The controller is the cluster’s protection against the drain disrupting the workload.

The PDB’s interaction with the eviction webhook

The PDB’s interaction with the eviction webhook is the cluster’s custom logic. The webhook is called before the PDB is checked.

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: evict-policy
spec:
  rules:
    - operations: ["CREATE"]
      apiGroups: ["policy"]
      apiVersions: ["v1"]
      resources: ["evictions"]

The webhook is the cluster’s custom logic. The webhook can reject the eviction based on the Pod’s metadata, the namespace, or the request.

The drain’s verification with the PDB

The drain’s verification with the PDB:

kubectl get pdb -A

The output shows the PDB’s status. The PDB’s status shows the current allowed disruptions.

The drain’s verification:

NODE=node-1               # the node just drained

kubectl get pods -o wide | grep "$NODE"

The output should be empty. The drain is complete when the node has no Pods.

The drain’s operational patterns

The drain’s operational patterns with the PDB:

• Design the PDB to allow the drain. The minAvailable should be one less than the replica count; the maxUnavailable should be one.
• Drain with the PDB in mind. The drain is rejected if the PDB rejects the eviction. The fix is to adjust the PDB.
• Use --force to bypass the PDB. The --force flag is destructive; the operator should use it carefully.
• Verify the PDB. The PDB’s status shows the current allowed disruptions. The operator should verify the PDB before the drain.

Quiz

Production discipline

• The PDB is the cluster’s protection against the drain. The PDB is the workload’s contract with the cluster.
• Design the PDB to allow the drain. The minAvailable should be one less than the replica count; the maxUnavailable should be one.
• Drain with the PDB in mind. The drain is rejected if the PDB rejects the eviction.
• Use --force to bypass the PDB. The --force flag is destructive; the operator should use it carefully.
• Audit the PDB at every release. The PDB’s configuration should be version-controlled; the audit catches the failures.
• Test the drain in non-production. A staging cluster that mirrors production is the right place to test the drain.