PDB interaction with preemption — budgets that block preemption

Pod Disruption Budgets interact with preemption: PDBs can block preemption if too restrictive. This lesson walks the interaction, the failure mode, the configuration, and the discipline.

How PDBs interact with preemption

flowchart LR
    A[Higher-priority Pod pending] --> B{Cluster has room?}
    B -->|No| C[Find lower-priority Pods]
    C --> D{PDB violated by eviction?}
    D -->|Yes| E[Skip this Pod]
    D -->|No| F[Evict this Pod]
    F --> G[Schedule higher-priority Pod]

The interaction:

1. The higher-priority Pod cannot fit.
2. The scheduler searches for lower-priority Pods.
3. For each candidate, the scheduler checks the candidate’s PDB.
4. If evicting the candidate would violate the PDB, the candidate is skipped.
5. If evicting the candidate would not violate the PDB, the candidate is evicted.

A restrictive PDB (e.g., minAvailable: 100%) blocks all preemptions; the higher-priority Pod stays Pending.

The PDB configuration

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: myapp-pdb
  namespace: prod-app
spec:
  minAvailable: 3  # at least 3 of 5 must be available
  selector:
    matchLabels:
      app: myapp

The PDB fields:

• minAvailable. Minimum number (or percentage) of Pods that must be available.
• maxUnavailable. Maximum number (or percentage) of Pods that can be unavailable.

A PDB with minAvailable: 3 for a Deployment with 5 replicas allows 2 Pods to be evicted (1 minimum must be unavailable to evict one).

The failure mode

flowchart LR
    A["PDB: minAvailable 100%"] --> B[No eviction candidates]
    C[Higher-priority Pod] --> D[Pending]
    D --> E[Resource pressure]
    E --> F[Production failure]

The failure mode:

1. A PDB has minAvailable: 100% for a Deployment.
2. A higher-priority Pod needs to schedule.
3. The cluster is full; the scheduler tries to preempt.
4. All candidates are protected by the PDB; no eviction is possible.
5. The higher-priority Pod stays Pending.
6. Production workload fails because the critical Pod cannot schedule.

The fix: configure PDBs with headroom.

Configuration with headroom

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: myapp-pdb
  namespace: prod-app
spec:
  minAvailable: 3  # 3 of 5 must be available; allows 2 evictions
  selector:
    matchLabels:
      app: myapp

The headroom: 3 of 5 = 2 Pods can be evicted. The scheduler can preempt up to 2 Pods without violating the PDB.

The discipline

flowchart LR
    A[PDB configuration] --> B["5 replicas: minAvailable 3"]
    C[PDB configuration] --> D["10 replicas: minAvailable 7"]
    E[PDB configuration] --> F["20 replicas: minAvailable 14"]

The discipline:

• 5 replicas. minAvailable: 3 (60%).
• 10 replicas. minAvailable: 7 (70%).
• 20+ replicas. minAvailable: 70% of replicas.

The percentage is the rule of thumb: leave at least 30-40% headroom for preemption.

Quiz

The operational discipline

PDB + preemption in production rests on five non-negotiable elements:

• Leave headroom in PDBs. At least 30% of replicas can be unavailable.
• Test PDB + preemption scenarios. Quarterly: simulate preemption; verify PDB allows.
• Monitor preemption failures. Alert on FailedScheduling due to PDB.
• Document the PDB policy. The runbook lists each PDB and its headroom.
• Adjust PDBs when preemption pressure increases. Quarterly review.

PDBs and preemption are a balancing act. PDBs protect workloads from disruption; preemption is a disruption. The balance is operational: PDBs with enough headroom to allow preemption, but not so much that availability is at risk.