PDB failure modes — when the protection backfires

The PDB is the cluster’s protection against voluntary disruption. The PDB can fail in five ways: too- restrictive, too-permissive, missing, wrong selector, or invalid. Each failure mode has a diagnostic and a fix. This lesson walks the failure modes, the diagnostics, and the operational patterns.

The five common PDB failure modes

The failure modes are the cluster’s silent failures. The cluster’s events show the failure; the operator’s diagnostic is the PDB’s status.

Failure mode 1: too restrictive

A PDB with minAvailable: 100% is a PDB that requires all Pods to be available. The drain is rejected:

# Substitute your own node name:
NODE=worker-03

kubectl drain "$NODE"

error when evicting pod "prod-app/billing-1": Cannot evict pod as it would violate the pod's disruption budget.

The diagnostic:

# Substitute your own PDB name:
PDB=billing-pdb

kubectl describe pdb "$PDB"

Name:           billing-pdb
Min Available:  100%
Disruptions Allowed:  0
Expected Pods:  3
Current Healthy:  3

The PDB’s Min Available: 100% is the cause. The fix is to adjust the PDB to allow the drain:

spec:
  minAvailable: 2

The fix is to set the minAvailable to one less than the replica count.

Failure mode 2: too permissive

A PDB with minAvailable: 0 is a PDB that does not protect the workload. The drain is allowed; the workload is unprotected.

The diagnostic:

# Substitute your own PDB name:
PDB=billing-pdb

kubectl describe pdb "$PDB"

Name:           billing-pdb
Min Available:  0
Disruptions Allowed:  3
Expected Pods:  3
Current Healthy:  3

The PDB’s Min Available: 0 is the cause. The fix is to adjust the PDB to protect the workload:

spec:
  minAvailable: 2

The fix is to set the minAvailable to the minimum number of Pods that can serve the workload.

Failure mode 3: missing

A missing PDB is a PDB that does not exist. The workload is unprotected.

The diagnostic:

# Substitute the workload whose PDB you are looking for:
WORKLOAD=billing

kubectl get pdb -A | grep "$WORKLOAD"

The output is empty. The PDB is missing.

The fix is to create the PDB:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: billing-pdb
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: billing

The production rule is to have a PDB for every production-critical workload.

Failure mode 4: wrong selector

A PDB with a wrong selector is a PDB that does not apply to the workload. The PDB is created but does not protect the Pods.

The diagnostic:

# Substitute your own PDB name:
PDB=billing-pdb

kubectl describe pdb "$PDB"

Name:           billing-pdb
Min Available:  2
Disruptions Allowed:  N/A
Expected Pods:  0

The PDB’s Expected Pods: 0 is the cause. The selector does not match any Pods.

The fix is to correct the selector:

spec:
  selector:
    matchLabels:
      app: billing

The production rule is to verify the PDB’s selector against the workload’s labels.

Failure mode 5: invalid

A PDB with an invalid spec is a PDB that is rejected by the API server.

The diagnostic:

kubectl get pdb

The output shows the PDB’s status. The PDB’s status shows the validation error.

The fix is to correct the PDB’s spec:

spec:
  minAvailable: 2  # absolute or percentage

The PDB’s validation rejects:

• Both minAvailable and maxUnavailable set.
• Negative values.
• Values that exceed the replica count.

The production rule is to validate the PDB’s spec before applying.

The PDB’s debugging workflow

The PDB’s debugging workflow:

flowchart TD
    A[PDB failure] --> B[describe pdb]
    B --> C{PDB applied?}
    C -->|No| D[Check selector]
    D --> E[Correct selector]
    C -->|Yes| F{DisruptionsAllowed = 0?}
    F -->|Yes| G[Check minAvailable]
    G --> H[Adjust minAvailable]
    F -->|No| I[PDB is fine]

The workflow:

1. Run kubectl describe pdb. The output shows the PDB’s status.
2. Check the selector. The PDB’s Expected Pods should match the workload’s Pods.
3. Check the disruptionsAllowed. The value should be greater than 0.
4. Adjust the PDB. The fix is to adjust the PDB’s spec.

The PDB’s logs

The PDB’s logs are not directly accessible. The PDB’s status is the only signal.

The PDB’s events are recorded in the cluster’s events:

kubectl get events --field-selector involvedObject.kind=PodDisruptionBudget

The events show the PDB’s state. The fix is to investigate the events.

The PDB’s metrics

The PDB’s metrics:

promtool query instant http://prometheus:9090 \
  'kube_poddisruptionbudget_status'

The metric returns the PDB’s status. The operator should monitor the metric and alert on the threshold.

The PDB’s anti-patterns

The PDB’s anti-patterns:

• PDB with minAvailable: 100%. The PDB that blocks the drain.
• PDB with minAvailable: 0. The PDB that does not protect the workload.
• PDB missing. The PDB that does not exist.
• PDB with wrong selector. The PDB that does not apply to the workload.
• PDB with invalid spec. The PDB that is rejected.

The PDB’s anti-patterns are the cluster’s silent failures. The production rule is to design the PDB to allow the drain while protecting the workload.

Quiz

Production discipline

• The PDB can fail in five ways. The failure modes are: too restrictive, too permissive, missing, wrong selector, or invalid.
• Use the PDB’s debugging workflow. The workflow is: describe, check selector, check disruptionsAllowed, adjust.
• Audit the PDB at every release. The PDB’s configuration should be version-controlled; the audit catches the failures.
• Monitor the PDB’s metrics. The PDB’s disruptionsAllowed is the operator’s primary signal.
• Document the PDB’s intent. A PDB that does not have a documented intent is a PDB that does not protect the workload.
• Test the PDB in non-production. A staging cluster that mirrors production is the right place to test the PDB.
• The PDB is the cluster’s contract. The cluster promises to respect the PDB; the workload promises to tolerate involuntary disruption.