PVC Pending — the diagnostic ladder for a stuck volume claim

A PVC in Pending state is the most common storage incident. The cause is one of four categories: the StorageClass, the provisioner, the capacity, or the topology. This lesson walks the diagnostic ladder and the production fixes.

The four common causes

flowchart TD
    A[PVC Pending] --> B{kubectl describe pvc}
    B --> C{StorageClass issue?}
    B --> D{Provisioner issue?}
    B --> E{Capacity issue?}
    B --> F{Topology issue?}
    C -->|yes| G[Fix StorageClass]
    D -->|yes| H[Fix provisioner]
    E -->|yes| I[Increase quota or reduce request]
    F -->|yes| J[Fix topology constraints]

Cause 1: StorageClass

The most common cause: the PVC requests a StorageClass that does not exist, has no provisioner, or has a provisioner that is not running.

# Substitute your own value before running:
PVC=data-postgres-0

kubectl describe pvc "$PVC"
# Events:
#   Warning  ProvisioningFailed  ... no persistent volumes available
#                                for this claim and no storage class is set
#   Warning  ProvisioningFailed  ... storageclass "nonexistent" not found

The fix:

• Create the StorageClass if it does not exist.
• Verify the provisioner name matches a running CSI driver.
• Verify the StorageClass parameters are valid.

Cause 2: Provisioner

The provisioner (CSI driver) is down, throttled, or has configuration errors.

# Substitute your own value before running - the `app` label the CSI
# driver's controller Deployment carries (e.g. ebs-csi-controller):
CSI_DRIVER=ebs-csi-controller

# Check the controller plugin
kubectl -n kube-system get pods -l "app=$CSI_DRIVER,role=controller"

# Check the controller plugin logs
kubectl -n kube-system logs -l "app=$CSI_DRIVER,role=controller" --tail=50

Common provisioner issues:

• Pod down: the controller plugin Deployment is missing replicas. The fix: investigate why.
• API throttling: the backend API is throttled (AWS, GCP, Azure). The fix: reduce the PVC submission rate or request a quota increase.
• IAM permissions: the CSI driver does not have permission to call the backend. The fix: update IAM permissions.
• Parameter validation: the StorageClass parameters are invalid. The fix: correct the parameters.

Cause 3: Capacity

The backend cannot fulfill the PVC’s capacity request:

• The PVC requests more than the backend allows (e.g., 100 TB on EBS).
• The cluster has reached its storage quota (e.g., AWS account limit).
• The StorageClass’s parameters exceed the backend’s limits (e.g., 100000 IOPS on gp3).

# Substitute your own value before running:
PVC=data-postgres-0

kubectl describe pvc "$PVC"
# Events:
#   Warning  ProvisioningFailed  ... exceeded quota: storage-gigabytes

The fix:

• Reduce the PVC’s capacity request.
• Reduce the StorageClass’s parameters (e.g., lower IOPS).
• Request a quota increase from the cloud provider.

Cause 4: Topology

The PVC cannot be created in any zone that matches the Pod’s topology:

• The StorageClass’s allowedTopologies does not include the cluster’s zones.
• The StorageClass has Immediate binding on a multi-AZ cluster, and the PV is created in a different AZ than the Pod.

# Substitute your own value before running:
PVC=data-postgres-0

kubectl describe pvc "$PVC"
# Events:
#   Warning  ProvisioningFailed  ... no topology constraints found

The fix:

• Set volumeBindingMode: WaitForFirstConsumer so the PV is created in the Pod’s AZ.
• Set allowedTopologies to include the cluster’s AZs.

The diagnostic ladder

flowchart TD
    A[PVC Pending] --> B[kubectl describe pvc]
    B --> C{Event message?}
    C -->|StorageClass not found| D[Create StorageClass]
    C -->|Provisioner error| E[kubectl logs controller plugin]
    C -->|Quota exceeded| F[Backend quota]
    C -->|Topology| G[Verify allowedTopologies]
    C -->|No events| H[Bare PVC: default StorageClass missing]
    E -->|throttling| I[Reduce rate or request quota]
    E -->|IAM| J[Fix permissions]
    E -->|parameters| K[Fix parameters]

The monitoring

A PVC in Pending state blocks the Pod that mounts it. The monitoring:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: storage-pending
spec:
  groups:
  - name: storage
    rules:
    - alert: PVCPendingTooLong
      expr: |
        kube_persistentvolumeclaim_status_phase{phase="Pending"} == 1
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "PVC {{ $labels.persistentvolumeclaim }} pending in ns {{ $labels.namespace }} for 5+ minutes"

The alert catches PVCs that are Pending for more than 5 minutes. The diagnostic then walks the ladder.

The production fix

For each cause:

Quiz

Production discipline

• Pending PVC is a SLO. A PVC in Pending blocks the Pod; the workload is unavailable.
• The diagnostic ladder is events -> logs -> backend. Each layer reveals a different cause.
• Monitor for Pending PVCs. Alert on Pending for 5+ minutes.
• Validate StorageClasses with test PVCs. A regular validation catches misconfigurations before workloads deploy.
• Document the fix in the runbook. The fix is per-cause; the runbook has the diagnostic for each.