PVC pending and dynamic-provision — the storage runtime

A PVC that stays Pending holds its Pod in Pending with it, indefinitely and without an error that names the cause — nothing times out, and the workload simply never starts. The four things that produce it need different fixes: a StorageClass that does not exist, a provisioner whose controller Pod is crash-looping, a WaitForFirstConsumer class waiting for a schedulable node that never appears, and a namespace storage quota that is already full. The provisioning events in kubectl describe pvc separate them, which makes it the first command to run rather than the last.

The PVC lifecycle

A PVC (Persistent Volume Claim) is the workload’s request for storage. The PVC lifecycle is:

1. Pending. The PVC is waiting for a PV to be bound.
2. Bound. The PVC is bound to a PV.
3. In Use. The PVC is mounted by a Pod.

flowchart TD
    A[PVC created] --> B{Pynamic provision?}
    B -->|Yes| C[StorageClass creates PV]
    B -->|No| D[Static PV]
    C --> E[PV bound to PVC]
    D --> E
    E --> F[Pod mounts PVC]

A PVC Pending is one that has not yet been bound.

The diagnostic

The canonical diagnostic:

# Substitute your own values before running:
NS=production
PVC=postgres-data-postgres-0
SC=gp3
CSI_APP=ebs-csi-controller

# 1. Check the PVC
kubectl get pvc -n "$NS"
kubectl describe pvc "$PVC" -n "$NS"

# 2. Check the StorageClass
kubectl get storageclass
kubectl describe storageclass "$SC"

# 3. Check the PV
kubectl get pv

# 4. Check the CSI driver
kubectl get pods -n kube-system -l app="$CSI_APP"

# 5. Check the events
kubectl get events -n "$NS" --field-selector involvedObject.name="$PVC"

The diagnostic is the PVC, the StorageClass, the PV, and the CSI driver.

Common failures

• StorageClass missing. The PVC references a StorageClass that does not exist. The remediation is to create the StorageClass.
• StorageClass provisioner failing. The StorageClass’s provisioner is failing. The remediation is to fix the CSI driver.
• No nodes available. The StorageClass’s volume binding mode is WaitForFirstConsumer and no nodes are available.
• Quota exceeded. The PVC’s namespace has a ResourceQuota for storage, and the quota is exceeded.

flowchart TD
    A[PVC Pending] --> B{StorageClass exists?}
    B -->|No| C[Create the StorageClass]
    B -->|Yes| D{Provisioner OK?}
    D---|No| E[Fix the CSI driver]
    D---|Yes| F{Nodes available?}
    F -->|No| G[Add nodes]
    F -->|Yes| H{Quota exceeded?}
    H -->|Yes| I[Increase the quota]
    H -->|No| J[Unknown]

The CSI driver

The CSI driver is the cluster’s storage runtime. The CSI driver is composed of:

• Controller plugin (a Deployment or StatefulSet).
• Node plugin (a DaemonSet).

# Substitute your own value before running (the `app` label your CSI
# driver's pods carry):
CSI_APP=ebs-csi-controller

# Check the CSI driver
kubectl get pods -n kube-system -l app="$CSI_APP"

# Check the CSI driver's logs
kubectl logs -n kube-system -l app="$CSI_APP" --tail=200

The CSI driver is the cluster’s storage bridge.

The remediation

The remediation depends on the cause:

# Substitute your own values before running:
CSI_DEPLOY=ebs-csi-controller
NODEPOOL=workers-a
NODEPOOL_REPLICAS=6
QUOTA=storage-quota
NS=production

# Option 1: Create the StorageClass
kubectl apply -f storageclass.yaml

# Option 2: Restart the CSI driver
kubectl rollout restart "deployment/$CSI_DEPLOY" -n kube-system

# Option 3: Add nodes
# `nodepool` is not a core Kubernetes resource: substitute your provider's
# node-group API (Karpenter NodePool, Cluster API MachineDeployment) or its
# cloud CLI.
kubectl scale nodepool "$NODEPOOL" --replicas="$NODEPOOL_REPLICAS"

# Option 4: Increase the quota
kubectl patch resourcequota "$QUOTA" -n "$NS" -p '{"spec":{"hard":{"requests.storage":"100Gi"}}}'

The remediation is the storage recovery.

Production discipline

A PVC Pending is the cluster’s hypothesis. The discipline is to walk the 11-step methodology applied to the storage layer, identify the cause, apply the remediation. The storage is the cluster’s data; the remediation is the storage recovery.

• Check the PVC. The PVC is the workload’s request.
• Check the StorageClass. The StorageClass is the workload’s storage class.
• Check the CSI driver. The CSI driver is the cluster’s storage runtime.

Quiz