PVC and storage-driven scheduling failures

A Pod that consumes a PersistentVolumeClaim is scheduled only after the PVC is bound and the node can host the underlying PersistentVolume. The scheduler’s VolumeBinding filter plugin checks this; the failure mode is a Pod that is Pending because the storage is not where the Pod wants to run. This lesson walks the diagnostic workflow.

The chain: Pod → PVC → PV → node

A Pod’s spec includes a volumes[].persistentVolumeClaim reference. The PVC refers to a PV (after binding). The PV is a piece of physical storage in the cluster’s backend (NFS, iSCSI, cloud disk, CSI). The PV’s location is recorded in its spec.nodeAffinity (for volumeBindingMode: WaitForFirstConsumer) or in the storage backend’s zone (for VolumeBinding: Immediate).

flowchart LR
    A[Pod] -->|references| B[PVC]
    B -->|bound to| C[PV]
    C -->|hosted on| D[Backend storage]
    C -->|affinity to| E[Node zone]
    E --> F{Pod's node in zone?}
    F -->|Yes| G[Schedulable]
    F -->|No| H[Reject]
    D --> I{kubelet can mount?}
    I -->|Yes| G
    I -->|No| J[Pending]

The scheduler’s VolumeBinding filter runs after the affinity filters. The plugin checks whether the node can host the PV; for cloud storage, this is the zone/region match.

The two binding modes

A StorageClass declares volumeBindingMode. The mode controls when the PV is created and where it is anchored:

Immediate is the simpler mode: the PV is created as soon as the PVC is created, and the Pod must run on a node that can reach the PV’s zone. A cluster with multiple zones and Immediate binding can have a Pod that is Pending because the PV is in a zone the Pod’s node selector cannot match.

WaitForFirstConsumer is the production-friendly mode: the PV is created when the first Pod is scheduled, and the PV is anchored to the same zone as the Pod. The Pod can target a zone with a node selector, and the PV is created in that zone.

The diagnostic message

A storage-driven failure looks like:

0/5 nodes are available: 2 node(s) didn't match Pod's persistent volume zone,
                          3 node(s) had taint {dedicated=prod:NoSchedule}.

The first reason is the volume binding. The Pod wants a node in zone us-east-1a; the cluster has two nodes in that zone, but those nodes are also cordoned or the Pod does not tolerate their taints.

Or:

0/5 nodes are available: 5 node(s) didn't match Pod's persistent volume zone.

The PV is in zone us-east-1c; the cluster has no nodes in that zone. The Pod is rejected everywhere.

Or:

0/5 nodes are available: 5 Insufficient memory.

The Pod’s resource requests exceed the node. If the Pod also has a PVC, the diagnosis is harder — the resource filter may be rejecting first, hiding the volume issue.

# Substitute your own values before running:
POD=web-5f9c7d8b6c-2xk9p
PVC=data-pvc

kubectl get pod "$POD" -o yaml | grep -A 5 "persistentVolumeClaim"
kubectl get pvc "$PVC"

NAME           STATUS    VOLUME                                     CAPACITY   ACCESS MODES
data-pvc       Bound     pvc-1234-5678-90ab-cdef                   10Gi       RWO

NAME                   CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                STORAGECLASS   REASON   AGE
pvc-1234-5678-90ab     10Gi       RWO            Delete           Bound    default/data-pvc      standard                5m

The PVC is bound. The PV is in zone us-east-1a (from the PV’s spec). The Pod’s node selector is topology.kubernetes.io/zone: us-east-1b. The mismatch is the cause.

The topology mismatch

The PV carries a node affinity in spec.nodeAffinity.required.nodeSelectorTerms. The affinity is added by the provisioner when the PV is created. For WaitForFirstConsumer, the affinity is set to the zone of the Pod that triggered the creation. For Immediate, the affinity is set to the zone of the storage backend.

# PV name from the VOLUME column of the `kubectl get pvc` output above:
PV=pvc-1234-5678-90ab

kubectl get pv "$PV" -o jsonpath='{.spec.nodeAffinity}' | jq

{
  "required": {
    "nodeSelectorTerms": [
      {
        "matchExpressions": [
          {
            "key": "topology.kubernetes.io/zone",
            "operator": "In",
            "values": ["us-east-1a"]
          }
        ]
      }
    ]
  }
}

The PV is zone-locked. The Pod must run on a node in us-east-1a. The scheduler’s VolumeBinding plugin applies the same logic as the NodeAffinity plugin: the node must match the affinity terms.

The unbound PVC

A Pod that references a PVC that is not yet bound is not a scheduling failure. The Pod is Pending because the PVC is Pending. The scheduler queues the Pod until the PVC is bound. The fix is to address the PVC’s Pending state.

# Substitute your own value before running:
PVC=data-pvc

kubectl get pvc "$PVC"

NAME           STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-pvc       Pending                                      standard       5m

The PVC is Pending. The reasons:

• The StorageClass’s provisioner has not created the PV.
• The StorageClass’s parameters are invalid.
• The cluster has no nodes that satisfy the PVC’s allowedTopologies.
• The CSI driver is failing (driver is not running, or the credentials are wrong).

The diagnostic is the PVC’s Status.Conditions and Events:

# Substitute your own value before running:
PVC=data-pvc

kubectl describe pvc "$PVC" | grep -A 10 "Events:"

The fix is at the storage layer, not the scheduler.

The RWO mode and the multi-node failure

A PVC with ReadWriteOnce can be mounted on a single node at a time. If the Pod is replicated and the scheduler needs to run two Pods on two nodes, the second Pod is rejected by the VolumeBinding filter because the volume is already in use.

0/5 nodes are available: 4 node(s) had PersistentVolumeClaim `data-pvc` bound to a node that is not in the current set of nodes.

The fix is to use ReadWriteMany (RWX) for the PVC, or to use a different storage backend that supports RWX. NFS and CSI drivers with RWX support are the answer.

The fix patterns

1. Use WaitForFirstConsumer binding. The PV is created when the first Pod is scheduled, anchored to the right zone.
2. Match the Pod’s zone selector to the PV’s zone. If the PV is in us-east-1a, the Pod must have a node selector or affinity that matches us-east-1a.
3. Use ReadWriteMany for replicated workloads. A PVC that needs to be mounted on multiple nodes must be RWX.
4. Validate the PVC’s provisioner is running. A broken CSI driver is the most common silent failure. The events on the PVC will show the provisioner’s error.
5. Audit the storage class. A misconfigured StorageClass (wrong parameters, missing secrets) is the most common cause of a PVC that never binds.

Quiz

Production discipline

• Default to WaitForFirstConsumer. Multi-zone clusters should not use Immediate binding. The PV should be anchored to the Pod’s zone.
• Match the Pod’s selector to the cluster’s topology. A Pod that targets a specific zone must have a node selector that matches the zone. A cluster with three zones and a Pod that targets only one zone has the scheduler picking from one-third of the nodes.
• Audit CSI drivers at every release. A CSI driver that fails silently is a cluster that cannot provision storage. The events on a ProvisioningFailed PVC are the only signal.
• Watch the unbound PVC count. A rising Pending PVC count is a storage platform problem. The operator should alert on kube_persistentvolumeclaim_status_phase{phase="Pending"}.
• Plan for binding-mode migrations. A cluster that changes volumeBindingMode does not retroactively re-bind existing PVs. The migration requires a StatefulSet re-deploy or a PVC re-create.