Pod stuck Pending due to PVC not binding

Reported symptoms

It is month-end. The payments StatefulSet was scaled from three replicas to five at 07:41 to absorb the peak. Fifty minutes later payments-3 and payments-4 are still Pending and the tier is running on the capacity it had yesterday.

Three other things are open in the tracker, none of them assigned to the same person:

• The nightly end-to-end suite creates a throwaway namespace, deploys a fixture database into it, and waits fifteen minutes for it to come up. It has timed out for nineteen consecutive nights. The ticket says “flaky e2e, deprioritised”.
• Last Thursday’s disaster-recovery restore drill restored every object it was asked to restore and left six PVCs Pending. It was written up as a defect in the backup tool.
• A node was replaced eight days ago and the workload that had storage on it came back without a murmur, which is why “storage is fine” is the settled opinion in the channel.

Two facts point the other way, and they are the ones worth holding on to. Another team deployed a brand-new service with its own PVC at 09:10 this morning and it bound in four seconds. And the CSI controller is not complaining about anything: the Pods are Running, leader election is healthy, and its log has nothing to say about any of the stuck claims — no error, no retry, no mention of them at all.

Evidence provided

$ kubectl -n payments get pvc -o custom-columns=NAME:.metadata.name,PHASE:.status.phase,CLASS:.spec.storageClassName,CREATED:.metadata.creationTimestamp

NAME              PHASE     CLASS       CREATED
data-payments-0   Bound     standard    2026-05-02T09:14:11Z
data-payments-1   Bound     standard    2026-05-02T09:15:02Z
data-payments-2   Bound     standard    2026-05-02T09:15:51Z
data-payments-3   Pending   <none>      2026-08-18T07:41:33Z
data-payments-4   Pending   <none>      2026-08-18T07:41:34Z

$ kubectl -n payments describe pvc data-payments-3 | sed -n '/Events/,$p'

Events:
Type    Reason         Age                  From                         Message
----    ------         ----                 ----                         -------
Normal  FailedBinding  3m (x21 over 50m)    persistentvolume-controller  no persistent volumes available for this claim and no storage class is set

$ kubectl get storageclass

NAME         PROVISIONER       RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
bulk         csi.example.com   Delete          WaitForFirstConsumer   true                   21d
ssd-delete   csi.example.com   Delete          WaitForFirstConsumer   true                   21d
ssd-retain   csi.example.com   Retain          WaitForFirstConsumer   true                   21d
standard     csi.example.com   Delete          Immediate              true                   412d

$ kubectl -n payments describe pod payments-3 | sed -n '/Events/,$p'

Events:
Type     Reason            Age                   From               Message
----     ------            ----                  ----               -------
Warning  FailedScheduling  4m (x11 over 49m)     default-scheduler  0/24 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/24 nodes are available: 24 Preemption is not helpful for scheduling.

$ kubectl -n kube-system logs -l app=csi-controller -c csi-provisioner --since=1h | grep -c data-payments-3

0

Work the evidence before reading on

Every command above is read-only and every one of them is cheap. The order they are run in is the whole exercise.

1. Two claims created from the same volumeClaimTemplates disagree about exactly one field. Which field is it, and at what moment in a claim’s life is that field’s value decided?
2. The provisioner log says nothing about the stuck claims. Is that evidence that the provisioner is healthy, or evidence about which claims a provisioner is ever asked to service? Those are different statements.
3. The scheduler says the Pod has unbound immediate PersistentVolumeClaims, and every StorageClass in the cluster is WaitForFirstConsumer. What binding mode does a claim have when it has no class at all?
4. Nineteen nights of failures produced no alert. Look again at the Type column of the PVC event and say why.

Before continuing: name the single command that would have answered this in four seconds on day one, and say why nobody ran it.

Root cause

1. storageClassName is decided at admission, not at binding

A PVC that names no class is not resolved against the cluster default every time the binding controller looks at it. The DefaultStorageClass admission plugin writes the current default into spec.storageClassName as the claim is admitted, and after that the claim carries the answer.

That is why data-payments-0 reads standard and data-payments-3 reads nothing. They came from the same template. They were admitted three months apart, and the cluster changed in between.

Three weeks ago the platform team finished a storage-class rationalisation. The new explicit classes had been in place for a fortnight; the closing step of the change was to clear the is-default-class annotation from standard so that no class was default and every claim would have to say what it wanted. That is the discipline this course teaches, and the change was announced, reviewed and merged.

It broke nothing on the day, because every claim in the cluster had already been admitted. A change with a blast radius of zero at the moment it lands and a blast radius of everything afterwards is very hard to associate with its own consequences.

2. A classless claim is offered to nobody

The external-provisioner sidecar next to a CSI driver watches PVCs and acts on the ones whose StorageClass names its own provisioner. A claim with no class names no provisioner, so no sidecar ever picks it up.

The silence in the CSI log is therefore not an absence of evidence. It is the diagnosis, stated in the negative: this claim was never routed to anything that could have provisioned it. Reading that silence as “the provisioner is fine, look elsewhere” is what cost forty of the fifty minutes.

The scheduler’s message follows from the same fact. A claim with no class has no volumeBindingMode, and the scheduler’s volume-binding plugin treats that as immediate binding — hence pod has unbound immediate PersistentVolumeClaims in a cluster where every StorageClass on disk says WaitForFirstConsumer. The word sends readers to check binding modes, where they find nothing wrong, because the claim in question is not governed by any of them.

3. The event is Normal

FailedBinding is emitted by the persistentvolume controller with Type: Normal. Not Warning.

Every alert rule, dashboard and triage query in this cluster that filters events by severity discards it. The nineteen failed e2e runs, the restore drill and the scale-out all emitted this event, all night, for weeks, into a channel nobody was watching.

Three teams, three tickets, one annotation.

Resolution

1. Capture the blast radius first, while it is still visible: list every PVC in the cluster whose spec.storageClassName is null, with its namespace and creation timestamp. This list cannot be reconstructed after any repair.
2. Confirm no volume exists behind the Pending claims. kubectl get pv shows nothing bound to them; a Pending claim has never been provisioned, so deleting it destroys nothing. Run this check every time — the same delete aimed at a Bound claim is a data-loss operation.
3. Recreate the two payments claims with storageClassName set explicitly, taking the value from the sibling claims that bound before the change rather than from the current default. For a StatefulSet the claim name is fixed by ordinal, so recreating it under the same name is enough for the controller to find it.
4. Delete the Pending Pods so the StatefulSet controller recreates them against the repaired claims. The Pods have never started, so there is no state to lose and no drain to perform.
5. Repair the sources, not just the instances: name the class in the fixture chart the e2e suite deploys, and in the restore manifests. Those two produce new claims on a schedule and will reproduce the incident tonight otherwise.
6. Repair the StatefulSet template. volumeClaimTemplates is not a field the API server accepts an update to, so this means deleting the StatefulSet object with cascade set to orphan and re-applying it with the class named, letting the controller adopt the running Pods and existing claims. Rehearse it in staging: a replacement whose selector differs adopts nothing and starts from zero.
7. Add a validating policy that rejects a PersistentVolumeClaim with no storageClassName, and the matching CI check over the chart repository so the rejection happens at review time rather than at deploy time.
8. Only now decide whether the cluster should have a default class again. The answer may well be no — but it should be a decision taken with the inventory from step one in front of you, not a decision taken to close a ticket.

Verification

1. Each repaired claim bound to the same class as its siblings. All five payments claims read ssd-retain. Checking only that the phase is Bound reproduces the original mistake, because Bound is exactly what the dangerous repair also produces.
2. The nightly end-to-end suite passes on its own schedule, unattended. A hand-run proves nothing here: it is usually run in a namespace somebody has already touched.
3. The restore drill is re-run and leaves zero Pending claims. This is the one that was misattributed to the backup tool, and re-running it is what retracts that finding.
4. The guard can fail. Submit a PVC with no storageClassName into a scratch namespace and require the policy to reject it. A check that has only ever passed is untested.
5. A further scale-out binds unattended. Scale the StatefulSet to six and confirm the new claim binds with no intervention. This is the only step that tests the template rather than the claims repaired by hand.
6. The cluster-wide inventory of null-class claims is empty, or every remaining entry has a named owner and a date. Zero is the goal; a short list with owners is an acceptable interim and an unbounded list is the original condition.
7. A Pending-PVC alert exists and fires. Create a claim naming a class that does not exist, wait past the threshold, and confirm the alert arrives. The alert must not be built on warning-severity events, since this failure never produces one.

Prevention

• Every production PVC names its StorageClass. A claim that omits it inherits a cluster-wide annotation whose value is invisible from the manifest and can change without the manifest changing.
• Enforce it, do not document it. A validating policy that rejects a classless claim, plus a CI check over the charts, converts this from a standing instruction into a property of the cluster.
• Treat the default-class annotation as a cluster-wide change with a delayed blast radius. Before touching it, enumerate what creates claims from templates: volumeClaimTemplates, ephemeral test namespaces, restore tooling, operators that provision storage on your behalf. Those are the things that will find out, and they will find out later.
• Alert on PVCs Pending for more than five minutes, by phase and not by event severity. This failure emits Normal events only.
• Audit for null storageClassName on a schedule. One command, run weekly, tells you how much of the estate still depends on a default. Run after an incident it tells you only how much of it you already broke.
• When three teams open three tickets in the same fortnight, look at the change log for the fortnight before that. The common cause of unrelated symptoms is usually a change whose effect was deferred.