Skip to main content
RunBook Academy

← All break/fix scenarios in Kubernetes

advancedkubernetes-pvc~35 min

PVC cannot bind

Reported symptoms

  • ●The analytics namespace was deleted by mistake and restored from Git; every Deployment came back, and the PostgreSQL StatefulSet did not
  • ●analytics-db-0 has been Pending for forty minutes with the scheduler message about unbound immediate PersistentVolumeClaims
  • ●kubectl describe pvc on the claim it needs shows no warning events at all - not a failure, not a retry, nothing
  • ●A 500Gi PersistentVolume with the right capacity, the right access mode and the right StorageClass name is sitting in the cluster, not bound to anything
  • ●A throwaway PVC created in a scratch namespace binds in under ten seconds, so storage provisioning is plainly working
  • ●The storage team confirms the NFS export is healthy and the database files are still on it, untouched

Evidence

  • · kubectl -n analytics get pvc shows data-analytics-db-0 as Pending with an empty VOLUME column
  • · kubectl -n analytics describe pvc data-analytics-db-0 prints an empty Events section
  • · kubectl get pv shows the 500Gi volume with STATUS Released and RECLAIM POLICY Retain
  • · The CLAIM column on that PV still reads analytics/data-analytics-db-0 - the claim that is Pending
  • · kubectl get pv with custom columns shows spec.claimRef carries a uid from the deleted claim
  • · The recreated PVC metadata.uid differs from the uid recorded in the PV claimRef
  • · kubectl get storageclass lists no object named nfs-retain, which is the class both the PV and the PVC name
Diagnosis and resolutionclick to reveal

Root cause

The PersistentVolume was created by hand for an existing NFS export and carries persistentVolumeReclaimPolicy Retain. Deleting the namespace deleted the PersistentVolumeClaim; Retain means the volume is left alone with its data intact, and the volume moves to the Released phase. Released is not a binding state. The control plane only considers volumes in the Available phase when it looks for a match, so the one volume that holds the data is invisible to the claim that wants it, and nothing about Released resolves itself with time. The volume also still carries spec.claimRef pointing at the deleted claim, including that claim's uid; the restored claim has the same name and namespace but a new uid, so the recorded reference no longer identifies it. Underneath all of that sits the reason the failure is silent: the storageClassName both objects name has no StorageClass object behind it, which is the ordinary shape of static provisioning. With no class there is no provisioner, with no provisioner there is nothing to fail, and with nothing failing there are no events. The absence of events was the loudest signal in the incident and it was read as an absence of information. A claim a provisioner cannot satisfy complains; a claim nothing is even attempting to satisfy says nothing at all.

Remediation

Establish first that this volume is the one that backs the right export, which is a question for the storage team and the NFS server rather than for the cluster, because the cluster cannot tell you whose data is on the other end of a path. Once that is settled, the recovery is a single field on the PersistentVolume: rewrite spec.claimRef so that it keeps namespace and name and drops uid and resourceVersion. That is the documented way to reserve a volume for one specific claim. It clears the stale identity that no longer matches anything, the volume returns to Available, and because the reservation still names the waiting claim no other claim in the cluster can win the race to it. Do not clear claimRef entirely unless you are certain nothing else is eligible, and do not delete and recreate the PersistentVolume - a hand-written volume that is deleted takes its access-mode, capacity and mount-option details with it, and the next person has to reconstruct them from memory. If the ownership of the export cannot be established inside the window you have, holding is a legitimate outcome and often the right one: a Pending claim is an outage, and a claim bound to the wrong export is a StatefulSet writing into somebody else's data. Give the hold an owner and a review time and say so on the incident channel rather than leaving it implied.

Verification

The claim reports Bound and names the volume, the volume reports Bound and names the claim, and the Pod leaves Pending - but none of those is the check that matters. The check that matters is that the database came up with its own data, measured against something agreed with the application owner before the change was made: a table list, a row count, a most-recent timestamp. A StatefulSet that starts cleanly on an empty volume looks exactly like a successful recovery until somebody queries it. Confirm as well that no second volume was provisioned while the incident was open, that no other volume for the same export is left in Released, and that the reservation you wrote is the one that took effect rather than an accidental match on capacity and access mode.

Prevention

Retain is the correct policy for a database and Retain means a human has to act after every claim deletion; budget for that work instead of being surprised by it. Alert on PersistentVolumes in Released and Failed, because both are decisions waiting on an operator rather than transient states, and alert on PersistentVolumeClaims that stay Pending for more than a few minutes so that a silent claim is not the thing nobody is watching. The deeper lesson is about what the restore actually restored: the claim is namespaced and lived in Git, the volume is cluster-scoped and did not, so replaying the repository rebuilt one half of a pair and left the other half in a state the replay could not reach. Any namespace holding stateful workloads needs a documented recovery path for the cluster-scoped objects that its manifests do not contain, and it needs protection against casual deletion in the first place. Finally, teach the crew to read silence: a Pending claim with no events is a claim nobody is working on, and that narrows the problem faster than any provisioner log.

Reported symptoms

Someone deleted the analytics namespace on Tuesday afternoon. The recovery was supposed to be uneventful: everything in that namespace is in Git, and re-applying the repository brought back the ConfigMaps, the Services, the two Deployments and the StatefulSet within a minute.

Everything came back running except the database.

analytics-db-0 has been Pending for forty minutes. The scheduler is explicit about why:

Warning  FailedScheduling  pod has unbound immediate PersistentVolumeClaims

That points straight at the claim, so the claim is where the shift starts. And the claim is where it stalls, because the claim has nothing to say.

Three things about the state of the cluster made this harder than it should have been:

  • Storage provisioning is obviously fine. A colleague created a throwaway 1Gi claim in a scratch namespace to test, and it bound in under ten seconds.
  • The volume is right there. kubectl get pv lists a 500Gi volume with the capacity, access mode and StorageClass name the claim asks for.
  • The data is safe. The storage team checked the NFS server directly: the export is healthy and the PostgreSQL data directory is intact.

So there is a claim that wants a volume, a volume that matches the claim, a working control plane, and forty minutes of nothing happening.

Evidence collected

Read-only / SafePending, and the VOLUME column is empty
$ kubectl -n analytics get pvc data-analytics-db-0
NAME                  STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
data-analytics-db-0   Pending                                             nfs-retain     41m

Illustrative output

Read-only / Safeno events at all - not one warning in 41 minutes
$ kubectl -n analytics describe pvc data-analytics-db-0 | tail -4
Access Modes:
VolumeMode:    Filesystem
Used By:       analytics-db-0
Events:        <none>

Illustrative output

Read-only / SafeReleased, not Available - and the CLAIM column names the claim that is Pending
$ kubectl get pv nfs-analytics-500g
NAME                 CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM                         STORAGECLASS   AGE
nfs-analytics-500g   500Gi      RWO            Retain           Released   analytics/data-analytics-db-0   nfs-retain     288d

Illustrative output

Read-only / Safethe volume still records which claim owned it, uid included
$ kubectl get pv nfs-analytics-500g -o custom-columns='PHASE:.status.phase,NS:.spec.claimRef.namespace,NAME:.spec.claimRef.name,UID:.spec.claimRef.uid'
PHASE      NS          NAME                  UID
Released   analytics   data-analytics-db-0   0d3f8b21-4c19-4a7e-9d55-5b0a6c1e7f42

Illustrative output

Read-only / Safethe claim that exists now, and its uid is not the one above
$ kubectl -n analytics get pvc data-analytics-db-0 -o jsonpath='{.metadata.uid}'
b7c40e9a-2f18-42d6-8a31-19c7f5d0ab63

Illustrative output

Read-only / Safethere is no StorageClass object called nfs-retain
$ kubectl get storageclass
NAME                 PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
standard (default)   ebs.csi.aws.com         Delete          WaitForFirstConsumer   true                   402d
db-ssd               ebs.csi.aws.com         Retain          WaitForFirstConsumer   true                   402d

Illustrative output

Work the evidence before reading on

The scratch claim bound in ten seconds and this one has not bound in forty minutes, so the control plane is working. Something about this pairing is different.

  1. Read the STATUS column on the volume again, carefully. There are four phases a PersistentVolume can be in. Which one is a candidate for binding?
  2. Compare the uid recorded on the volume with the uid of the claim that exists now. The name and namespace match. Does anything else?
  3. The claim names nfs-retain and kubectl get storageclass does not list it. What, concretely, is supposed to react to this claim?
  4. The scratch claim bound in ten seconds. Which StorageClass did it use, and what did that class have that this one does not?

Before continuing: the claim has raised no warnings in forty-one minutes. Is that because nothing has gone wrong, or because nobody is trying?

Root cause

1. Retain left the volume in a phase that cannot bind

A PersistentVolume moves through four phases: Available, Bound, Released and Failed. Available and Bound are the healthy pair. Released means the claim that owned this volume has been deleted and the volume has not been reclaimed.

Which of those happens on claim deletion is decided by persistentVolumeReclaimPolicy. This volume was written by hand for an existing NFS export and carries Retain, which is the right choice for a database: it says the data is too important to delete on a controller’s say-so, and an operator must decide what happens next.

The consequence is the part that surprised the shift. Released is not a state the control plane resolves on its own, and it is not a state a volume can bind from. The matching logic only ever looks at Available volumes. So the 500Gi volume that holds the database is, from the point of view of the binding logic, not in the pool at all - which is exactly what Retain is supposed to mean, because the alternative would be to hand a stranger’s data to whichever claim asked next.

2. The recorded owner is an object that no longer exists

spec.claimRef on a bound volume records which claim owns it: namespace, name, and uid. The uid is what makes the reference precise. Names are reused constantly - data-analytics-db-0 is generated from the StatefulSet’s volumeClaimTemplate and will be identical every time the StatefulSet is recreated - but a uid is minted per object and never reused.

The restore created a claim with the same name in the same namespace and a brand new uid. The volume is still pointing at the old one. The two objects agree on everything a human looks at and disagree on the only field the control plane uses to decide they are the same object.

3. Nothing was ever going to raise an event

Both objects name storageClassName: nfs-retain, and there is no StorageClass object with that name. That is not a mistake; it is the ordinary shape of static provisioning, where the class name is a label used to keep hand-written volumes and their claims paired and away from the dynamic pool.

But it means there is no provisioner in this story. Nothing is being asked to create a volume, nothing is being refused, nothing is retrying. A claim waiting on a broken dynamic provisioner accumulates ProvisioningFailed warnings and tells you where to look. A claim waiting on a static volume that is not Available produces silence, forever, and silence is easy to read as “no information” when it is in fact the diagnosis.

Resolution

  1. Establish that this volume backs the right export, with the storage team and against the NFS server. The cluster cannot tell you whose data is behind a path; only the backend can. Nothing below is safe until this is settled.
  2. Agree the acceptance test now, before touching anything: a table list, a row count, a most-recent timestamp - something the application owner will recognise as their data. Write it in the incident channel.
  3. Record the current state of the volume so the change is reversible: kubectl get pv nfs-analytics-500g -o yaml > /tmp/pv-before.yaml. Keep it for the duration of the incident.
  4. Rewrite spec.claimRef on the volume with kubectl edit pv nfs-analytics-500g, keeping namespace and name and removing uid and resourceVersion. This is the documented reservation form: the volume becomes Available, and available only to the claim it names.
  5. Watch the claim rather than the clock. The binding loop reconsiders it within seconds; kubectl -n analytics get pvc -w shows the transition to Bound as it happens.
  6. Confirm the pairing in both directions before the Pod starts: the claim names the volume in its VOLUME column, and the volume names the claim in its CLAIM column. A one-sided match means something else bound.
  7. Let the StatefulSet start. It needs no help - the Pod was Pending on the claim and the scheduler reconsiders it as soon as the claim binds.
  8. Run the acceptance test that was agreed in step two, with the application owner watching. Do not close on the Pod being Ready.
  9. If ownership of the export cannot be established, hold. A Pending claim is an outage; a claim bound to the wrong export is a StatefulSet writing into data that belongs to another system. Name who owns the hold and when it will be reviewed, and post that rather than leaving it implied.

Verification

  1. The claim reports Bound and names nfs-analytics-500g, and the volume reports Bound and names analytics/data-analytics-db-0. Check both directions; a match in one is not a binding.
  2. The reservation is what took effect. Read spec.claimRef back and confirm it now carries a uid matching the live claim, which is the control plane recording the binding it just made.
  3. The database has its own data. Run the acceptance test agreed before the change - table list, row count, most-recent timestamp - and have the application owner confirm it. This is the only check that distinguishes recovery from a clean start on an empty volume.
  4. No second volume was created while the incident was open. kubectl get pv should show one volume for this export, not two, and nothing new in the last hour.
  5. No other volume is stranded. Look for anything else in Released or Failed: kubectl get pv --field-selector=status.phase=Released. The namespace deletion may have taken more than one claim with it.
  6. The alert would have fired. Create a claim naming a class that does not exist, confirm the Pending alert fires within its threshold, and delete it. An alert nobody has seen fire is an assumption.
  7. The recovery is written down. The next person to hit this has minutes, not forty of them, and the reservation edit is not something to derive under pressure.

Prevention

  • Alert on PersistentVolumes in Released and Failed. Neither is transient. Both mean a decision is sitting unmade, and this incident is what an unmade decision looks like six weeks later.
  • Alert on PersistentVolumeClaims that stay Pending past a few minutes. A claim that fails loudly gets attention on its own; a claim that fails silently needs the alert more, not less.
  • Treat Retain as a commitment to operational work. It is the right policy for a database and it means someone must act every time a claim is deleted. A cluster with Retain and no cleanup process accumulates stranded volumes until one of them is in the path of a restore.
  • Know what your restore does not restore. Claims are namespaced and were in Git; volumes are cluster-scoped and were not. Replaying the repository rebuilt one half of a pair. Every namespace with stateful workloads needs a documented path for the cluster-scoped objects its manifests do not contain.
  • Make the namespace hard to delete by accident. This incident started with a kubectl delete namespace that should not have been possible against a namespace holding a production database.
  • Read silence as evidence. An empty Events section on a Pending claim is not missing information - it says no provisioner is involved, which eliminates most of the diagnostic ladder in one step.