Lab 18: Diagnose a PVC stuck in Pending

Objective

By the end of this lab you will have five PersistentVolumeClaims in Pending on one cluster at the same time, stuck for five different reasons, and a written procedure that tells them apart. Two of the five emit the same event message, word for word, from the same controller — and separating those two is the point of the whole exercise.

kubectl get pvc shows you Pending and nothing else. The lab is about what you run next, in what order, and what each answer eliminates.

Architecture

One namespace, one dynamic provisioner, and five claims that will never bind until you intervene — plus a sixth failure that produces no claim at all.

flowchart TD
    subgraph NS[namespace lab18]
        A["pending-a · class fast-ssd"]
        B["pending-b · no class named"]
        C["pending-c · class local-path + consumer Pod"]
        D["pending-d · class local-path, no consumer"]
        E["pending-e · class empty string"]
    end
    A -.->|class does not exist| X1[ProvisioningFailed]
    B -.->|no class, no default| X2[FailedBinding]
    C -.->|provisioner scaled to zero| X3[ExternalProvisioning]
    D -.->|nothing is wrong| X4[WaitForFirstConsumer]
    E -.->|static PV too small| X5[FailedBinding]
    PV["PV lab18-static · 100Mi · Available"] --- E

pending-b and pending-e share a reason and a message. Everything in Task 7 follows from that.

Requirements

• A disposable kubeadm cluster, one control-plane node and two workers, Kubernetes 1.34.x, built per the Part LXXIV lessons or Lab 01, all nodes Ready.
• kubectl 1.34.x with cluster-admin. The lab creates cluster-scoped objects: a StorageClass, a PersistentVolume, and a storage provisioner in its own namespace.
• A cluster with no StorageClass before you begin. Task 1 checks. If one is already installed and marked default, fault B cannot be produced — a default class is precisely what fault B is the absence of. Record and remove the default annotation before starting, and put it back in Cleanup.
• Ability to pull docker.io/rancher/local-path-provisioner, docker.io/library/busybox and busybox:1.37. The provisioner runs a short-lived busybox helper Pod for each volume.
• Roughly 500 MiB free under /opt on each worker. The lab provisions two small volumes and Cleanup removes them.
• No out-of-band access requirement, and no SSH requirement. Every observation in this lab is available through the API. That is deliberate: this is the diagnostic you will run at 03:00 from a laptop with a kubeconfig and nothing else.
• Blast radius: one namespace plus three cluster-scoped objects. Nothing touches kube-system, the CNI, the kubelet or any node’s configuration.

Scenario

A deploy went out an hour ago. Six services in one namespace, four of them healthy, two Pods stuck in Pending. Somebody has already looked and reported “it’s a storage problem” — which is true and useless. kubectl get pvc -n lab18 returns five claims in Pending, submitted at various times by various people, and nobody knows which of them are new, which have been like that for a week and which are supposed to be like that.

The instinct is to fix the first one and see what happens. The discipline is to collect the evidence for all five before changing anything, because three of these have different causes that produce visually identical rows in kubectl get pvc, and one of them is not broken at all.

Tasks

Task 1: Build the baseline and prove one claim binds

Nothing you observe later means anything unless the cluster can bind a claim when nothing is wrong. Establish that first.

WORKDIR="$HOME/k8s-lab18"
mkdir -p "$WORKDIR"
cd "$WORKDIR"

kubectl get nodes -o wide  | tee nodes.pre-lab.txt
kubectl get storageclass   | tee storageclass.pre-lab.txt
kubectl get pv             | tee pv.pre-lab.txt
kubectl get ns             > ns.pre-lab.txt

storageclass.pre-lab.txt and pv.pre-lab.txt should both say No resources found. If a StorageClass is already installed, see Requirements.

Install a working dynamic provisioner:

cd "$HOME/k8s-lab18"
LPP_VERSION=v0.0.37
curl -fsSLo local-path-storage.yaml \
  "https://raw.githubusercontent.com/rancher/local-path-provisioner/$LPP_VERSION/deploy/local-path-storage.yaml"
kubectl apply -f local-path-storage.yaml
kubectl -n local-path-storage rollout status deployment/local-path-provisioner --timeout=180s

kubectl create namespace lab18
kubectl get storageclass

The local-path class is present and is not marked (default) — that is how the manifest ships, and fault B depends on it staying that way.

Now the control. Write control.yaml:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: control
  namespace: lab18
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: local-path
  resources:
    requests:
      storage: 64Mi
---
apiVersion: v1
kind: Pod
metadata:
  name: control
  namespace: lab18
spec:
  containers:
    - name: shell
      image: busybox:1.37
      command: ["sleep", "infinity"]
      volumeMounts:
        - name: data
          mountPath: /data
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: control

$ kubectl apply -f control.yaml

kubectl -n lab18 wait --for=condition=Ready pod/control --timeout=180s
kubectl -n lab18 get pvc control
kubectl get pv

control reaches Bound and its Pod reaches Running. Leave both in place for the rest of the lab. Every time an observation later looks alarming, this claim is the answer to “is the cluster’s storage path working at all”, and it costs one command.

Task 2: Walk the diagnostic while nothing is wrong

There are five commands in the PVC-Pending diagnostic. Run all five now, against a claim that is healthy, so that you know what each one looks like when the answer is “not this”.

cd "$HOME/k8s-lab18"

# 1. The claim itself: phase, class, volume, and the events on it.
kubectl -n lab18 describe pvc control

# 2. The classes that exist, and which one is default.
kubectl get storageclass

# 3. The volumes that exist, and their status.
kubectl get pv

# 4. The provisioner named by the claim's class.
kubectl -n local-path-storage get pods

# 5. The consumer, if there is one.
kubectl -n lab18 get pods -o wide

What each one can and cannot tell you:

• describe pvc is the only command that reports which controller has looked at the claim. The event’s From column names it — persistentvolume-controller for everything in this lab. The Reason column tells you how far the controller got before it stopped. This is always the first command and it is never the last.
• get storageclass answers “does the thing this claim asked for exist”, and separately “is there a fallback for a claim that asked for nothing”. Those are two different questions and one command answers both, which is why it is second.
• get pv answers “is there already a volume that should have matched”. On a cluster with dynamic provisioning most people skip it. Fault E is the reason not to.
• The provisioner Pods answer “is anything listening”, and only matter once the first three have established that the claim named a real class with a real provisioner.
• The consumer matters for exactly one reason: on a WaitForFirstConsumer class, the absence of a Pod is not a fault, and its presence changes which event the claim carries.

Task 3: Break it five ways at once

In production you do not get one Pending PVC to think about. You get a list. Write faults.yaml:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pending-a
  namespace: lab18
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: fast-ssd
  resources:
    requests:
      storage: 64Mi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pending-b
  namespace: lab18
spec:
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 64Mi
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pending-c
  namespace: lab18
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: local-path
  resources:
    requests:
      storage: 64Mi
---
apiVersion: v1
kind: Pod
metadata:
  name: consumer-c
  namespace: lab18
spec:
  containers:
    - name: shell
      image: busybox:1.37
      command: ["sleep", "infinity"]
      volumeMounts:
        - name: data
          mountPath: /data
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: pending-c
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pending-d
  namespace: lab18
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: local-path
  resources:
    requests:
      storage: 64Mi
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: lab18-static
spec:
  capacity:
    storage: 100Mi
  accessModes: ["ReadWriteOnce"]
  persistentVolumeReclaimPolicy: Retain
  storageClassName: ""
  hostPath:
    path: /opt/lab18-static
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pending-e
  namespace: lab18
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: ""
  resources:
    requests:
      storage: 1Gi

Two of the five faults need a second action that YAML cannot express. Fault C requires the provisioner to be down:

$ kubectl -n local-path-storage scale deployment/local-path-provisioner --replicas=0

$ kubectl apply -f faults.yaml

Give the controllers thirty seconds, then take the whole picture at once:

cd "$HOME/k8s-lab18"
kubectl -n lab18 get pvc | tee pvc-all.txt

for P in pending-a pending-b pending-c pending-d pending-e; do
  echo "===== $P"
  kubectl -n lab18 describe pvc "$P" | sed -n '/^Events:/,$p'
done | tee describe-all.txt

$ kubectl -n lab18 get pvc

NAME        STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
control     Bound     pvc-0a1c9f7d-2e44-4c0a-93bd-0c6f4b2e88a1   64Mi       RWO            local-path     6m
pending-a   Pending                                                                         fast-ssd       35s
pending-b   Pending                                                                                        35s
pending-c   Pending                                                                         local-path     35s
pending-d   Pending                                                                         local-path     35s
pending-e   Pending                                                                                        35s

The only thing that column view distinguishes is which class each claim named, and pending-c and pending-d named the same one. Read describe-all.txt instead. Five claims, five different event lines. The next four tasks take them in order of how much the event tells you.

Task 4: Fault A — the event that names its own cause

kubectl -n lab18 describe pvc pending-a | sed -n '/^Events:/,$p'

$ kubectl -n lab18 describe pvc pending-a | sed -n '/^Events:/,$p'

Events:
Type     Reason              Age                From                         Message
----     ------              ----               ----                         -------
Warning  ProvisioningFailed  8s (x4 over 35s)   persistentvolume-controller  storageclass.storage.k8s.io "fast-ssd" not found

This is the easy one, and it is worth being precise about why it is easy. The claim named a class, so the controller went to look one up; the lookup failed with a not-found error; the controller put that error verbatim into the event. The message is the API error, not a summary of it, which is why it names the exact resource type and the exact string that was not found.

Confirm the second half rather than assuming it:

kubectl get storageclass

fast-ssd is not in the list. There are two fixes and they are not equivalent:

• The claim is wrong. Someone copied a manifest from a cluster that had a fast-ssd class. Fix the claim. spec.storageClassName is immutable on a bound PVC but this one has never bound, and in practice the reliable move is to delete and re-apply the claim with the right name.
• The cluster is wrong. The class is supposed to exist and does not, because a storage driver was uninstalled or never installed on this cluster. Fix the cluster, and expect other claims to be affected too: kubectl get pvc -A -o json filtered on that class name will find them.

Fix it, and watch it bind:

cd "$HOME/k8s-lab18"
kubectl -n lab18 get pvc pending-a
kubectl -n lab18 delete pvc pending-a

Read the STATUS column in that first command before running the second. It says Pending, so the delete costs nothing.

Do not re-create it yet. The provisioner is still scaled to zero from Task 3, so a corrected pending-a would immediately become a fault C instead — which is itself the observation worth having: the order you fix things in changes what the next symptom looks like, and a cluster with two faults will hand you the second one dressed as a failed fix. Task 5 restores the provisioner and then re-creates this claim.

Task 5: Fault C — the event that names a component, not a defect

kubectl -n lab18 describe pvc pending-c | sed -n '/^Events:/,$p'
kubectl -n lab18 get pod consumer-c

$ kubectl -n lab18 describe pvc pending-c | sed -n '/^Events:/,$p'

Events:
Type    Reason                Age                From                         Message
----    ------                ----               ----                         -------
Normal  WaitForFirstConsumer  40s                persistentvolume-controller  waiting for first consumer to be created before binding
Normal  ExternalProvisioning  5s (x5 over 38s)   persistentvolume-controller  Waiting for a volume to be created either by the external provisioner 'rancher.io/local-path' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.

Two events, in order, and the order is the story. The class is WaitForFirstConsumer, so the first thing the controller did was wait. Then consumer-c was scheduled, the scheduler recorded which node it chose, and the controller handed the claim to the external provisioner named in the class — and has been waiting ever since.

Three things to take from the message itself:

• The event type is Normal, not Warning. Nothing has failed. The controller’s job here is to delegate and wait, and it is doing that correctly. An alert that only looks at Warning events misses this completely.
• It names the provisioner string, rancher.io/local-path. That is the value from the StorageClass, and it is what you take to the next command.
• It tells you what to check, in the last sentence, which is unusually helpful and is the reason this event is repeated verbatim in so many runbooks.

Follow it:

kubectl get storageclass local-path -o jsonpath='{.provisioner}{"\n"}'
kubectl -n local-path-storage get deployment,pods
kubectl -n lab18 get pod consumer-c -o wide

$ kubectl -n local-path-storage get deployment,pods

NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/local-path-provisioner   0/0     0            0           9m

No resources found in local-path-storage namespace.

Note what did not happen. control is still Bound and its Pod is still Running: an already-provisioned volume does not care whether the provisioner exists. A provisioner outage is invisible until somebody creates a new claim, which means it is usually discovered during a deploy, by a team who assume they broke it.

kubectl -n lab18 get pvc control

Restore it:

$ kubectl -n local-path-storage scale deployment/local-path-provisioner --replicas=1

kubectl -n local-path-storage rollout status deployment/local-path-provisioner --timeout=180s
kubectl -n lab18 get pvc pending-c
kubectl -n lab18 get pod consumer-c

pending-c binds and consumer-c starts, with no action taken on either object. That is the signature of this fault class: the claim was never wrong, so nothing about the claim needed changing. Any fix that involved editing, deleting or recreating pending-c would have “worked” too, and would have taught the operator the wrong lesson and left the real outage in place for the next claim.

Now re-apply fault A’s claim with the corrected class, and confirm it binds too:

cd "$HOME/k8s-lab18"
kubectl -n lab18 create -f - <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pending-a
  namespace: lab18
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: local-path
  resources:
    requests:
      storage: 64Mi
EOF

kubectl -n lab18 get pvc pending-a

pending-a is Pending again — and now with a WaitForFirstConsumer event, because it has no Pod. Which is Task 6.

Task 6: Fault D — the Pending that is not a fault

kubectl -n lab18 describe pvc pending-d | sed -n '/^Events:/,$p'

$ kubectl -n lab18 describe pvc pending-d | sed -n '/^Events:/,$p'

Events:
Type    Reason                Age   From                         Message
----    ------                ----  ----                         -------
Normal  WaitForFirstConsumer  2m    persistentvolume-controller  waiting for first consumer to be created before binding

pending-d has been Pending for the whole lab and there is nothing to fix. The class is WaitForFirstConsumer; no Pod references the claim; the controller is deferring binding exactly as configured. It will sit here indefinitely and that is correct.

Confirm the two facts that make it correct, rather than trusting the event:

kubectl get storageclass local-path \
  -o jsonpath='{.volumeBindingMode}{"\n"}'

kubectl -n lab18 get pods -o json \
  | jq -r '.items[] | . as $p | .spec.volumes[]?
           | select(.persistentVolumeClaim != null)
           | "\($p.metadata.name)\t\(.persistentVolumeClaim.claimName)"'

The first prints WaitForFirstConsumer. The second lists every Pod in the namespace with the claim it mounts, and pending-d is not among them.

There is one refinement worth knowing. If a Pod referencing the claim exists but has not been scheduled, the same controller emits a different message naming that Pod — waiting for pod NAME to be scheduled. So the reason column distinguishes three states, not two: no consumer at all, a consumer that cannot be scheduled, and a consumer that was scheduled and is waiting on provisioning. The middle one is a scheduling problem wearing a storage costume, and its real evidence is on the Pod.

Task 7: Faults B and E — one message, two causes

These two are the reason the lab exists.

cd "$HOME/k8s-lab18"
kubectl -n lab18 describe pvc pending-b | sed -n '/^Events:/,$p'
kubectl -n lab18 describe pvc pending-e | sed -n '/^Events:/,$p'

$ kubectl -n lab18 describe pvc pending-b | sed -n '/^Events:/,$p'

Events:
Type    Reason         Age                 From                         Message
----    ------         ----                ----                         -------
Normal  FailedBinding  12s (x9 over 2m)    persistentvolume-controller  no persistent volumes available for this claim and no storage class is set

$ kubectl -n lab18 describe pvc pending-e | sed -n '/^Events:/,$p'

Events:
Type    Reason         Age                 From                         Message
----    ------         ----                ----                         -------
Normal  FailedBinding  12s (x9 over 2m)    persistentvolume-controller  no persistent volumes available for this claim and no storage class is set

Identical. Same reason, same message, same controller, same event type. The two claims have nothing in common except that neither of them named a usable class: pending-b omitted the field, pending-e set it to the empty string, which means “do not provision anything for me, bind me to an existing volume”.

Two commands separate them, and neither is describe pvc:

kubectl get storageclass
kubectl get pv

$ kubectl get pv

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM             STORAGECLASS   AGE
lab18-static                               100Mi      RWO            Retain           Available                                          3m
pvc-0a1c9f7d-2e44-4c0a-93bd-0c6f4b2e88a1   64Mi       RWO            Delete           Bound       lab18/control     local-path     9m
pvc-6c2d8b41-77aa-41ba-9e30-5d18a0f39c22   64Mi       RWO            Delete           Bound       lab18/pending-c   local-path     4m

lab18-static is Available, its STORAGECLASS column is empty — which is the empty string pending-e asked for — and its capacity is 100Mi against a request of 1Gi. That is the whole answer, and it took one command that the event never suggested running.

Work the four matching criteria explicitly, because this is the check you will repeat for every static-binding failure:

kubectl get pv lab18-static \
  -o jsonpath='class={.spec.storageClassName} cap={.spec.capacity.storage} modes={.spec.accessModes}{"\n"}'

kubectl -n lab18 get pvc pending-e \
  -o jsonpath='class={.spec.storageClassName} req={.spec.resources.requests.storage} modes={.spec.accessModes}{"\n"}'

The PV’s capacity must be greater than or equal to the request. It is not, so no PV matched, so the default branch ran, so the message blamed the missing class. Fix the request:

cd "$HOME/k8s-lab18"
kubectl -n lab18 delete pvc pending-e
kubectl -n lab18 create -f - <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pending-e
  namespace: lab18
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: ""
  resources:
    requests:
      storage: 64Mi
EOF

kubectl -n lab18 get pvc pending-e
kubectl get pv lab18-static

pending-e binds to lab18-static, and the PV’s capacity stays 100Mi — a claim gets the volume it bound to, not the size it asked for. Both facts are worth writing down.

Now fault B. Its second half is the true one:

kubectl get storageclass

No class is marked (default), and pending-b named none. Two fixes, and as with fault A they are not equivalent:

• Name the class in the claim. Correct, explicit, and affects nothing else.
• Mark a class default. Fixes this claim and every other classless unbound claim in the cluster at the same time, in every namespace, because default assignment is retroactive.

Take the second one deliberately, so you can watch the blast radius:

$ kubectl annotate storageclass local-path storageclass.kubernetes.io/is-default-class="true" --overwrite

kubectl -n lab18 get pvc pending-b -o jsonpath='{.spec.storageClassName}{"\n"}'
kubectl -n lab18 describe pvc pending-b | sed -n '/^Events:/,$p'
kubectl -n lab18 get pvc

pending-b now carries storageClassName: local-path, which you did not write, and its event has changed to WaitForFirstConsumer — it has moved from fault B to fault D. Note that carefully: the fix did not bind the claim. It moved the claim into the state where it is waiting for a consumer, and an operator who stops watching at “the error went away” will report the incident closed while the workload is still down.

Task 8: The storage failure with no Pending PVC to look at

Not every storage fault produces a claim you can describe. Apply a quota:

cd "$HOME/k8s-lab18"
kubectl -n lab18 create quota lab18-storage \
  --hard=requests.storage=512Mi,persistentvolumeclaims=8
kubectl -n lab18 describe quota lab18-storage

Now ask for more than is left:

$ kubectl -n lab18 create -f - <<'EOF'
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: pending-f
namespace: lab18
spec:
accessModes: ["ReadWriteOnce"]
storageClassName: local-path
resources:
  requests:
    storage: 5Gi
EOF

Error from server (Forbidden): error when creating "STDIN": persistentvolumeclaims "pending-f" is forbidden: exceeded quota: lab18-storage, requested: requests.storage=5Gi, used: requests.storage=384Mi, limited: requests.storage=512Mi

Typed at a terminal this is unmissable. Inside a controller it is invisible: a StatefulSet’s volumeClaimTemplates, a Helm hook, or an operator creating claims on your behalf gets the same rejection, and the only trace is an event on the object that tried, not on a PVC that does not exist.

kubectl -n lab18 get events --sort-by=.lastTimestamp | tail -20
kubectl -n lab18 describe quota lab18-storage

Remove the quota before the cleanup, so it does not interfere:

kubectl -n lab18 delete resourcequota lab18-storage

Task 9: Write the triage order down

You now have five worked faults and one that is not a PVC at all. Write the order out, and annotate each step with what it rules out — not what it fixes. A step that only tells you what to do next is worth less than a step that eliminates a branch.

0. Does a claim exist at all?
   kubectl -n NS get pvc
   No claim  -> not a PVC problem. Go to describe quota and the event log.

1. What has looked at the claim, and how far did it get?
   kubectl -n NS describe pvc NAME
   WaitForFirstConsumer  -> healthy, or a Pod that cannot schedule. STOP.
   ProvisioningFailed    -> the message is the API error. Read it literally.
   ExternalProvisioning  -> claim is fine; the named provisioner is not
                            answering. Rules out every claim-side cause.
   FailedBinding         -> ambiguous. Both remaining branches are live.

2. Does the class the claim asked for exist, and is there a default?
   kubectl get storageclass
   Rules out: wrong class name, missing default.

3. Is there already a volume that should have matched?
   kubectl get pv
   Compare class, capacity, access modes, selector - in that order.
   Rules out: static binding mismatch.

4. Is the provisioner running?
   kubectl -n PROVISIONER_NS get pods
   Only reachable if steps 2 and 3 found nothing. Rules out: outage.

5. Did the fix bind the claim, or only change its error?
   kubectl -n NS get pvc NAME
   Bound, with a Pod Running. Anything else is not done.

Step 5 is the one people skip, and Task 7 is the proof that it matters: setting a default class made fault B’s error disappear and left the claim Pending.

Validation

cd "$HOME/k8s-lab18"

# 1. Every fault produced a distinguishable event, and you captured it.
grep -c "Reason" describe-all.txt
grep -E "ProvisioningFailed|ExternalProvisioning|WaitForFirstConsumer|FailedBinding" describe-all.txt

# 2. The two identical messages are both in the capture.
grep -c "no persistent volumes available for this claim" describe-all.txt

# 3. The claims that should be bound are bound.
kubectl -n lab18 get pvc

# 4. The claim that should still be Pending, is - and for the right reason.
kubectl -n lab18 describe pvc pending-d | sed -n '/^Events:/,$p'

# 5. The static PV is bound to the claim you resized.
kubectl get pv lab18-static

Expected results:

• describe-all.txt contains all four distinct event reasons: ProvisioningFailed, FailedBinding, ExternalProvisioning and WaitForFirstConsumer.
• The FailedBinding message appears exactly twice in the capture: once for pending-b and once for pending-e.
• control, pending-c and pending-e are Bound.
• pending-a, pending-b and pending-d are Pending, and each carries a WaitForFirstConsumer event and nothing else. All three are now the same fault — the healthy one — which is the point: pending-a was fixed and has no consumer, pending-b was given a default class and has no consumer, and pending-d never had one. Three different histories, one identical end state, and only the history explains which of them anyone should still care about.
• lab18-static shows Bound with lab18/pending-e in its CLAIM column and capacity still 100Mi.

Expected Outcome

A namespace whose storage faults you can name individually, and a working directory that proves you collected the evidence before acting:

k8s-lab18/
├── local-path-storage.yaml
├── control.yaml
├── faults.yaml
├── pvc-all.txt
├── describe-all.txt
├── nodes.pre-lab.txt
├── storageclass.pre-lab.txt
├── pv.pre-lab.txt
└── ns.pre-lab.txt

You can state, with a transcript behind each: which event reason each of the five causes produces, which two are indistinguishable from the event alone, which single command separates them, which Pending claim must be left alone, and which storage fault never produces a Pending claim at all.

Troubleshooting

pending-c binds immediately instead of hanging. The provisioner was not actually scaled to zero when faults.yaml was applied. Check kubectl -n local-path-storage get deployment and redo Task 3 in order: scale down first, apply second.

pending-b binds instead of staying Pending. A default StorageClass exists. kubectl get storageclass will show (default) on one row. Remove the annotation with --overwrite to "false", delete and re-create pending-b.

pending-e shows ProvisioningFailed instead of FailedBinding. The claim has storageClassName absent rather than set to "", and a default class exists, so it is being provisioned rather than matched. The empty string and the absent field are different: the empty string means “never provision for me”.

describe pvc shows no events at all. Events expire — the default retention is one hour. Re-create the claim to regenerate them, and capture the output this time rather than reading it. This is the practical reason Task 3 writes describe-all.txt to a file.

The jq command in Task 6 prints nothing. That is the expected result for pending-d. If it prints nothing for every claim, check that jq is installed and that Pods exist: kubectl -n lab18 get pods.

Cleanup

cd "$HOME/k8s-lab18"
kubectl -n lab18 delete resourcequota lab18-storage --ignore-not-found
kubectl delete namespace lab18

Deleting the namespace deletes the claims, which triggers each volume’s reclaim policy. The local-path volumes are Delete and go away by themselves, provided the provisioner is running — which is why it must be scaled back up before this point, and why Task 5 restored it.

kubectl -n local-path-storage get deployment local-path-provisioner
kubectl get pv

$ kubectl delete pv lab18-static --ignore-not-found

Undo the cluster-wide change from Task 7 before removing the provisioner, so that the class is not default at the moment it is deleted:

kubectl annotate storageclass local-path \
  storageclass.kubernetes.io/is-default-class="false" --overwrite
kubectl delete -f local-path-storage.yaml

kubectl get storageclass
kubectl get pv
kubectl get ns
kubectl get nodes -o wide

Compare each against its pre-lab capture. On a cluster that started clean, kubectl get storageclass and kubectl get pv both return No resources found.

Production notes

Capture before you act, and make it cheap enough that you actually do. Every fix in this lab destroys the evidence for the diagnosis. One loop over kubectl describe pvc into a file, as in Task 3, costs ten seconds and is the difference between a post-incident review that identifies a cause and one that records “storage was broken, we recreated the claims”. Put that loop in the runbook as step zero, not as an afterthought.

Two of these faults are cluster-wide and will be reported as one team’s problem. A provisioner outage (fault C) and a missing default class (fault B) both surface as one namespace’s PVC being Pending, because that is the team who happened to deploy. Before fixing the claim in front of you, run kubectl get pvc -A | grep Pending — if the list has claims from namespaces you were not called about, the incident is bigger than the ticket and the comms are different.

A fix that changes the error is not a fix. Task 7 ends with a claim whose error is gone and which is still not bound. In a change window, the exit criterion for a storage remediation is Bound plus a Running Pod that has the volume mounted, verified by reading a file through it. Anything short of that is a state change, not a resolution.

“Hold” is the right answer more often here than anywhere else in storage. A Pending PVC harms nothing by itself: no data is at risk, no volume is degrading, and the blast radius of waiting is one workload that is already down. That is not true of the fixes. Deleting a claim can destroy a volume; setting a default class reaches every namespace; increasing a quota commits capacity you may not have. If the evidence is ambiguous at 03:00, holding until the storage owner is awake costs one service’s availability and risks nothing — say so explicitly, name the owner, and set a time.

Alert on the discriminator, and alert on the invisible one. Pending-PVC alerts must exclude WaitForFirstConsumer or on-call will learn to ignore them. Separately, nothing in a phase-based alert catches Task 8 at all: a quota that refuses claims produces no PVC and no PV, so add a rule on quota utilisation approaching its hard limit, before the rejection rather than after.

What You Learned

• kubectl get pvc distinguishes almost nothing. Five claims, five causes, and the list view separated them only by which class each named — and two of them named the same one.
• The event’s Reason tells you which branch of the controller ran. ProvisioningFailed carries a verbatim API error and is the most specific; ExternalProvisioning clears the claim entirely and points at a component; WaitForFirstConsumer means stop.
• FailedBinding is one message for two independent facts. pending-b and pending-e were indistinguishable from the event and separated instantly by kubectl get pv — a command the message never suggests.
• A Pending PVC can be healthy, and the alert that does not know that is the alert nobody reads.
• Some storage faults never produce a PVC. A quota rejection leaves a refused create and an event on the controller that tried, and the PVC ladder does not apply to it.
• Fixing the error and fixing the fault are different outcomes. Setting a default class removed pending-b’s message and left the claim unbound, which is exactly what a premature “resolved” looks like.