Lab 17: Dynamic provisioning with PVCs

Objective

By the end of this lab a PVC you submit will cause a PersistentVolume to come into existence that you never wrote, and you will be able to point at the exact directory on the exact node that it is. You will have produced, deliberately, the three states that a storage request can be stuck in — Pending because it is healthy, Pending because nothing is listening, and Pending because the volume and the Pod ended up on different nodes — and you will be able to tell them apart from their events alone.

Lab 19 builds static local PVs by hand, one per directory, and binds them with kubernetes.io/no-provisioner. This lab is the other half: the same node storage, created on demand, by a controller, from a template. The interesting question is not how to write a StorageClass — it is which of its fields have teeth and which are polite requests to a driver that may ignore them.

Architecture

One namespace, one provisioner, three StorageClasses over the same backing directory, and a workload that mounts the result.

flowchart TD
    P[PVC in namespace lab17] -->|storageClassName| SC[StorageClass]
    SC -->|provisioner rancher.io/local-path| PR[local-path-provisioner Deployment]
    PR -->|creates helper Pod| H[helper-pod busybox on the target node]
    H -->|mkdir| DIR["/opt/local-path-provisioner/pvc-uid_lab17_name"]
    PR -->|creates| PV[PersistentVolume pvc-uid]
    PV -->|nodeAffinity kubernetes.io/hostname| N[the node that got the directory]
    P -->|Bound| PV
    W[Pod web] -->|mounts| P

The three classes differ in exactly one field each, so that every observation in the lab has a single cause:

Requirements

• A disposable kubeadm cluster, one control-plane node and two workers, Kubernetes 1.34.x, built per the Part LXXIV lessons or Lab 01, all nodes Ready. The lab creates cluster-scoped objects — StorageClasses, PersistentVolumes and a namespace of its own — and writes directories onto the worker filesystems.
• kubectl 1.34.x with cluster-admin.
• SSH with sudo to both workers. Several observations in this lab are only visible on the node: the directory the provisioner created, its name, and whether it is still there after a delete. There is no kubectl for that, and that is itself the lesson about node-local storage.
• Roughly 1 GiB free under /opt on each worker. The lab writes a few hundred megabytes at most, in Task 8, and Cleanup removes it.
• Ability to pull nginx:1.27.2, busybox:1.37, docker.io/library/busybox and docker.io/rancher/local-path-provisioner. The provisioner runs a short-lived busybox helper Pod for every volume it creates and destroys, so a cluster that cannot pull that image will hang at the first claim.
• No out-of-band access requirement. Nothing in this lab touches networking, SSH, the firewall or any node’s boot configuration. Nothing here can lock you out. It can leave Released PersistentVolumes and orphaned directories behind, which Cleanup addresses explicitly.

Scenario

An application team has asked for “a PVC that just works”, the way it does in the managed cluster they used at their last employer. On that cluster somebody had installed a CSI driver and marked its StorageClass default, so a PVC with six lines of YAML and no storageClassName bound in two seconds and nobody ever had to know why.

You are on a kubeadm cluster with nothing installed. Before you give them the one-line answer, you are going to build the thing that makes it work and watch each field decide something — because the first production incident on this platform will be a PVC that is Pending, and the difference between a five minute fix and a two hour one is being able to read which of the several possible causes the controller is actually reporting.

Tasks

Task 1: Capture the starting state, and submit a claim with nothing behind it

WORKDIR="$HOME/k8s-lab17"
mkdir -p "$WORKDIR"
cd "$WORKDIR"

kubectl get nodes -o wide     | tee nodes.pre-lab.txt
kubectl get storageclass      | tee storageclass.pre-lab.txt
kubectl get pv                | tee pv.pre-lab.txt
kubectl get pvc -A            | tee pvc.pre-lab.txt
kubectl get ns                > ns.pre-lab.txt

Both storageclass.pre-lab.txt and pv.pre-lab.txt should say No resources found. That empty output is the starting condition the lab is written for. Record the two worker node names exactly as kubectl get nodes prints them; Task 6 needs both.

kubectl create namespace lab17

Now the claim. Write pvc-orphan.yaml:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-orphan
  namespace: lab17
spec:
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 128Mi

There is no storageClassName. That is not a mistake; it is the six-line PVC from the scenario, and it is the most common shape of PVC in the wild.

$ kubectl apply -f pvc-orphan.yaml

kubectl -n lab17 get pvc data-orphan
kubectl -n lab17 describe pvc data-orphan
kubectl -n lab17 get events --field-selector involvedObject.name=data-orphan

$ kubectl -n lab17 describe pvc data-orphan

Name:          data-orphan
Namespace:     lab17
StorageClass:
Status:        Pending
Volume:
Labels:        <none>
Annotations:   <none>
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode:    Filesystem
Used By:       <none>
Events:
Type    Reason         Age   From                         Message
----    ------         ----  ----                         -------
Normal  FailedBinding  9s    persistentvolume-controller  no persistent volumes available for this claim and no storage class is set

Leave data-orphan in place. Task 5 comes back for it.

Task 2: Install a dynamic provisioner, reading the manifest first

The provisioner used here is Rancher’s local-path-provisioner. It is not a CSI driver: it is an external provisioner that watches for PVCs on its StorageClass, runs a short-lived helper Pod on the target node to create a directory, and writes a PersistentVolume pointing at it. That makes it the smallest honest example of dynamic provisioning that runs on a cluster with no cloud behind it — and its limitations, in Task 8, are as instructive as its behaviour.

cd "$HOME/k8s-lab17"
LPP_VERSION=v0.0.37
curl -fsSLo local-path-storage.yaml \
  "https://raw.githubusercontent.com/rancher/local-path-provisioner/$LPP_VERSION/deploy/local-path-storage.yaml"

grep -n 'kind: StorageClass' -A 8 local-path-storage.yaml
grep -n 'nodePathMap' -A 6 local-path-storage.yaml
grep -n 'name: helper-pod' -A 4 local-path-storage.yaml

$ grep -n 'kind: StorageClass' -A 8 local-path-storage.yaml

141:kind: StorageClass
142-metadata:
143-  name: local-path
144-provisioner: rancher.io/local-path
145-volumeBindingMode: WaitForFirstConsumer
146-reclaimPolicy: Delete

Read those four lines as four decisions someone made on your behalf:

• provisioner: rancher.io/local-path names the controller that must be running for this class to do anything. A class whose provisioner nobody implements is a valid object that binds nothing — the same shape of failure as Lab 15’s unenforced NetworkPolicy.
• volumeBindingMode: WaitForFirstConsumer means a PVC on this class stays Pending until a Pod that uses it is scheduled. That is Task 3.
• reclaimPolicy: Delete means deleting the claim deletes the volume and its contents. That is Task 7, and it is the wrong default for anything you care about.
• There is no storageclass.kubernetes.io/is-default-class annotation. This class is not default. That is Task 5.

The nodePathMap in the ConfigMap gives the directory that every volume will live under: /opt/local-path-provisioner, for every node not listed individually.

$ kubectl apply -f local-path-storage.yaml

kubectl -n local-path-storage rollout status deployment/local-path-provisioner --timeout=180s
kubectl get storageclass
kubectl -n lab17 get pvc data-orphan

kubectl get storageclass now lists local-path with no (default) marker, and data-orphan is still Pending with the same FailedBinding message. Installing a provisioner changed nothing for a claim that never asked for one: a class exists now, but this PVC does not name it and no class is marked default, so the controller’s second complaint still stands.

Task 3: A PVC that is Pending on purpose

Write pvc-web.yaml:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-web
  namespace: lab17
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: local-path
  resources:
    requests:
      storage: 128Mi

$ kubectl apply -f pvc-web.yaml

kubectl -n lab17 get pvc
kubectl -n lab17 describe pvc data-web | tail -8
kubectl get pv

$ kubectl -n lab17 describe pvc data-web | tail -8

Used By:       <none>
Events:
Type    Reason                Age   From                         Message
----    ------                ----  ----                         -------
Normal  WaitForFirstConsumer  6s    persistentvolume-controller  waiting for first consumer to be created before binding

kubectl get pv still returns No resources found. Nothing has been created anywhere: not a PV, not a directory, not a helper Pod. The claim is a reservation of intent.

Task 4: The consumer arrives, and the volume is born

Write web.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  namespace: lab17
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: web
          image: nginx:1.27.2
          volumeMounts:
            - name: data
              mountPath: /usr/share/nginx/html
      volumes:
        - name: data
          persistentVolumeClaim:
            claimName: data-web

Watch the provisioner while you apply it — the interesting part lasts a few seconds:

cd "$HOME/k8s-lab17"
kubectl -n local-path-storage logs -f deployment/local-path-provisioner &
kubectl apply -f web.yaml
kubectl -n lab17 rollout status deployment/web --timeout=180s
kill %1

kubectl -n lab17 get pvc data-web
kubectl get pv
kubectl -n lab17 get pods -o wide

$ kubectl get pv

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                 STORAGECLASS   AGE
pvc-3f0f5c2e-9f34-4a1d-9ac0-6f2a1c8b7d55   128Mi      RWO            Delete           Bound    lab17/data-web        local-path     12s

Read where each column came from. CAPACITY and ACCESS MODES came from the PVC’s request. RECLAIM POLICY and STORAGECLASS came from the class. The NAME came from the provisioner, which names dynamically provisioned volumes pvc- followed by the claim’s UID — which is why you can always find the PV for a claim, and why a PV name tells you nothing about what is in it.

Now find it on disk. The provisioner names the directory from a template, {{ .PVName }}_{{ .PVC.Namespace }}_{{ .PVC.Name }}, under the path from the nodePathMap:

# Substitute your own worker node names before running:
W1=k8s-w-1
W2=k8s-w-2

for NODE in "$W1" "$W2"; do
  echo "== $NODE"
  ssh "$NODE" 'sudo ls -l /opt/local-path-provisioner/ 2>/dev/null || echo "no such directory"'
done

Exactly one of the two workers has the directory: the one the web Pod was scheduled to. Write through the Pod and read from the node, to prove they are the same bytes:

kubectl -n lab17 exec deploy/web -- \
  sh -c 'echo "written by the pod" > /usr/share/nginx/html/index.html'

kubectl -n lab17 exec deploy/web -- cat /usr/share/nginx/html/index.html

# Substitute the node the Pod landed on, and the directory name you saw above:
NODE=k8s-w-1
VOLDIR=pvc-3f0f5c2e-9f34-4a1d-9ac0-6f2a1c8b7d55_lab17_data-web

ssh "$NODE" "sudo cat /opt/local-path-provisioner/$VOLDIR/index.html"

That is the whole of dynamic provisioning, on this driver: a controller made a directory and wrote an object describing it.

Task 5: Make it the default, and watch the orphan bind itself

data-orphan from Task 1 is still Pending, still with storageClassName unset. Mark local-path as the cluster default and watch what happens to a claim you are not going to touch:

kubectl -n lab17 get pvc data-orphan -o jsonpath='{.spec.storageClassName}{"\n"}'

That prints nothing: the field is absent.

$ kubectl annotate storageclass local-path storageclass.kubernetes.io/is-default-class="true" --overwrite

kubectl get storageclass
kubectl -n lab17 get pvc data-orphan -o jsonpath='{.spec.storageClassName}{"\n"}'
kubectl -n lab17 describe pvc data-orphan | tail -6

The class now shows (default) in kubectl get storageclass, and data-orphan — which you have not edited, re-applied, or deleted — now has spec.storageClassName: local-path and a WaitForFirstConsumer event. The control plane found every unbound PVC with an empty or absent storageClassName and filled it in.

Task 6: Reproduce the conflict WaitForFirstConsumer exists to prevent

Everything so far has used deferred binding. Build a class that does not, and see what it costs.

Write sc-immediate.yaml:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: lab17-immediate
provisioner: rancher.io/local-path
volumeBindingMode: Immediate
reclaimPolicy: Delete

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-immediate
  namespace: lab17
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: lab17-immediate
  resources:
    requests:
      storage: 128Mi

Put that second document in pvc-immediate.yaml, then:

$ kubectl apply -f sc-immediate.yaml -f pvc-immediate.yaml

kubectl -n lab17 get pvc data-immediate
kubectl get pv

This claim binds within seconds with no Pod anywhere. A volume now exists, on a node the scheduler had no say in. Find out which one:

PV=$(kubectl -n lab17 get pvc data-immediate -o jsonpath='{.spec.volumeName}')
echo "PV: $PV"

PVNODE=$(kubectl get pv "$PV" \
  -o jsonpath='{.spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[0].values[0]}')
echo "volume is pinned to: $PVNODE"

OTHER=$(kubectl get nodes -l '!node-role.kubernetes.io/control-plane' \
          -o jsonpath='{.items[*].metadata.name}' \
        | tr ' ' '\n' | grep -v "^$PVNODE$" | head -1)
echo "scheduling the Pod to: $OTHER"

Every dynamically provisioned volume on this driver carries a nodeAffinity on kubernetes.io/hostname, because the storage is a directory on one machine and no other machine can reach it. Now schedule a Pod that wants the volume onto the other worker:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: pinned
  namespace: lab17
spec:
  nodeSelector:
    kubernetes.io/hostname: $OTHER
  containers:
    - name: shell
      image: busybox:1.37
      command: ["sleep", "infinity"]
      volumeMounts:
        - name: data
          mountPath: /data
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: data-immediate
EOF

kubectl -n lab17 get pod pinned
kubectl -n lab17 describe pod pinned | tail -6

$ kubectl -n lab17 describe pod pinned | tail -6

Events:
Type     Reason            Age   From               Message
----     ----              ----  ----               -------
Warning  FailedScheduling  18s   default-scheduler  0/3 nodes are available: 1 node(s) had volume node affinity conflict, 2 node(s) didn't match Pod's node affinity/selector.

volume node affinity conflict is the message. It is permanent: the Pod will never schedule, because the volume cannot move and the Pod is pinned away from it. Nothing here is broken — every object is valid and every controller did its job. The mistake was made in a single field on the StorageClass, one step earlier, by a controller that provisioned before anyone knew where the workload would run.

WaitForFirstConsumer inverts the order: the scheduler picks the node using every constraint the Pod has, then the volume is created there. On this driver the constraint is a hostname; on a cloud provider it is an availability zone and the same conflict costs a cross-AZ outage instead of a Pending Pod.

Clean up this branch before continuing:

kubectl -n lab17 delete pod pinned --wait=true
kubectl -n lab17 delete pvc data-immediate --wait=true

Task 7: reclaimPolicy decides whether deleting the claim deletes the data

Write sc-retain.yaml:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: lab17-retain
provisioner: rancher.io/local-path
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Retain

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-keep
  namespace: lab17
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: lab17-retain
  resources:
    requests:
      storage: 128Mi

Put the claim in pvc-keep.yaml, apply both, and give it a consumer that writes something worth keeping:

cd "$HOME/k8s-lab17"
kubectl apply -f sc-retain.yaml -f pvc-keep.yaml

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: keeper
  namespace: lab17
spec:
  containers:
    - name: shell
      image: busybox:1.37
      command: ["sleep", "infinity"]
      volumeMounts:
        - name: data
          mountPath: /data
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: data-keep
EOF

kubectl -n lab17 wait --for=condition=Ready pod/keeper --timeout=180s
kubectl -n lab17 exec keeper -- sh -c 'echo "the only copy" > /data/ledger.txt'
kubectl get pv

Record both PV names and both node placements now — you will need them after the claims are gone:

kubectl get pv -o custom-columns=\
'NAME:.metadata.name,CLAIM:.spec.claimRef.name,POLICY:.spec.persistentVolumeReclaimPolicy,NODE:.spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[0].values[0]' \
  | tee pv-before-delete.txt

Now delete both claims. The web Deployment must go first, because a PVC in use by a Pod is held by a finalizer and its deletion blocks until the Pod is gone.

$ kubectl -n lab17 delete deployment web pod/keeper --wait=true && kubectl -n lab17 delete pvc data-web data-keep --wait=true

kubectl get pv

$ kubectl get pv

NAME                                       CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS     CLAIM              STORAGECLASS   AGE
pvc-b71a02c4-15ce-4e77-9b6e-2b1d0c9e4a37   128Mi      RWO            Retain           Released   lab17/data-keep    lab17-retain   6m

The Delete volume is not in the list at all: the provisioner ran its teardown helper Pod, removed the directory, and deleted the PV object. Confirm that on the node — the directory is gone.

The Retain volume is Released, which means the claim is gone but the cluster has not reclaimed the storage. Confirm on its node that ledger.txt is still there:

# Substitute the node and directory names from pv-before-delete.txt:
NODE=k8s-w-2
VOLDIR=pvc-b71a02c4-15ce-4e77-9b6e-2b1d0c9e4a37_lab17_data-keep

ssh "$NODE" "sudo cat /opt/local-path-provisioner/$VOLDIR/ledger.txt"
ssh "$NODE" "sudo ls -l /opt/local-path-provisioner/"

Retain is the correct policy for anything whose loss would be an incident, and Delete — which is what the shipped local-path class uses, and what most cloud default classes use — is the correct policy for scratch space and nothing else. That decision is made once, in the StorageClass, by whoever installs it, and inherited silently by every claim afterwards.

Task 8: What the class promises, and what the driver actually does

Two fields on a StorageClass read like guarantees from Kubernetes. Neither is.

requests.storage is accounting, not a limit — unless the driver enforces it. Recreate a small claim and overrun it:

cd "$HOME/k8s-lab17"
kubectl apply -f pvc-web.yaml

cat <<'EOF' | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: filler
  namespace: lab17
spec:
  containers:
    - name: shell
      image: busybox:1.37
      command: ["sleep", "infinity"]
      volumeMounts:
        - name: data
          mountPath: /data
  volumes:
    - name: data
      persistentVolumeClaim:
        claimName: data-web
EOF

kubectl -n lab17 wait --for=condition=Ready pod/filler --timeout=180s
kubectl -n lab17 exec filler -- \
  dd if=/dev/zero of=/data/fill.bin bs=1M count=300
kubectl -n lab17 exec filler -- df -h /data

The claim asked for 128Mi. The write of 300 MiB succeeds, and df reports the node’s filesystem, not the claim. On this driver the “volume” is a directory, and nothing on the path from requests.storage to the disk enforces a quota. The figure is used by the scheduler for topology decisions and by the API for accounting against a ResourceQuota; it is not a ceiling.

allowVolumeExpansion is a claim about the driver. The shipped local-path class does not set it. Try to grow the claim anyway:

kubectl -n lab17 patch pvc data-web \
  -p '{"spec":{"resources":{"requests":{"storage":"256Mi"}}}}'

$ kubectl -n lab17 patch pvc data-web -p '{"spec":{"resources":{"requests":{"storage":"256Mi"}}}}'

Error from server (Forbidden): persistentvolumeclaims "data-web" is forbidden: only dynamically provisioned pvc can be resized and the storageclass that provisions the pvc must support resize

That rejection is real validation, done by the API server, and it is the only part of expansion that Kubernetes guarantees. Upstream is explicit about the rest: expansion is supported for CSI volumes and a handful of deprecated in-tree types, and “it also requires a specific CSI driver to support volume expansion”. Setting allowVolumeExpansion: true on a class removes the API server’s objection; it does not give the driver a capability it lacks. On a class whose driver cannot resize, the edit is accepted, the PVC’s spec.resources.requests.storage changes, status.capacity does not, and no error is ever produced.

Validation

cd "$HOME/k8s-lab17"

# 1. A provisioner is running and one class is default.
kubectl -n local-path-storage get deployment local-path-provisioner
kubectl get storageclass

# 2. The three classes exist and differ where they should.
kubectl get storageclass -o custom-columns=\
'NAME:.metadata.name,PROVISIONER:.provisioner,BINDING:.volumeBindingMode,RECLAIM:.reclaimPolicy'

# 3. One Released volume survives, holding data whose claim is gone.
kubectl get pv

# 4. The retained data is still on its node.
#    Substitute the node and directory from pv-before-delete.txt:
NODE=k8s-w-2
VOLDIR=pvc-b71a02c4-15ce-4e77-9b6e-2b1d0c9e4a37_lab17_data-keep
ssh "$NODE" "sudo cat /opt/local-path-provisioner/$VOLDIR/ledger.txt"

# 5. A dynamically provisioned PV carries node affinity.
kubectl get pv -o jsonpath=\
'{range .items[*]}{.metadata.name}{"\t"}{.spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[0].values[0]}{"\n"}{end}'

Expected results:

• local-path-provisioner shows 1/1 ready.
• kubectl get storageclass lists local-path (default), lab17-immediate and lab17-retain, and the custom-columns output shows Immediate on exactly one row and Retain on exactly one row.
• Exactly one PV remains, Released, on lab17-retain, with the data-keep claim named in its CLAIM column.
• ledger.txt prints the only copy.
• Every listed PV prints a node name, not an empty field.

Expected Outcome

A cluster with working dynamic provisioning, and a working directory that records what each field decided:

k8s-lab17/
├── local-path-storage.yaml
├── pvc-orphan.yaml
├── pvc-web.yaml
├── pvc-keep.yaml
├── pvc-immediate.yaml
├── sc-immediate.yaml
├── sc-retain.yaml
├── web.yaml
├── pv-before-delete.txt
├── nodes.pre-lab.txt
├── storageclass.pre-lab.txt
├── pv.pre-lab.txt
├── pvc.pre-lab.txt
└── ns.pre-lab.txt

You can state, from evidence you produced: which two Pending PVCs look identical in kubectl get pvc and are told apart by one event line; what a default StorageClass did to a claim that already existed; the exact scheduler message that a volume in the wrong place produces; and which of the two volumes you deleted still has its data on a node.

Troubleshooting

The first PVC stays Pending with a Provisioning event that never completes. Look for the helper Pod: kubectl -n local-path-storage get pods during the attempt. The provisioner runs a busybox Pod on the target node for every create and delete; if that image cannot be pulled, provisioning hangs with no further event. Mirror docker.io/library/busybox or check the node’s registry route.

kubectl get pv shows the volume, but the Pod is stuck in ContainerCreating. Check the node the Pod landed on against the PV’s nodeAffinity. If they differ, something scheduled the Pod before the volume existed — which on a WaitForFirstConsumer class should be impossible, and on Immediate is Task 6.

Deleting a PVC hangs. A PVC in use carries the kubernetes.io/pvc-protection finalizer and its deletion blocks until every Pod using it is gone. Find them with kubectl -n lab17 get pods -o wide and delete the workload, not the claim, first. Do not remove the finalizer by hand; that is how a volume gets deleted out from under a running Pod.

data-orphan did not bind after Task 5. Confirm the annotation actually landed: kubectl get storageclass local-path -o yaml | grep is-default-class. The annotation value must be the string "true". Then confirm the claim’s class was filled in: kubectl -n lab17 get pvc data-orphan -o yaml | grep storageClassName. If it is still absent, check that no second class is also marked default — with two defaults the behaviour is undefined and the control plane will not choose for you.

A Released PV will not bind to a new claim of the same name. That is correct and not a fault. See the callout in Task 7.

Cleanup

Cleanup here has to remove three kinds of thing: namespaced objects, cluster objects, and bytes on nodes that no controller will ever remove for you.

cd "$HOME/k8s-lab17"
kubectl -n lab17 delete pod filler --ignore-not-found
kubectl delete namespace lab17

Deleting the namespace deletes the remaining PVCs, which triggers the Delete reclaim on their volumes. Wait for that to finish before removing the provisioner — the teardown needs the provisioner running to do its job:

kubectl get pv
kubectl -n local-path-storage get pods

Any PV still listed is a Retain volume. Those are yours to dispose of, which is the policy working as designed:

$ kubectl delete pv --all

Remove the cluster-scoped objects the lab added, and un-default the class before removing it:

kubectl delete storageclass lab17-immediate lab17-retain --ignore-not-found
kubectl delete -f local-path-storage.yaml
kubectl get storageclass

kubectl get storageclass must now match storageclass.pre-lab.txt — on a cluster that started clean, No resources found.

Finally, the bytes. The Retain volume’s directory, and anything left behind by a teardown that could not run, are still on the workers:

$ ssh "$NODE" 'sudo ls -l /opt/local-path-provisioner/ && sudo rm -rf /opt/local-path-provisioner'

kubectl get nodes -o wide
kubectl get storageclass
kubectl get pv
kubectl get ns

Compare each against the pre-lab file it corresponds to. The cluster is back to the state Task 1 recorded.

Production notes

The four fields are a change-control question, not a YAML question. A StorageClass is written once and inherited by every claim on it, usually without the claim’s author reading it. Adding a class is therefore a low-risk change; editing an existing one is not, because reclaimPolicy and volumeBindingMode on a class apply to volumes provisioned after the edit while the existing PVs keep the values they were born with. The result is a cluster where two volumes on the same class behave differently and nothing in kubectl get storageclass explains why. Treat class edits as immutable in practice: create a new class, migrate workloads to it, and retire the old one when nothing references it.

Marking a class default is a cluster-wide change with a backward reach. Task 5 is a one-line command that altered an object in another namespace which nobody had touched for an hour. In a change window that means: enumerate unbound classless PVCs first (kubectl get pvc -A -o json filtered on an absent spec.storageClassName), because those are the claims that will move; announce it; and do it when somebody is watching, not on a Friday. The reverse — removing the default — is equally cluster-wide and produces Pending claims rather than wrong ones, which is the safer failure but not a silent one.

Rehearse the reclaim policy before you need it. The only way to know what your cluster does when a claim is deleted is to delete one, on a class you control, with data you can afford to lose. Do it in the same week you install the driver, record the result next to the class definition, and put the answer in the runbook — because the moment somebody needs it, they are deleting a StatefulSet’s PVCs during an incident and reading documentation is not on the table.

“Hold” looks like a Released volume and an unbound claim. If a restore or a migration reaches the point where a PV is Released and you are not certain what is in it, stopping there is a legitimate and stable state. Nothing is degrading: the data is on disk, the object is inert, and nobody can bind it by accident. The owner is whoever owns the data, the end condition is a decision about the bytes, and the wrong move is to force it Available to make a dashboard green.

Monitor the discriminator, not the phase. A Pending PVC alone is not an incident — Task 3 produced a healthy one deliberately. Alert on a PVC that has been Pending for longer than a threshold and whose most recent event is not WaitForFirstConsumer, and pair it with an alert on Released PVs, which have no phase-based alert anywhere and represent data with no owner.

What You Learned

• Two Pending PVCs, identical in kubectl get pvc, told apart by one line. data-orphan carried FailedBinding — no matching volume and no class to provision one from, two facts in one sentence; data-web carried WaitForFirstConsumer, which means the controller looked and chose to wait. kubectl describe pvc is the first command for a reason, and the two commands after it are kubectl get storageclass and kubectl get pv.
• Dynamic provisioning is a controller making a thing and writing an object about it. You watched a PV named pvc- plus a UID appear, and then read the same bytes through the Pod and over SSH on the node.
• Retroactive default assignment reaches into claims that already exist. A single annotation gave a PVC a StorageClass an hour after it was created, without touching the PVC.
• Immediate binding creates a volume before anything knows where the workload goes, and the resulting volume node affinity conflict is permanent. WaitForFirstConsumer is not an optimisation; it is the ordering that makes topology-constrained storage work at all.
• Delete and Retain differ in what survives a kubectl delete pvc, and Released is a state that needs a human, by design.
• requests.storage and allowVolumeExpansion describe intent. One was overrun by a 300 MiB write into a 128Mi claim; the other was refused by the API server, and would have been accepted-and-ignored on a class that set it over a driver that cannot resize.