Lab 19: Operate a StatefulSet with persistent storage

Objective

By the end of this lab a three-replica StatefulSet will be running on storage you provisioned yourself, and you will have proved — not assumed — the four properties that separate a StatefulSet from a Deployment:

1. The Pod’s ordinal name and hostname survive the Pod.
2. The PVC is bound to the ordinal by name, so the same data comes back to the same ordinal.
3. The rollout walks ordinals from the highest down, and stops at the first one that will not become Ready.
4. The data outlives a scale-down, and it outlives deleting the StatefulSet itself.

You will also stall a rollout on purpose and recover it. That path — a half-updated StatefulSet with two ordinals still on the old revision and no obvious error anywhere — is the one operators meet at 03:00 having never rehearsed it.

Architecture

One namespace, one headless Service, one StatefulSet, and storage you build by hand because a disposable kubeadm cluster has no dynamic provisioner.

flowchart TB
    SC["StorageClass: lab-local<br/>no-provisioner<br/>WaitForFirstConsumer<br/>reclaimPolicy: Retain"]
    SS["StatefulSet: store<br/>replicas: 3<br/>serviceName: store-h"]
    HS["Service: store-h<br/>clusterIP: None"]
    P0["Pod: store-0"]
    P1["Pod: store-1"]
    P2["Pod: store-2"]
    C0["PVC: data-store-0"]
    C1["PVC: data-store-1"]
    C2["PVC: data-store-2"]
    V["6 local PVs<br/>3 per worker<br/>/mnt/lab19/vol0..2"]
    SS --> P0
    SS --> P1
    SS --> P2
    HS -.->|one A record per Pod| P0
    HS -.-> P1
    HS -.-> P2
    P0 --> C0
    P1 --> C1
    P2 --> C2
    C0 --> V
    C1 --> V
    C2 --> V
    SC --> V

Six PVs for three claims is deliberate. local PVs are pinned to a node by nodeAffinity, so the scheduler needs a spare on whichever worker it picks for each ordinal. Three per worker also leaves room for the scale-up in Task 8 without a second provisioning round.

Requirements

• A disposable kubeadm cluster: one control-plane node and two workers, Kubernetes 1.34.x, kubectl 1.34.x, built per the Part LXXIV lessons. Nothing in this lab is safe to run on a cluster somebody else is using — it creates cluster-scoped objects (StorageClass, PersistentVolumes) and writes directories onto the worker filesystems.
• kubectl configured with cluster-admin, from a workstation or the control-plane node.
• SSH with sudo to both workers. Task 2 creates the backing directories that the local PVs point at. There is no way to do this from kubectl alone, which is itself the lesson: local storage is node storage, and somebody has to be on the node.
• Roughly 3 GiB free under /mnt on each worker. The lab writes a few kilobytes; the capacity figures in the PVs are nominal, because the local volume plugin does not enforce them.
• No out-of-band access requirement. This lab does not touch networking, SSH or the firewall on any node, so no step can lock you out. It can leave orphaned PVs and directories behind, which Cleanup addresses.

Scenario

Your team is about to run its first real stateful workload on Kubernetes. The application team has read that “StatefulSets give you stable storage” and has written a manifest. Before it goes anywhere near production data, you want the four claims in that sentence demonstrated on a cluster you can throw away, together with the two operations that the team has not thought about: the staged rollout, and what deleting the object does to the data.

The workload here is deliberately trivial — a BusyBox container that appends one line to its volume every time it starts. A real database would add its own bootstrap semantics on top of everything below, and would obscure the controller behaviour you are here to see. What the container does not do is exactly what makes the volume a clean witness: every line in identity.log is one container start, and the file can only accumulate lines if the volume is the same volume.

Tasks

Task 1: Capture the starting state

WORKDIR="$HOME/k8s-lab19"
mkdir -p "$WORKDIR"
cd "$WORKDIR"

kubectl get nodes -o wide | tee nodes.pre-lab.txt
kubectl get storageclass | tee storageclass.pre-lab.txt
kubectl get pv | tee pv.pre-lab.txt

Read storageclass.pre-lab.txt before continuing. If any row is annotated (default), either remove that annotation for the duration of the lab or accept that Task 5 will behave differently — see the warning above. An empty list is the expected state and the one the lab is written for.

Record the two worker node names exactly as kubectl get nodes prints them. Every local PV below is pinned to one of them by kubernetes.io/hostname, and a typo there produces a PVC that stays Pending for ever with an error that does not mention the typo.

kubectl create namespace lab19

Task 2: Build the storage the lab needs

First the directories, on the nodes, over SSH:

# Substitute your own worker node names before running:
W1=worker-1
W2=worker-2

for NODE in "$W1" "$W2"; do
  ssh "$NODE" 'sudo mkdir -p /mnt/lab19/vol0 /mnt/lab19/vol1 /mnt/lab19/vol2 && ls -ld /mnt/lab19/vol*'
done

Then the StorageClass. Every field in it is a decision, and three of them are the decisions that a production StatefulSet gets wrong.

storage.yaml:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: lab-local
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Retain
allowVolumeExpansion: false

• provisioner: kubernetes.io/no-provisioner says nothing creates volumes on demand. Claims bind only to PVs that already exist. A PVC with no matching PV stays Pending silently — there is no controller to report a failure, because there is no controller.
• volumeBindingMode: WaitForFirstConsumer delays binding until a Pod referencing the claim is scheduled. Without it, the PVC would bind to some PV immediately, and the scheduler would then be forced onto that PV’s node whether or not it had room. This is the field that makes node-pinned storage usable at all.
• reclaimPolicy: Retain means deleting a PVC leaves the PV and its data. This is the correct setting for anything stateful and it is not the default on most cloud StorageClasses, which use Delete.

Apply it, then generate the six PVs:

cd "$HOME/k8s-lab19"
kubectl apply -f storage.yaml

# Substitute your own worker node names before running:
W1=worker-1
W2=worker-2

for NODE in "$W1" "$W2"; do
  for I in 0 1 2; do
    cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolume
metadata:
  name: lab19-${NODE}-vol${I}
  labels:
    lab: lab19
spec:
  capacity:
    storage: 1Gi
  accessModes: ["ReadWriteOnce"]
  persistentVolumeReclaimPolicy: Retain
  storageClassName: lab-local
  local:
    path: /mnt/lab19/vol${I}
  nodeAffinity:
    required:
      nodeSelectorTerms:
        - matchExpressions:
            - key: kubernetes.io/hostname
              operator: In
              values: ["${NODE}"]
EOF
  done
done

kubectl get pv -l lab=lab19

$ kubectl get pv -l lab=lab19

NAME                   CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM   STORAGECLASS   AGE
lab19-worker-1-vol0    1Gi        RWO            Retain           Available           lab-local      10s
lab19-worker-1-vol1    1Gi        RWO            Retain           Available           lab-local      10s
lab19-worker-1-vol2    1Gi        RWO            Retain           Available           lab-local      9s
lab19-worker-2-vol0    1Gi        RWO            Retain           Available           lab-local      9s
lab19-worker-2-vol1    1Gi        RWO            Retain           Available           lab-local      8s
lab19-worker-2-vol2    1Gi        RWO            Retain           Available           lab-local      8s

nodeAffinity is mandatory on a local PV, not optional. It is the only thing that stops the scheduler placing a Pod on a node where the directory does not exist.

Task 3: Apply the workload and watch ordered creation

Two objects: the headless Service the StatefulSet’s serviceName points at, and the StatefulSet itself.

store.yaml:

apiVersion: v1
kind: Service
metadata:
  name: store-h
  namespace: lab19
spec:
  clusterIP: None
  selector:
    app: store
  ports:
    - port: 8080
      name: app
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: store
  namespace: lab19
spec:
  serviceName: store-h
  replicas: 3
  selector:
    matchLabels:
      app: store
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      partition: 3
  template:
    metadata:
      labels:
        app: store
    spec:
      terminationGracePeriodSeconds: 10
      initContainers:
        - name: settle
          image: busybox:1.37
          command: ["sh", "-c", "sleep 20"]
      containers:
        - name: store
          image: busybox:1.37
          env:
            - name: LAB_RELEASE
              value: "r1"
          command:
            - sh
            - -c
            - |
              echo "boot $(date -u +%Y-%m-%dT%H:%M:%SZ) pod=$(hostname) release=$LAB_RELEASE" >> /data/identity.log
              while true; do sleep 3600; done
          readinessProbe:
            exec:
              command: ["sh", "-c", "test -s /data/identity.log"]
            initialDelaySeconds: 3
            periodSeconds: 5
          volumeMounts:
            - name: data
              mountPath: /data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: lab-local
        resources:
          requests:
            storage: 1Gi

The initContainers sleep 20 exists for one reason: to make the ordering visible in real time. Without it the three Pods reach Ready fast enough that the sequence is easy to miss.

Note partition: 3 with replicas: 3. That means no ordinal is eligible for update — ordinals are updated only when their index is greater than or equal to partition, and no ordinal is 3 or higher. The rollout in Task 6 starts from that safe position and walks down.

Open a watch in a second terminal, then apply:

kubectl get pods -n lab19 -w

cd "$HOME/k8s-lab19"
kubectl apply -f store.yaml

Watch the sequence. store-0 appears, sits in Init:0/1 for twenty seconds, becomes 1/1 Running, and only then does store-1 appear. Roughly a minute passes before store-2 exists at all. Nothing here is parallel, and nothing here is a coincidence: the controller does not create ordinal N until ordinal N-1 reports Ready.

$ kubectl get pods,pvc -n lab19 -o wide

NAME          READY   STATUS    RESTARTS   AGE   NODE
pod/store-0   1/1     Running   0          92s   worker-1
pod/store-1   1/1     Running   0          64s   worker-2
pod/store-2   1/1     Running   0          35s   worker-1

NAME                                STATUS   VOLUME                CAPACITY   ACCESS MODES   STORAGECLASS
persistentvolumeclaim/data-store-0  Bound    lab19-worker-1-vol0   1Gi        RWO            lab-local
persistentvolumeclaim/data-store-1  Bound    lab19-worker-2-vol0   1Gi        RWO            lab-local
persistentvolumeclaim/data-store-2  Bound    lab19-worker-1-vol1   1Gi        RWO            lab-local

The PVC names are the first thing to look at. They are data-store-0, data-store-1, data-store-2 — the volumeClaimTemplates entry’s metadata.name joined to the Pod name. A Deployment cannot produce that, because a Deployment’s Pod names are not predictable.

Task 4: Prove stable identity

Identity has two halves: the hostname the container sees, and the DNS record the rest of the cluster resolves. Check both.

kubectl exec -n lab19 store-1 -- hostname
kubectl exec -n lab19 store-1 -- hostname -f
kubectl get pod store-1 -n lab19 -o jsonpath='{.metadata.labels}' ; echo

The statefulset.kubernetes.io/pod-name label in that last output is set by the controller and equals the Pod name. It is what lets you write a Service that selects exactly one ordinal — the standard way to publish “the primary” as a stable endpoint.

Now the DNS half, from a throwaway Pod:

kubectl run -n lab19 -it --rm --restart=Never --image=busybox:1.37 dns-test -- \
  nslookup store-1.store-h.lab19.svc.cluster.local

kubectl run -n lab19 -it --rm --restart=Never --image=busybox:1.37 dns-test -- \
  nslookup store-h.lab19.svc.cluster.local

The first query returns one address. The second returns three — one A record per Pod, because clusterIP: None means CoreDNS answers with the endpoints instead of with a Service IP. That difference is the entire reason a StatefulSet requires a headless Service: clustered software needs to address store-1 specifically, and a load-balanced ClusterIP cannot express that.

Task 5: Prove the volume follows the ordinal

Read the log the container wrote at startup, then delete the Pod:

kubectl exec -n lab19 store-1 -- cat /data/identity.log
kubectl delete pod store-1 -n lab19
kubectl wait --for=condition=Ready pod/store-1 -n lab19 --timeout=180s
kubectl exec -n lab19 store-1 -- cat /data/identity.log

$ kubectl exec -n lab19 store-1 -- cat /data/identity.log

boot 2026-08-18T09:14:07Z pod=store-1 release=r1
boot 2026-08-18T09:21:44Z pod=store-1 release=r1

Two lines, and kubectl get pod store-1 -n lab19 reports RESTARTS 0 — because this is a different Pod object with the same name, not a restarted container. That is the whole claim, demonstrated: the ordinal outlived the Pod, and the PVC came back to it by name.

Now check where it landed:

kubectl get pod store-1 -n lab19 -o wide
kubectl get pvc data-store-1 -n lab19 -o jsonpath='{.spec.volumeName}' ; echo

It is on the same node as before. That is not the StatefulSet’s doing — it is the local PV’s nodeAffinity. The claim is already bound to a node-pinned PV, so the scheduler has exactly one candidate node.

Task 6: Stage a rollout with partition

Change the release marker in the Pod template, one ordinal at a time. First record where you are:

cd "$HOME/k8s-lab19"
kubectl get statefulset store -n lab19 \
  -o jsonpath='{"current="}{.status.currentRevision}{"\nupdate="}{.status.updateRevision}{"\n"}'
kubectl get pods -n lab19 -L controller-revision-hash

currentRevision and updateRevision are equal right now: every ordinal is on the same revision. controller-revision-hash on each Pod is the label that tells you which one — and it is the only honest way to read a partial rollout, because kubectl get pods shows three Running Pods either way.

Edit the template and apply with partition still at 3:

sed -i 's/value: "r1"/value: "r2"/' store.yaml
kubectl apply -f store.yaml
kubectl get pods -n lab19 -L controller-revision-hash

Nothing restarts. updateRevision now differs from currentRevision, kubectl rollout status reports the set as up to date at zero updated replicas, and every Pod still carries the old hash. The new revision exists and is waiting.

Now walk the partition down, checking between steps:

cd "$HOME/k8s-lab19"
sed -i 's/partition: 3/partition: 2/' store.yaml
kubectl apply -f store.yaml
kubectl rollout status statefulset/store -n lab19 --timeout=180s
kubectl get pods -n lab19 -L controller-revision-hash
kubectl exec -n lab19 store-2 -- cat /data/identity.log

Only store-2 is deleted and recreated. Its identity.log gains a release=r2 line while store-0 and store-1 still show only r1 lines. Repeat for partition: 1, then partition: 0, checking the hash table each time.

cd "$HOME/k8s-lab19"
sed -i 's/partition: 2/partition: 1/' store.yaml
kubectl apply -f store.yaml
kubectl rollout status statefulset/store -n lab19 --timeout=180s

sed -i 's/partition: 1/partition: 0/' store.yaml
kubectl apply -f store.yaml
kubectl rollout status statefulset/store -n lab19 --timeout=180s
kubectl get pods -n lab19 -L controller-revision-hash

Ordinal 0 goes last, and that is the point. In a primary-plus-replicas topology, ordinal 0 is conventionally the primary; walking down from the highest ordinal means every replica has proved the new revision before the primary takes it.

Task 7: Break the rollout, then recover it

Reset the partition to 2 and ship a template whose readiness probe can never pass. This is the shape of a real bad release: the container starts, the process runs, and the application never reports healthy.

cd "$HOME/k8s-lab19"
sed -i 's/partition: 0/partition: 2/' store.yaml
sed -i 's#test -s /data/identity.log#test -s /data/ready-marker#' store.yaml
kubectl apply -f store.yaml

Watch what happens, and give it two minutes:

kubectl get pods -n lab19 -w

store-2 is deleted and recreated. The container runs. It never reaches 1/1. kubectl rollout status blocks indefinitely, and kubectl describe pod store-2 -n lab19 shows repeated Unhealthy: Readiness probe failed events with a non-zero exit code from test.

Now the part worth rehearsing. Lower the partition to 1 and observe:

cd "$HOME/k8s-lab19"
sed -i 's/partition: 2/partition: 1/' store.yaml
kubectl apply -f store.yaml
kubectl get pods -n lab19 -L controller-revision-hash

Nothing happens to store-1. The controller will not touch a lower ordinal until the higher one is Running and Ready, so the broken ordinal blocks the entire remaining rollout. From the outside this reads as “the deploy is stuck” with no failing Pod in the namespace — store-2 is Running, just 0/1, and the two ordinals that were never updated look perfectly healthy.

Recover by reverting the template, not by deleting things:

cd "$HOME/k8s-lab19"
sed -i 's#test -s /data/ready-marker#test -s /data/identity.log#' store.yaml
kubectl apply -f store.yaml
kubectl rollout status statefulset/store -n lab19 --timeout=300s
kubectl get pods -n lab19 -L controller-revision-hash

The controller creates a third revision, replaces store-2 with it, and — now that ordinal 2 is Ready — proceeds to store-1, because the partition is still 1. No PVC was touched at any point; every identity.log keeps every line it had.

Task 8: Scale down, then delete — what survives

Scale down and look at the claim, not at the Pod:

kubectl scale statefulset store -n lab19 --replicas=2
kubectl get pods -n lab19
kubectl get pvc -n lab19

store-2 is gone. data-store-2 is still Bound, still holding its PV, still holding the file. The default persistentVolumeClaimRetentionPolicy is Retain for both whenScaled and whenDeleted, and this is a default that costs money in production: scaled-down claims keep billing for volumes nobody is using until an operator removes them by name.

Scale back up and read the file:

kubectl scale statefulset store -n lab19 --replicas=3
kubectl wait --for=condition=Ready pod/store-2 -n lab19 --timeout=180s
kubectl exec -n lab19 store-2 -- cat /data/identity.log

Every earlier boot line is still there, with a new one appended. The ordinal came back to its own data. For a database replica that is exactly the desired behaviour — and it is also the trap, because a replica that rejoins with a stale on-disk state may need application level re-seeding before it is safe to serve reads.

Now delete the StatefulSet itself:

$ kubectl delete statefulset store -n lab19

statefulset.apps "store" deleted

kubectl get pods -n lab19
kubectl get pvc -n lab19

No Pods. Three PVCs, still Bound. Recreate the StatefulSet from the same manifest and read the log one more time:

cd "$HOME/k8s-lab19"
kubectl apply -f store.yaml
kubectl rollout status statefulset/store -n lab19 --timeout=300s
kubectl exec -n lab19 store-0 -- cat /data/identity.log

The history is intact. The controller found PVCs already named data-store-0, data-store-1 and data-store-2 and used them rather than creating new ones — which is what makes “delete and reapply” a survivable mistake on a StatefulSet and an unrecoverable one on anything that owns its storage.

Validation

Every item below is a command whose output either shows the property or does not. Do not accept a screenshot of three Running Pods as evidence of any of them.

# 1. Per-Pod claims exist and are named after the ordinal
kubectl get pvc -n lab19 -o name

# 2. Each claim is bound to a distinct PV
kubectl get pvc -n lab19 \
  -o custom-columns=CLAIM:.metadata.name,VOLUME:.spec.volumeName

# 3. Every Pod carries the same revision hash after a completed rollout
kubectl get pods -n lab19 -L controller-revision-hash

# 4. currentRevision equals updateRevision when the rollout is done
kubectl get statefulset store -n lab19 \
  -o jsonpath='{.status.currentRevision}{"\n"}{.status.updateRevision}{"\n"}'

# 5. The volume accumulated one line per container start
kubectl exec -n lab19 store-1 -- wc -l /data/identity.log

Pass criteria:

• Three PVCs named data-store-0, data-store-1, data-store-2, each Bound to a different lab19-* PV.
• All three Pods share one controller-revision-hash, and currentRevision equals updateRevision.
• store-1’s identity.log has at least four lines: the initial boot, the boot after you deleted the Pod in Task 5, and one for each revision it took in Tasks 6 and 7 — while kubectl get pod store-1 reports RESTARTS 0.
• store-2’s identity.log retains lines written before the scale-down in Task 8.
• nslookup store-h.lab19.svc.cluster.local returns three addresses and nslookup store-0.store-h.lab19.svc.cluster.local returns one.

Expected Outcome

k8s-lab19/
├── nodes.pre-lab.txt
├── pv.pre-lab.txt
├── storageclass.pre-lab.txt
├── storage.yaml          <- StorageClass lab-local
└── store.yaml            <- headless Service + StatefulSet

In the cluster: namespace lab19 with one headless Service, one three-replica StatefulSet whose ordinals are all on one revision, three bound PVCs, and six lab19-* PVs of which three are Bound and three Available. On each worker, /mnt/lab19/vol0..2 exists and at least one of them contains an identity.log with several boot lines in it.

Troubleshooting

A PVC stays Pending and kubectl describe pvc says waiting for first consumer to be created before binding. This is normal with WaitForFirstConsumer and resolves as soon as the Pod is scheduled. If it persists, the Pod is unschedulable for some other reason — check kubectl describe pod for the real event.

A PVC stays Pending with no persistent volumes available for this claim. No Available PV matches on StorageClass, access mode, capacity and node. The usual cause is a hostname typo in the PV’s nodeAffinity: compare kubectl get pv lab19-worker-1-vol0 -o yaml against kubectl get nodes -o name character by character. The second cause is that all PVs on the chosen node are already bound — which is why the lab creates three per worker.

store-0 is Init:0/1 for far longer than twenty seconds. The image is being pulled. kubectl describe pod store-0 -n lab19 shows the Pulling event; on a cluster with no registry mirror the first pull of busybox:1.37 dominates the timing of the whole task.

A Pod is Running but stuck at 0/1 and you did not break it on purpose. The readiness probe is failing. test -s /data/identity.log fails when the file is empty, which happens if the container’s echo could not write — check that the volume mounted at /data and that the directory on the node is writable by the container’s user.

kubectl rollout status never returns. Something is not Ready. Ctrl-C it and run kubectl get pods -n lab19 -L controller-revision-hash: the ordinal with the new hash and no Ready condition is the one blocking every ordinal below it. This is Task 7 happening to you by accident.

Deleting a PVC hangs in Terminating. A Pod is still using it. The kubernetes.io/pvc-protection finalizer holds the object until the last consumer is gone. Delete the StatefulSet first, then the claims — the order in Cleanup below is not arbitrary.

Cleanup

This lab created cluster-scoped objects and wrote to the workers’ filesystems. Removing the namespace is not enough.

Step 1. Confirm what you are about to delete belongs to the lab:

kubectl get all -n lab19
kubectl get pvc -n lab19
kubectl get pv -l lab=lab19

Step 2. Delete the workload, then the claims, in that order:

$ kubectl delete namespace lab19

namespace "lab19" deleted

Deleting the namespace removes the PVCs with it. Watch the PVs move from Bound to Released — not to Available, because reclaimPolicy: Retain deliberately refuses to recycle them:

kubectl get pv -l lab=lab19

Step 3. Delete the PVs and the StorageClass. A Released PV is not reusable and is not cleaned up by anything:

kubectl delete pv -l lab=lab19
kubectl delete storageclass lab-local

Step 4. Remove the data from the workers. The local volume plugin has no deleter, so the files are still on disk after every object above has gone.

$ for NODE in "$W1" "$W2"; do ssh "$NODE" 'sudo rm -rf /mnt/lab19'; done

(no output on success)

Step 5. Verify the cluster is back where Task 1 found it:

kubectl get storageclass
kubectl get pv
kubectl get namespace lab19

Compare the first two against storageclass.pre-lab.txt and pv.pre-lab.txt. The third should report NotFound. Keep the working directory — the two manifests are the deliverable.

What You Learned

• Ordinal identity is a controller guarantee, not a Pod property. Deleting store-1 produced a new Pod object with the same name, the same hostname, the same DNS record and the same volume — and RESTARTS 0, which is how you tell the two apart.
• The claim is bound to the ordinal by name. data-store-1 is matched to store-1 because of what it is called. That is why deleting and reapplying a StatefulSet reuses the existing data instead of provisioning fresh volumes.
• A headless Service is a hard requirement with a soft failure. Removing it breaks per-Pod DNS and nothing reports an error.
• partition is a threshold on the ordinal, not a progress counter. You read the rollout from controller-revision-hash and status.updateRevision, because the Pod list looks identical either way.
• A stuck high ordinal blocks every ordinal below it. The namespace showed three Running Pods and a deploy that would never finish. The fix is to correct the template first and delete the Pod second, never the other way round.
• Retain is the default for PVC retention, and Delete is the default on most cloud StorageClasses. Those two defaults pull in opposite directions, and the combination that loses data is a Delete StorageClass plus a runbook step that says “clean up the PVCs”.
• local storage pins an ordinal to a node. Everything about stable identity still holds; the failure domain changed underneath it.