Persistent storage — volumeClaimTemplates, per-Pod PVCs, and reclaim

A StatefulSet’s volumeClaimTemplates create one PVC per Pod, bound to the Pod by ordinal. The Pod-template-to-PVC relationship outlives the Pod: when a Pod is rescheduled, the same PVC is rebound. This is the storage half of the stable identity contract. The other half — reclaim policy, deletion sequence, and the difference between PVC and PV — is where most StatefulSet data-loss incidents originate.

volumeClaimTemplates

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  serviceName: postgres-h
  replicas: 3
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
      - name: postgres
        image: postgres:16
        volumeMounts:
        - name: data
          mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: ssd
      resources:
        requests:
          storage: 100Gi

For each replica, the StatefulSet controller creates:

flowchart LR
    A["StatefulSet: postgres<br/>volumeClaimTemplates: data"] --> B["Pod: postgres-0"]
    A --> C["Pod: postgres-1"]
    A --> D["Pod: postgres-2"]
    B --> B1["PVC: data-postgres-0"]
    C --> C1["PVC: data-postgres-1"]
    D --> D1["PVC: data-postgres-2"]
    B1 --> B2["PV: pvc-7c8f2d8e-..."]
    C1 --> C2["PV: pvc-3a9b4c1d-..."]
    D1 --> D2["PV: pvc-5e6f8a2b-..."]

The PVC name is <claim-name>-<pod-name>. The Pod mounts the volume by volumeMounts[].name; the kubelet resolves the mount by walking the Pod’s bound PVCs.

Per-Pod PVC creation

When the StatefulSet scales from 0 to 3 replicas:

kubectl get pvc -n data -l app.kubernetes.io/name=postgres -w
# NAME               STATUS   VOLUME                                     CAPACITY   ACCESS MODES
# data-postgres-0    Pending  pvc-7c8f2d8e-...                          0          RWO
# data-postgres-1    Pending  pvc-3a9b4c1d-...                          0          RWO
# data-postgres-2    Pending  pvc-5e6f8a2b-...                          0          RWO
# data-postgres-0    Bound    pvc-7c8f2d8e-...                          100Gi      RWO
# ...

Each PVC is Pending until the StorageClass’s provisioner creates the underlying PV. The PVC binds to the PV, the Pod mounts the volume, and the workload’s data directory is initialised.

The PVCs are created in ordinal order, mirroring the Pod order. The StatefulSet controller does not proceed to the next ordinal until the current PVC is Bound. A slow-provisioning StorageClass blocks the entire StatefulSet.

PVC bound to Pod by name

A Deployment’s Pods are matched to PVCs by selector. A StatefulSet’s Pods are matched to PVCs by name. The controller creates data-postgres-0; when the StatefulSet later recreates postgres-0, the new Pod is bound to the same data-postgres-0 PVC.

sequenceDiagram
    participant K as StatefulSet controller
    participant N1 as node-1
    participant N2 as node-2
    participant P as PVC data-postgres-0
    K->>N1: create postgres-0
    N1->>P: bind
    P->>N1: mount
    Note over N1: node-1 fails
    K->>N2: recreate postgres-0
    N2->>P: bind (same PVC)
    P->>N2: mount (same data)

The Pod’s volume is the same across reschedules. This is the storage half of stable identity: the Pod’s identity is the hostname and the volume.

StorageClass and reclaim policy

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: ssd
provisioner: kubernetes.io/no-provisioner   # or ebs.csi.aws.com, csi.tigera.io, etc.
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Retain    # <-- the critical field

reclaimPolicy decides what happens to the PV (and the underlying storage) when the PVC is deleted:

StatefulSet deletion and PVCs

When a StatefulSet is deleted, the Pods are deleted in reverse ordinal order. The PVCs are not deleted:

kubectl delete statefulset postgres -n data
# statefulset.apps "postgres" deleted

kubectl get pvc -n data -l app.kubernetes.io/name=postgres
# NAME               STATUS   VOLUME                                     CAPACITY
# data-postgres-0    Released pvc-7c8f2d8e-...                          100Gi
# data-postgres-1    Released pvc-3a9b4c1d-...                          100Gi
# data-postgres-2    Released pvc-5e6f8a2b-...                          100Gi

With reclaimPolicy: Delete, the PVs may also be deleted. With reclaimPolicy: Retain, the PVs become Released and the underlying storage remains.

The Pod-specific binding makes the PVCs unusable by a new Pod with a different name. The data is there; the operator must manually clean up the PVC and PV and rebind.

Recovering PVCs after deletion

# 1. Inspect the released PV
kubectl get pv pvc-7c8f2d8e-... -o yaml
# spec.claimRef.namespace: prod
# spec.claimRef.name: data-postgres-0
# spec.claimRef.uid: ...
# status.phase: Released

# 2. Remove the claimRef to make the PV Available
kubectl edit pv pvc-7c8f2d8e-...
# Delete spec.claimRef (or set it to a new PVC)
# status.phase will move from Released to Available

# 3. Create a new PVC with the same name
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-postgres-0
  namespace: data
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: ssd
  resources:
    requests:
      storage: 100Gi
  volumeName: pvc-7c8f2d8e-...
EOF

# 4. Recreate the StatefulSet
kubectl apply -f postgres-statefulset.yaml

The PV is now bound to the new PVC, and the new postgres-0 Pod mounts the existing volume.

Volume expansion

Production databases grow. StatefulSet volumes can be expanded online (CSI dependent):

kubectl edit pvc data-postgres-0 -n data
# spec.resources.requests.storage: 200Gi

The CSI driver handles the expansion; the filesystem is resized; the application sees the new capacity without downtime. Not all CSI drivers support online expansion; check the StorageClass’s allowVolumeExpansion field.

flowchart LR
    A["PVC: 100Gi"] -->|edit| B["PVC: 200Gi"]
    B --> C[CSI driver expands volume]
    C --> D[Filesystem resized]
    D --> E[Application sees new capacity]

Quiz

Production discipline

• reclaimPolicy: Retain is the default for databases. Treat Delete as the exception with a written justification.
• The PVC outlives the StatefulSet. Deletion of the StatefulSet is not a deletion of the data. The runbook must spell out what happens to the PVCs.
• Snapshot policy is part of the StatefulSet design. CSI snapshots of the PV are the standard backup; the schedule must align with the database’s RPO.
• Volume expansion is online, but only with the right CSI driver. Verify the StorageClass supports it; older drivers may require offline expansion (the database is stopped while the volume grows).
• A Released PV is a smell. It indicates a PVC was deleted but the PV was not reclaimed. Audit the cluster for Released PVs regularly; they consume storage without serving workloads.

StatefulSet storage is the storage half of stable identity. The PV is the broker; the PVC is the binding; the Pod is the process. Operators who understand the three layers understand what “deleting the StatefulSet” actually means.