Volume lifecycle — from PVC creation to PV deletion and reclamation

The volume lifecycle has several stages, each with operational implications. The reclaim policy is the most consequential decision: it determines whether the data survives PVC deletion. This lesson walks the lifecycle and the policies.

The complete lifecycle

stateDiagram-v2
    [*] --> Created: PVC submitted
    Created --> Pending: no matching PV
    Pending --> Bound: PV created or matched
    Created --> Bound: matching PV exists
    Bound --> Mounted: Pod mounts PVC
    Mounted --> Bound: Pod unmounts (deletes)
    Bound --> Released: PVC deleted
    Released --> Available: reclaim succeeds (Retain: manual)
    Released --> [*]: reclaim succeeds (Delete)
    Released --> Failed: reclaim fails
    Failed --> [*]: manual intervention

Each stage has its own concerns:

Reclaim policies

The reclaim policy is set on the PV (or in the StorageClass, which the provisioner copies to the PV):

# Retain: PV is kept; data is preserved
spec:
  persistentVolumeReclaimPolicy: Retain

# Delete: PV and underlying storage are deleted
spec:
  persistentVolumeReclaimPolicy: Delete

Retain

With Retain:

1. PVC is deleted.
2. PV transitions to Released.
3. PV remains in Released state; the underlying storage is not deleted.
4. An operator must manually:
   • Remove the claimRef from the PV spec to make it Available.
   • Delete the underlying storage, or rebind the PV to a new PVC.

The production rule: Retain for stateful data. The operator decides when to reclaim; the data is preserved until then.

Delete

With Delete:

1. PVC is deleted.
2. PV transitions to Released.
3. The provisioner is called to delete the underlying storage (e.g., EBS volume).
4. The PV is deleted from the cluster.

The production rule: Delete for ephemeral data. The storage is cleaned up automatically; no operator intervention is required.

Recycle (deprecated)

Recycle performs a basic scrub — effectively rm -rf /thevolume/* run by a recycler Pod — and returns the PV to the pool as Available for a new claim.

It is deprecated, not removed. Recycle is still an accepted value of persistentVolumeReclaimPolicy in the PersistentVolume v1 API on 1.34, and a PV that carries it will still be scrubbed and reused. Only the in-tree HostPath and NFS plugins ever implemented it; no CSI driver does, so on a modern cluster setting Recycle usually means the PV simply sits in Released and is never reclaimed at all.

Treat it the way you would treat any deprecated field you find in an inherited cluster: it is not an error that halts you, it is a signal that the manifest predates dynamic provisioning and should be migrated to Delete with snapshots, or Retain with a documented cleanup step. Do not assume the API server will reject it — it will not, and a “the cluster would have caught it” assumption is how deprecated fields survive upgrades.

The deletion sequence

When a StatefulSet is deleted, the order is:

1. StatefulSet is deleted.
2. Pods are deleted in reverse ordinal order.
3. PVCs are not deleted — they remain Bound to their PVs.
4. PVs are reclaimed per their reclaim policy.

A common mistake: an operator deletes the StatefulSet expecting the data to be cleaned up. The Pods are deleted; the PVCs are not; the PVs are not (with Retain). The data remains.

# Substitute your own value before running:
NS=production

# After deleting the StatefulSet
kubectl get pvc -n "$NS"
# NAME               STATUS   VOLUME
# data-postgres-0    Bound    pvc-7c8f2d8e-...
# data-postgres-1    Bound    pvc-3a9b4c1d-...
# data-postgres-2    Bound    pvc-5e6f8a2b-...

kubectl get pv
# NAME                  CAPACITY   STATUS   RECLAIM POLICY
# pvc-7c8f2d8e-...      100Gi      Bound    Retain

The PVCs and PVs are still there. The operator must decide what to do with them.

The orphan PV problem

A Released PV with Retain is an orphan: the PVC was deleted, the PV remains, the data is preserved but inaccessible. The standard cleanup:

# PV name from the `kubectl get pv` listing above:
PV=pvc-7c8f2d8e-4b1a-4f6d-9a03-6d2e5b8c71f4

# 1. Inspect the released PV
kubectl get pv "$PV" -o yaml

# 2. Remove the claimRef to make the PV Available
kubectl edit pv "$PV"
# Delete spec.claimRef

# 3. Recycle or delete the underlying storage
# Option A: rebind to a new PVC
# Option B: delete the underlying storage (EBS volume, NFS export, etc.)
# Option C: keep the storage and audit before deleting

Production lifecycle management

The operational discipline:

• Reclaim policy matches the backup strategy. Retain for stateful data; Delete for ephemeral.
• Snapshots are scheduled and tested. Snapshots are the safety net for Delete reclaim.
• PVC deletion is gated by RBAC. Production PVC deletion requires explicit approval.
• Released PVs are audited. A monthly audit catches the accumulation.
• Cluster bootstrap documents the reclaim policies. Every StorageClass has a documented reclaim policy.

Quiz

Production discipline

• Retain for stateful data; Delete for ephemeral. The reclaim policy matches the backup strategy.
• Snapshots are the safety net for Delete. Every Delete-reclaim workload must have a tested snapshot restore.
• PVC deletion is gated by RBAC. Production PVC deletion requires explicit approval.
• Released PVs are audited monthly. The audit catches the accumulation.
• Recycle is deprecated, and still accepted. The API server will not stop you; nothing implements it. Migrate inherited Recycle PVs to Delete with snapshots.
• The cluster bootstrap documents the reclaim policies. Every StorageClass has a documented reclaim policy; every production workload has a documented recovery procedure.