PersistentVolumeClaims and their lifecycle

Why this matters in production

The PVC, the PV, and the RBD image are three objects with independent lifetimes, and they can diverge.

The lifecycle

flowchart TD
  A[PVC created] --> B[Provisioner creates RBD image]
  B --> C[PV created, bound to PVC]
  C --> D[Pod scheduled; image mapped and mounted]
  D --> E[Pod deleted; unmounted and unmapped]
  E --> F{PVC deleted?}
  F -->|no| D
  F -->|yes| G{reclaimPolicy}
  G -->|Delete| H[PV removed, RBD image deleted]
  G -->|Retain| I[PV Released; image remains]

kubectl get pvc,pv
rbd -p k8s-rbd ls

Where orphaning occurs

# reconcile: images in Ceph without a PV
rbd -p k8s-rbd ls | while read img; do
  kubectl get pv -o json | grep -q "$img" || echo "orphan: $img"
done

Running this periodically is what prevents orphaned images accumulating indefinitely, and on a busy cluster they do.

Expansion

# Substitute your own PVC name before running:
PVC=pgdata-postgres-0

kubectl patch pvc "$PVC" -p '{"spec":{"resources":{"requests":{"storage":"20Gi"}}}}'
kubectl get pvc "$PVC" -o jsonpath='{.status.capacity.storage}'

Expansion sequence:
  the resizer expands the RBD image
  the node plugin expands the filesystem on next mount, or online if supported
  the PVC status updates

Deletion

NAME=acme
IMAGE=vm-disk-01
kubectl delete pvc ${NAME}
kubectl get pv | grep ${NAME}
rbd -p k8s-rbd ls | grep ${IMAGE}

With Retain, the cleanup is manual:

NAME=acme
IMAGE=vm-disk-01
kubectl delete pv ${NAME}
rbd -p k8s-rbd rm ${IMAGE}

Both steps, in that order. Deleting the image first leaves a PV referencing nothing.

Reconciling state

# PVs whose images are gone
kubectl get pv -o json | python3 -c '
import sys, json, subprocess
pvs = json.load(sys.stdin)["items"]
imgs = set(subprocess.check_output(["rbd","-p","k8s-rbd","ls"]).decode().split())
for pv in pvs:
    csi = pv.get("spec", {}).get("csi", {})
    h = csi.get("volumeAttributes", {}).get("imageName")
    if h and h not in imgs:
        print("PV", pv["metadata"]["name"], "references missing image", h)'

Quiz

Production discipline

Script a periodic reconciliation between Ceph images and PV objects — Retained volumes accumulate silently and nothing else reports them. When cleaning up manually, delete the image before the PV; the PV is the only record of which image belongs to what.

Cross-course references

• Kubernetes: orphaned resources across controller boundaries need the same reconciliation
• Linux: any two systems tracking the same objects will diverge without reconciliation