PVC, PV, and snapshot relationships — what to back up and in what order

A Kubernetes backup program that ignores the dependency graph between PVC, PV, VolumeSnapshot, and VolumeSnapshotContent will either fail to capture what it should or fail to clean up what it did. This lesson walks the dependency graph, the deletion order, the restore order, and the operational discipline of treating ordering as a runbook rather than a guess.

The dependency graph

flowchart LR
    A[StorageClass] --> B[PV]
    B --> C[PVC]
    C --> D[Pod]
    E[VolumeSnapshotClass] --> F[VolumeSnapshotContent]
    F --> G[VolumeSnapshot]
    G -->|references| C
    H[Restore PVC] -->|dataSource| G

The forward dependencies:

• A StorageClass defines how a PV is provisioned (which CSI driver, which parameters, which reclaimPolicy).
• A PV is the cluster-scoped materialisation of a volume; the PVC claims it.
• A PVC is mounted by a Pod and provides the volume inside the container.
• A VolumeSnapshotClass defines how a VolumeSnapshotContent is provisioned.
• A VolumeSnapshotContent is the cluster-scoped materialisation of a snapshot.
• A VolumeSnapshot references the source PVC (or another VolumeSnapshot) and is bound to a VolumeSnapshotContent.
• A restore PVC references a VolumeSnapshot via dataSource to provision a new volume from the snapshot.

The deletion order

flowchart TD
    A[1. VolumeSnapshot] --> B[2. VolumeSnapshotContent]
    B --> C[3. Restore PVC]
    C --> D[4. PVC]
    D --> E[5. PV]
    E --> F[6. StorageClass]

When tearing down a backup’s infrastructure, the order is the reverse of the dependency:

1. VolumeSnapshot first. This triggers the external-snapshotter sidecar to call CSI DeleteSnapshot. The cloud-side snapshot is removed.
2. VolumeSnapshotContent (if still bound). The sidecar deletes this after the CSI call returns. You rarely need to delete this manually.
3. Restore PVC. Only if it was created from the snapshot and is no longer needed.
4. PVC. Deleting a PVC triggers the CSI DeleteVolume on the underlying volume (if the reclaimPolicy is Delete). With Retain, the PV stays.
5. PV. Only if reclaimPolicy was Delete and the PV is orphaned, or if you want to force the cleanup.
6. StorageClass. Almost never deleted; it is part of the cluster’s infrastructure.

The restore order

When restoring a workload after a cluster loss, the order is the forward dependency order:

flowchart TD
    A[1. CRDs] --> B[2. StorageClass and CSI driver]
    B --> C[3. ConfigMaps and Secrets]
    C --> D[4. VolumeSnapshot]
    D --> E[5. PVC referencing VolumeSnapshot]
    E --> F[6. Workloads and Services]
    F --> G["7. Ingress, NetworkPolicy, RBAC"]
    G --> H[8. Validate]

1. CRDs first. If a workload uses a CRD (cert-manager, ingress-nginx, Argo CD), the CRD definitions must exist before any object of that kind is restored. Restoring a Certificate before the CRD exists causes the API to reject it.
2. StorageClass and CSI driver. A restore PVC references a StorageClass. The CSI driver must be running and the StorageClass must be defined before the PVC can be provisioned.
3. ConfigMaps and Secrets. Workloads often mount configuration. Restore the data plane config first so the workload can read its config when it starts.
4. VolumeSnapshots. The cloud-side snapshots must exist (either recreated by Velero or by the restore tool) before any restore PVC can reference them.
5. PVCs referencing the snapshots. The PVC’s dataSource points to the VolumeSnapshot; the CSI driver clones the snapshot into a new volume.
6. Workloads and Services. Once the PVCs are bound, the StatefulSet/Deployment can start and the Services can route traffic.
7. Ingress, NetworkPolicy, RBAC. The routing and access controls come last so they do not block the workload from starting.
8. Validate. End-to-end tests against the live workloads.

The validation cadence

The restore order is part of the runbook; the validation cadence proves it works:

Each test exercises a different scope. Skipping any one leaves a gap. The full cluster restore is the only test that exercises the complete ordering chain.

The operational failure modes

Restoration fails in production for predictable reasons:

• CRDs missing. The restore tool creates objects of a kind whose CRD does not exist. The API server rejects with “no matches for kind”.
• StorageClass missing. The restore PVC references a StorageClass that does not exist. The PVC stays Pending.
• VolumeSnapshot not found. The restore PVC’s dataSource points to a VolumeSnapshot that was not restored. The PVC stays Pending with FailedBinding.
• CSI driver not running. The PV cannot be provisioned because no driver handles the StorageClass. The PVC stays Pending.
• NetworkPolicy restored before workloads. The workload starts but cannot reach its dependencies because the NetworkPolicy has not yet been reconciled. In practice this is rare but happens when NetworkPolicies are restored before the namespaces they target.
• RBAC missing. The workload’s ServiceAccount lacks the RoleBindings. The Pod starts but cannot read ConfigMaps, Secrets, or the API server.

Quiz

Production discipline

The dependency graph in production rests on five non-negotiable elements:

• Document the deletion order. The runbook lists the order in which to tear down backup infrastructure, namespace by namespace.
• Document the restore order. The runbook lists the forward dependency order, with explicit pauses for reconciliation between phases.
• Validate the orders quarterly. A runbook that has never been executed is a hope. The full cluster restore test exercises both.
• Use the same tool for backup and restore. Mixing Velero for backup with hand-rolled restore scripts is a recipe for ordering bugs. The same tool that captured the data knows the order to restore it.
• Capture ordering in the tool, not in human memory. Velero’s restore order is encoded in its controller logic. Hand-rolled scripts require the operator to remember the order. The tool is more reliable than the human.

Restore ordering is part of the backup program. A backup program without an ordering runbook is a collection of objects waiting to be corrupted.