Persistent volumes and CSI snapshots · Backup & DR

An etcd snapshot restored in the previous lesson brings the API objects back and then stops at the edge of the volumes those objects point to, and that edge is where this lesson works. A recovered cluster can report a PersistentVolumeClaim as Bound and a pod as 1/1 Running over a mount point holding nothing at all, and every one of those states is correct as far as the control plane is concerned. What the volume contains is decided where the API never looks.

A claim is a request; a volume is the thing that answers it

The Kubernetes storage model separates the thing a workload asks for from the thing that satisfies the request, and the separation is what makes recovery reasoning possible at all.

A PersistentVolumeClaim is a request. It names a capacity, one or more access modes, a StorageClass, and optionally a data source — the field the second half of this lesson turns on. It is a namespaced object, and it is the only storage object a pod ever refers to. A pod spec mounts a claim by name; it does not know what answered the claim, where that storage physically is, or how many times it has been replaced.

A PersistentVolume is the object that represents the storage that answered. It carries the driver-specific handle for a real piece of storage: a directory on a node, an iSCSI target, an RBD image, a cloud disk. Binding pairs one claim with one volume, and once bound the pair is exclusive. The claim’s identity is a name in a namespace; the volume’s identity is an opaque handle to something a storage system created.

A StorageClass is the policy that gets applied when no volume exists yet. It names the provisioner that will create one, carries parameters that provisioner understands, and sets two fields that the rest of this lesson turns on. volumeBindingMode decides whether provisioning happens as soon as the claim appears or waits until a pod is scheduled, which is why a claim re-created during a recovery can sit Pending indefinitely on a cluster whose nodes cannot yet run the workload: the storage is waiting on a scheduling decision that has not been made, and nothing is wrong with the claim. reclaimPolicy decides what happens to the provisioned storage when the claim that caused it to exist is deleted.

The reclaim policy is worth stating in full because it is the field that converts an ordinary namespace cleanup into a data-loss event. Kubernetes documents Retain and Delete as the policies in current use, and Recycle as deprecated. Under Retain, deleting the claim leaves the volume object behind in a Released state with the underlying storage untouched, and an administrator has to decide what to do with it. Under Delete, removing the claim removes the volume object and asks the provisioner to destroy the storage asset it represents. Dynamically provisioned volumes inherit the policy from their class, and the class default is Delete. The value is stamped onto each volume as persistentVolumeReclaimPolicy at provisioning time, and it can be patched on a bound volume afterwards — which is the only way to change the outcome for storage that already exists.

So the sentence that matters for recovery is short. Deleting a claim is not a metadata operation. Under the default policy it is a storage operation, and the API gives you no more warning about it than it gives you about deleting a ConfigMap.

Measured: the claim came back `Bound` and the orders did not come back

A single-node k3s cluster running the local-path provisioner was given four objects — a namespace, a claim, a ConfigMap and a pod — and the pod wrote two orders into the volume. The provisioner satisfies a claim by creating a directory on the node, so the storage in question is visible from the host.

Read-only / Safethe application data, and the directory the provisioner created for it— EXECUTION VALIDATED on k3s v1.36.3+k3s1, single-node cluster with the local-path provisioner. Step 2 of the desired-state capture, which records this step's output and not the command lines that produced it, so the command shown stands for the read; the md5 line and the node path under the marker were emitted by the capture, and its two-space indentation is preserved.

$ kubectl -n rbdr-shop exec rbdr-orders-writer -- cat /data/orders.csv

  ORDER-1001,4500.00
ORDER-1002,1250.00
orders.csv md5: 9eb4e2ad8e08e1dcaaf87ababab964b0

--- where that data physically lives on the node ---
/var/lib/rancher/k3s/storage/pvc-6edd5db0-25f4-4b33-b26b-e2ad78aac9cf_rbdr-shop_rbdr-orders/orders.csv

The directory name encodes the whole binding: the volume’s identity pvc-6edd5db0-25f4-4b33-b26b-e2ad78aac9cf, the namespace rbdr-shop, and the claim name rbdr-orders. That string is the only durable link between the business data and the objects that describe it. The namespace was then deleted, which is an entirely routine operation and takes the claim with it.

Destructivedeleting a namespace is a storage operation under a Delete reclaim policy— EXECUTION VALIDATED on k3s v1.36.3+k3s1. Step 4 of the desired-state capture; the check under the marker line was run against the node filesystem path shown above, and the capture's two-space indentation is preserved.

$ kubectl delete namespace rbdr-shop

  namespace "rbdr-shop" deleted
namespace fully removed after 0s

--- is the PersistentVolume data still on disk? ---
GONE - the local-path provisioner reclaimed the volume with the PVC

Nothing here was a mistake in the tooling. The claim was deleted, the volume’s reclaim policy said what to do about that, and the provisioner did it. The orders were destroyed on the node before any recovery had even been attempted, which is the detail that makes the next transcript worth reading carefully: the recovery below is not a failed restore, it is a successful reconstruction of everything the cluster knew about.

Data-loss riskevery object restored, bound to a volume that never held the data— EXECUTION VALIDATED on k3s v1.36.3+k3s1, after the manifests were re-applied to the same cluster. Step 5 of the desired-state capture, taken once the pod reported Ready; the capture records this step's output and not the command lines it ran, and the marker lines and two-space indentation are the capture's own.

$ kubectl -n rbdr-shop get pvc,configmap,pod

--- every object is back ---
persistentvolumeclaim/rbdr-orders   Bound   pvc-a29538af-09d4-4bb8-82bb-b2c222b9db93   64Mi   RWO   local-path   <unset>   6s
configmap/kube-root-ca.crt   1     6s
configmap/rbdr-config        1     6s
pod/rbdr-orders-writer   1/1   Running   0     6s

--- and the business data? ---
total 8
drwxrwxrwx    2 root     root          4096 Aug 28 14:34 .
drwxr-xr-x    1 root     root          4096 Aug 28 14:34 ..
cat: can't open '/data/orders.csv': No such file or directory
command terminated with exit code 1

Read the volume identity in that listing against the one in the first transcript. The claim is Bound, but it is bound to pvc-a29538af-09d4-4bb8-82bb-b2c222b9db93 — a volume the provisioner created fresh, in a directory whose entire listing is two entries long and holds no file at all. Bound is a statement about the claim’s relationship to a volume. It says nothing whatsoever about what that volume contains, and a health check that reads Bound and 1/1 Running will report a fully recovered service on top of an empty disk.

The capture goes on to show what does bring the orders back, and it is worth being exact about the sequence because the exactness is the point. The order lines had to be written again before anything could be copied, since the originals were already destroyed. A tar archive of the volume directory was then written to /tmp on the node — 2560 bytes — the namespace was destroyed a second time, the manifests recreated the objects against a third volume identity, and only after the archive was unpacked into that third directory did the file read back at md5 9eb4e2ad8e08e1dcaaf87ababab964b0, the value recorded when the application first wrote it. The archive was the one artefact in the sequence that the reclaim policy had no authority over, because it was never inside a volume the cluster managed.

`VolumeSnapshot`, `VolumeSnapshotContent`, `VolumeSnapshotClass`

The obvious response to the transcripts above is that the cluster should have been taking snapshots of the volume, and Kubernetes has an API for precisely that. It mirrors the claim-and-volume model one level up.

A VolumeSnapshot is a namespaced request for a point-in-time copy of a named claim; it is the snapshot world’s PVC. A VolumeSnapshotContent is the cluster-scoped object representing a snapshot that actually exists in a storage system, carrying the driver’s snapshotHandle; it is the snapshot world’s PV. A VolumeSnapshotClass names the CSI driver to use, the parameters it understands, and a deletionPolicy of Delete or Retain that governs whether removing the content object also destroys the storage-side snapshot.

None of this is built into the Kubernetes control plane. The three resources are custom resource definitions shipped by the external-snapshotter project, and they need a snapshot controller running in the cluster plus a csi-snapshotter sidecar alongside each CSI driver that supports snapshots. A cluster where kubectl get volumesnapshotclass returns nothing does not have a broken snapshot feature; it has no snapshot feature. The capture cluster used above is one of those: it ran the local-path provisioner with no CSI snapshot driver and no snapshot controller installed, which is why every command in this section is shown without output rather than with output invented for it.

Read-only / Safethe first question to ask of any cluster that claims snapshot protection— Documented in the Kubernetes volume-snapshot documentation and the external-snapshotter project. This command was NOT executed on the capture cluster, which had no CSI snapshot driver installed, so no output is shown.

$ kubectl get volumesnapshotclass

Creating a snapshot is a request against a claim, and reading it back means watching its status:

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: rbdr-orders-0900
  namespace: rbdr-shop
spec:
  volumeSnapshotClassName: csi-example-snapclass
  source:
    persistentVolumeClaimName: rbdr-orders

The controller binds that object to a VolumeSnapshotContent, and the status fields worth alerting on are readyToUse, which only becomes true once the storage system reports the snapshot usable, and restoreSize. Recovery then happens by naming the snapshot as the data source of a new claim:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: rbdr-orders-restored
  namespace: rbdr-shop
spec:
  storageClassName: csi-example-sc
  dataSource:
    name: rbdr-orders-0900
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
  accessModes: ['ReadWriteOnce']
  resources:
    requests:
      storage: 64Mi

Why a snapshot restore hands you a second volume rather than an older one

The dataSource field does not rewind anything. It changes what the provisioner is asked to do at creation time.

When a claim with no data source is created, the CSI provisioner calls CreateVolume with an empty source and the storage system allocates blank space. When a claim names a VolumeSnapshot as its data source, the provisioner resolves that object to a VolumeSnapshotContent, reads the snapshotHandle out of it, and calls CreateVolume with that handle as the content source. The storage system produces a new volume pre-populated from the snapshot, a new PersistentVolume object is created to represent it, and the new claim binds to that. The original volume is not touched, not detached, and not altered.

Three operational consequences follow, and each of them is where a Kubernetes restore actually consumes time.

The pod must be repointed. Nothing in the workload spec changes just because a restored claim exists next to the original. A Deployment has to be edited to mount the new claim, and a StatefulSet cannot be edited that way at all, because its volumeClaimTemplates generate claim names from the ordinal — recovering a StatefulSet member normally means creating the restored claim under the exact name the template will produce and letting the controller adopt it.

Capacity is consumed twice. The original volume still exists, still bills, and still counts against any quota, until somebody deletes it — and deleting it runs the reclaim policy from the first half of this lesson.

The restore is as fast as the storage system chooses to make it. A driver may be able to present the new volume almost at once because it shares blocks with the snapshot, or it may copy the contents before the volume is reported ready, and the Kubernetes API does not distinguish the two: the claim is Pending and then it is Bound. Nothing in the VolumeSnapshot status — which carries readyToUse, restoreSize and a creation time — describes how long the volume behind it will take to become usable. That makes restore duration a number to measure in your own cluster rather than one to read off a feature matrix.

The two things a VolumeSnapshot is not

A CSI snapshot gives you a genuine, useful recovery point, and on storage that does copy-on-write it costs almost nothing at the moment it is taken: the ZFS capture this section returns to below recorded a freshly taken snapshot at rbdrprod/ledger@0900 0B 50.1M. That figure describes the moment rather than the recovery point. Once the live dataset had been rewritten past it, the same snapshot read rbdrprod/ledger@0900 47.0M 50.1M, because it had become the only thing still referencing the blocks production overwrote. A snapshot retention window is therefore a capacity commitment that grows with write volume, not with snapshot count, and a schedule sized against the first number is the one that fills the array at the second.

None of that is an argument against taking snapshots. It is an argument about what a schedule of them covers, because a CSI snapshot falls two things short of a backup and both shortfalls have already been measured elsewhere in this course.

It is not application-consistent unless the workload was quiesced. The snapshot controller asks the storage system to copy a volume at a point in time. Nothing in that sequence tells the process inside the container to flush its buffers, finish its transaction, or stop writing. What lands in the snapshot is whatever was on disk at that instant, which is the crash-consistent state Part IV of this course examined: a database will usually replay its log and come up, and a system that spreads one logical write across several files may not. Making it application-consistent requires a pre-snapshot action inside the container — a freeze, a flush, a BEGIN BACKUP-style call — orchestrated by something that knows the application. The CSI layer has no such knowledge and never claims to.

It usually lives in the same storage system as the volume it came from. The snapshotHandle in a VolumeSnapshotContent is a reference the driver hands back to its own array, its own Ceph pool, its own cloud disk service. That is the shared-fate problem from Part VI, restated in Kubernetes vocabulary, and Part VI measured what it costs:

Data-loss riska snapshot in the storage it came from, after that storage is gone— EXECUTION VALIDATED on zfs-2.4.1-1ubuntu5. Step 7 of the ZFS snapshot capture: the production pool's backing device was overwritten and re-attached, while the backup pool is a separate device that had received the same snapshots by zfs send.

$ zpool import

--- zpool import (can ZFS find anything to import?) ---
no pools available to import

--- can the 09:00 snapshot be reached? ---
cannot open 'rbdrprod': dataset does not exist

--- what the INDEPENDENT backup pool still holds ---
NAME                  USED  REFER
rbdrbkp               100M    24K
rbdrbkp/ledger        100M  50.1M
rbdrbkp/ledger@0900  50.0M  50.1M
rbdrbkp/ledger@0930     0B  50.1M

The ZFS mechanism and the CSI mechanism are unrelated in implementation and identical in this respect. A copy that the storage system holds internally disappears with the storage system. A copy that was sent somewhere else does not. Nothing about wrapping the first arrangement in Kubernetes objects changes which of the two you have.

Anti-pattern: hourly VolumeSnapshots on the tier-1 array, so snapshots are enough

The slide says hourly CSI VolumeSnapshots, thirty days of retention, and restores served by the array rather than by a copy across the network, and no individual claim on it is false. The arrangement still protects against exactly one class of event: something that damages the volume while the storage system itself keeps working. Deleted rows, a bad migration, a corrupted file — those it handles well.

It protects against nothing that takes the array with it, and it protects against nothing that reaches the array’s own control plane, because credentials that can create snapshots can usually delete them. The corrective is not to stop taking snapshots. It is to say out loud which failures the snapshot schedule covers, and to name the separate copy that covers the rest.

The snapshot objects are cluster state, and cluster state is lost with etcd

There is a second dependency that is easy to miss because it points the other way. VolumeSnapshot and VolumeSnapshotContent are API objects. They live in etcd, exactly like the claim, the pod and the ConfigMap in the transcripts above, and they are lost in exactly the same circumstances.

Losing etcd, or restoring it to a point before a snapshot was created, removes the cluster’s knowledge that the snapshot exists. Under a deletionPolicy of Retain the storage-side snapshot itself survives that — but a snapshot the cluster cannot see is not a recovery point you can use. Reconnecting it means creating a pre-provisioned VolumeSnapshotContent by hand, with the driver name and the exact snapshotHandle filled in, and that handle only exists in two places: inside the storage system, and in whatever record you kept outside the cluster.

Which makes the recording of those handles an operational obligation, not a nicety. The etcd snapshot from the previous lesson protects the objects; the storage system protects the blocks; and the handle that ties one to the other is the piece nobody backs up until the first time they need it. Export the snapshot inventory — namespace, claim, snapshot name, class, handle, creation time — on the same schedule as the snapshots themselves, and keep it where a lost cluster cannot take it with it.

What to take from this

On k3s v1.36.3+k3s1 with the local-path provisioner, deleting the namespace deleted the claim and the capture recorded GONE - the local-path provisioner reclaimed the volume with the PVC. The reclaim policy decided that, not the delete command.
Re-applying the manifests bound the same claim name rbdr-orders to pvc-a29538af-09d4-4bb8-82bb-b2c222b9db93, a different volume identity from the pvc-6edd5db0-25f4-4b33-b26b-e2ad78aac9cf directory that had held the orders. Bound describes a relationship, not contents.
The recovered pod reported 1/1 Running after 6s and answered cat: can't open '/data/orders.csv': No such file or directory with command terminated with exit code 1. A healthy workload is not evidence of present data.
The orders returned only from a 2560-byte tar archive held on the node outside any volume, unpacked into the directory of a third volume identity, where the file read back at md5 9eb4e2ad8e08e1dcaaf87ababab964b0. No reclaim policy had authority over that archive.
Shared fate is measurable. On zfs-2.4.1-1ubuntu5, after the production device was overwritten, zpool import reported no pools available to import and the 09:00 snapshot answered cannot open 'rbdrprod': dataset does not exist, while the independent pool still listed rbdrbkp/ledger@0900 50.0M 50.1M. A CSI snapshot beside its volume is that same arrangement.
Cheapness is a property of the moment, not of the recovery point. The same ZFS capture recorded rbdrprod/ledger@0900 0B 50.1M when the snapshot was taken and rbdrprod/ledger@0900 47.0M 50.1M once the live dataset had been rewritten past it, because the snapshot then held the only reference to the overwritten blocks.

Cross-course references

Kubernetes for Production Sysadmins — Part L (PersistentVolumes and Claims) and Part LI (StorageClasses) develop the binding and provisioning model that this lesson uses only as far as recovery requires, and Part LV (Storage Snapshots) teaches the VolumeSnapshot API as an operational feature; read them for the mechanism, and read this lesson for what the mechanism does and does not protect.
Ceph & Distributed Storage for Production Sysadmins — Part LXXXVIII (Kubernetes Storage Failure Scenarios) is the concrete answer to the shared-fate question posed abstractly here: when the CSI driver is Ceph, the snapshot the cluster created lives in the same RADOS cluster as the image it was taken from, and that part shows what that costs when the cluster degrades.
Linux for Production Sysadmins — Part III (Filesystems and Files) is the layer the volume in this capture actually was, a directory under /var/lib/rancher/k3s/storage on one node, so its durability is the durability of that node’s filesystem and nothing more, no matter what the claim object says.