Snapshot and restore at CSI — the storage recovery

A VolumeSnapshot lives on the same storage backend as the volume it copies, so it protects against a bad migration or a careless deletion and not against the array itself failing. Restoring never happens in place: you create a new PVC naming the snapshot as its dataSource, which means the workload must be repointed at the new claim before anything is really recovered. And because the CSI driver copies blocks rather than understanding the application, a snapshot taken while a database is mid-write restores exactly that — a torn file the application may refuse to open.

The VolumeSnapshot

A VolumeSnapshot is a Kubernetes object that represents a point-in-time copy of a PVC. The snapshot is created by the CSI driver.

flowchart LR
    A[PVC] --> B[VolumeSnapshot]
    B --> C[CSI driver]
    C --> D[Storage backend snapshot]

The VolumeSnapshot is the cluster’s data snapshot.

The diagnostic

The canonical diagnostic:

# Substitute your own values before running:
NS=prod
SNAPSHOT=data-billing-snapshot
CSI_APP=ebs-csi-controller

# 1. Check the VolumeSnapshot
kubectl get volumesnapshot -n "$NS"
kubectl describe volumesnapshot "$SNAPSHOT" -n "$NS"

# 2. Check the VolumeSnapshotContent
kubectl get volumesnapshotcontent

# 3. Check the CSI driver
kubectl logs -n kube-system -l app="$CSI_APP" --tail=200

# 4. Check the storage backend
# (backend-specific)

# 5. Check the events
kubectl get events -n "$NS" --field-selector involvedObject.name="$SNAPSHOT"

The diagnostic is the VolumeSnapshot, the VolumeSnapshotContent, the CSI driver, and the storage backend.

The restore

The restore is the recovery from a snapshot. The restore is performed by:

1. Creating a new PVC from the VolumeSnapshot.
2. The new PVC is bound to a new PV.
3. The PV is restored from the snapshot.

# Create a new PVC from the snapshot
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-billing-restored
  namespace: prod
spec:
  dataSource:
    name: data-billing-snapshot
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi

The restore is the cluster’s data recovery.

Common failures

• CSI driver does not support snapshots. The CSI driver does not support snapshots. The remediation is to replace the CSI driver.
• StorageClass does not allow snapshots. The StorageClass’s volumeSnapshotClass is not configured. The remediation is to configure the StorageClass.
• Snapshot is failing. The snapshot is failing. The remediation is to investigate the CSI driver.
• Restore is failing. The restore is failing. The remediation is to read the PVC’s events and confirm the requested size is at least the snapshot’s restoreSize.

The remediation

The remediation depends on the cause:

# Substitute your own values before running:
SC=gp3-encrypted
SNAPSHOT_CLASS=csi-snapclass
CSI_APP=ebs-csi-controller

# Option 1: Replace the CSI driver
# (CSI-specific)

# Option 2: Configure the StorageClass
kubectl patch storageclass "$SC" -p "{\"volumeSnapshotClass\":\"$SNAPSHOT_CLASS\"}"

# Option 3: Investigate the CSI driver
kubectl logs -n kube-system -l app="$CSI_APP"

The remediation is the data recovery.

Production discipline

A CSI snapshot and restore failure is the cluster’s hypothesis. The discipline is to walk the 11-step methodology applied to the storage layer, identify the cause, apply the remediation. The storage is the cluster’s data; the remediation is the data recovery.

• Check the VolumeSnapshot. The VolumeSnapshot is the cluster’s snapshot.
• Check the CSI driver. The CSI driver is the cluster’s storage runtime.
• Check the StorageClass. The StorageClass is the workload’s storage class.

Quiz