Storage restoration — CSI drivers, StorageClasses, and snapshot recovery

Phase 4 of complete cluster loss recovery is restoring storage — the CSI drivers, StorageClasses, and VolumeSnapshots that workloads depend on for persistent data. This lesson walks the CSI driver install, StorageClass configuration, IAM role setup, VolumeSnapshot restoration from Velero, and the operational discipline.

The CSI driver install

flowchart LR
    A["kubeadm init: control plane up"] --> B[Apply CSI driver manifest]
    B --> C["CSI controller: Deployment"]
    B --> D["CSI node plugin: DaemonSet"]
    C --> E[PV provisioning functional]
    D --> F[Volume mount functional]

The CSI driver is installed as a Deployment (the controller) and a DaemonSet (the node plugin). The controller talks to the cloud’s API to provision volumes; the node plugin mounts volumes on nodes.

# AWS EBS CSI driver
kubectl apply -k "github.com/kubernetes-sigs/aws-ebs-csi-driver/deploy/kubernetes/overlays/stable/?ref=release-1.34"

# Azure Disk CSI driver
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/azuredisk-csi-driver/master/deploy/example/storageclass-azure.yaml

# GCE PD CSI driver
kubectl apply -f https://raw.githubusercontent.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver/master/deploy/example/storageclass.yaml

# Ceph RBD CSI driver
kubectl apply -f https://raw.githubusercontent.com/ceph/ceph-csi/master/deploy/rbd/kubernetes/csi-rbdplugin.yaml

The snapshot-controller and external-snapshotter

flowchart LR
    A[VolumeSnapshot CRD] --> B[snapshot-controller]
    B --> C[VolumeSnapshotContent]
    C --> D[external-snapshotter sidecar]
    D --> E[CSI CreateSnapshot]

For VolumeSnapshot support, install:

• snapshot-controller — cluster-wide controller that watches VolumeSnapshot and binds VolumeSnapshotContent. From the external-snapshotter repository.
• external-snapshotter sidecar — bundled with most CSI drivers; watches VolumeSnapshotContent and calls the CSI driver.

# snapshot-controller
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/v8.0.0/deploy/kubernetes/snapshot-controller/setup-snapshot-controller.yaml

# VolumeSnapshotClass
kubectl apply -f - <<EOF
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: csi-aws-vsc
driver: ebs.csi.aws.com
deletionPolicy: Delete
parameters:
  tagSpecification: "true"
EOF

StorageClass configuration

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: gp3
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
  iops: "3000"
  throughput: "125"
  fsType: ext4
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete
allowVolumeExpansion: true

The StorageClass parameters are driver-specific. The EBS CSI driver accepts type, iops, throughput, fsType; the Azure Disk driver accepts skuName, location, cachingMode; Ceph RBD accepts pool, imageFormat, imageFeatures.

IAM role configuration

Cloud CSI drivers need IAM permissions:

# AWS: create the EBS CSI driver's IAM role
eksctl create iamserviceaccount \
  --name ebs-csi-controller-sa \
  --namespace kube-system \
  --cluster prod-cluster \
  --attach-policy-arn arn:aws:iam::aws:policy/service-role/AWS_EBS_CSI_DriverPolicy \
  --role-only \
  --approve

The IAM role must be referenced in the CSI driver’s ServiceAccount annotation:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: ebs-csi-controller-sa
  namespace: kube-system
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::123:role/AmazonEKS_EBS_CSI_DriverRole

Without the IAM role, the CSI driver fails to provision volumes.

VolumeSnapshot restoration from Velero

velero restore create cluster-snapshots \
  --from-backup daily-full-20260816030000 \
  --include-resources volumesnapshots,volumesnapshotcontents

Velero restores the VolumeSnapshot and VolumeSnapshotContent objects. The VolumeSnapshots become available for restore PVCs to reference.

kubectl get volumesnapshot -A

NAMESPACE   NAME                       READYTOUSE   SOURCEPVC      AGE
prod-data   postgres-daily-2026-08-16  true         postgres-data  2m

The VolumeSnapshot must be ReadyToUse=true. If it is not, the snapshot-controller has not bound the VolumeSnapshotContent, or the CSI driver has not yet readied the snapshot.

End-to-end storage test

# Create a test PVC
kubectl apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-pvc
spec:
  storageClassName: gp3
  accessModes: [ReadWriteOnce]
  resources:
    requests:
      storage: 1Gi
EOF

# Wait for the PVC to bind
kubectl get pvc test-pvc
# STATUS should be Bound within seconds

A Bound PVC indicates the CSI driver can provision volumes and the StorageClass is correct. An unbound PVC indicates a CSI driver or StorageClass misconfiguration.

The operational failure modes

Phase 4 fails for predictable reasons:

• CSI driver not installed. No CSI driver is running; every PVC stays Pending.
• IAM role missing. The CSI driver cannot provision volumes in the cloud. PVC creation returns permission errors.
• StorageClass parameters wrong. The driver rejects the parameters (e.g., invalid IOPS value).
• VolumeSnapshot not ReadyToUse. The snapshot exists but the cloud-side snapshot is not ready.
• Snapshot in different region. The cloud-side snapshot is in a different region than the new cluster. The CSI driver cannot read it.

Quiz

Production discipline

Phase 4 (storage restoration) in production rests on five non-negotiable elements:

• Match the CSI driver to the storage backend. EBS for AWS, Azure Disk for Azure, GCE PD for GCP, Ceph RBD for Ceph. Mismatched drivers produce unreadable snapshots.
• Install the snapshot-controller. It is not shipped with kubeadm. A cluster without it cannot snapshot volumes at all.
• Configure IAM roles for cloud drivers. The CSI driver needs IAM permissions to provision and attach volumes. Test the IAM role before workloads depend on it.
• Test storage end-to-end. Create a test PVC, verify it binds, mount it in a debug Pod, write data. Verify the round-trip before production.
• Document the StorageClass parameters. Each StorageClass’s parameters must be in the runbook. Mismatched parameters are a common error.

Storage is the prerequisite for stateful workloads. Storage that has not been tested is storage whose recovery is unknown. The discipline is to verify storage end-to-end before applying workloads.