Provisioners and CSI drivers — what calls the storage backend

The provisioner is the field that connects a StorageClass to a CSI driver. The CSI driver is the software that talks to the storage backend. This lesson walks the provisioner, the controller and node plugins, and how to verify the provisioner is healthy.

What a provisioner does

The provisioner (a CSI driver) implements the operations that create, attach, mount, format, snapshot, expand, and delete volumes:

• CreateVolume: the provisioner creates a volume on the backend.
• DeleteVolume: the provisioner deletes the volume.
• ControllerPublishVolume / ControllerUnpublishVolume: attach/detach the volume to/from a node (for cloud block storage).
• NodeStageVolume / NodePublishVolume: stage and publish the volume on a node.
• CreateSnapshot / DeleteSnapshot: snapshot operations.
• ExpandVolume: expand the volume.

The provisioner is implemented as one or more Pods in the cluster:

flowchart LR
    A[StorageClass] --> B[CSI controller plugin<br/>Deployment / StatefulSet]
    B --> C[Storage backend: EBS, Ceph, NFS, etc.]
    D[CSI node plugin<br/>DaemonSet] --> E[Node operations]
    B <--> D

The controller plugin

The controller plugin runs the cluster-wide operations: CreateVolume, DeleteVolume, CreateSnapshot, ExpandVolume. It runs as a Deployment or StatefulSet, typically in the kube-system namespace.

# Substitute your own value before running:
CSI=ebs-csi

# List the controller plugin Pods
kubectl -n kube-system get pods -l "app=$CSI,role=controller"

The controller plugin is a singleton (one active replica, with standby replicas for HA). A failure of the controller plugin blocks PVCs that reference its StorageClasses.

The node plugin

The node plugin runs the per-node operations: NodeStageVolume, NodePublishVolume, NodeUnstageVolume, NodeUnpublishVolume. It runs as a DaemonSet, one Pod per node.

# Substitute your own value before running:
CSI=ebs-csi

# List the node plugin Pods
kubectl -n kube-system get pods -l "app=$CSI,role=node"

A failure of the node plugin on one node blocks volume mounts on that node. A failure on all nodes blocks all volume mounts.

Common provisioners

Each provisioner has its own parameters, performance characteristics, and operational considerations.

Verifying provisioner health

# Substitute your own values before running:
CSI=ebs-csi
NODE=node-1

# 1. The controller plugin is running
kubectl -n kube-system get pods -l "app=$CSI,role=controller"
# READY   STATUS
# 1/1     Running

# 2. The node plugin is running on every node
kubectl -n kube-system get pods -l "app=$CSI,role=node" -o wide
# READY   STATUS   NODE
# 1/1     Running  node-1
# 1/1     Running  node-2
# 1/1     Running  node-3

# 3. The CSIDriver object exists
kubectl get csinode "$NODE" -o yaml
# drivers:
# - name: ebs.csi.aws.com
#   nodeID: i-0123456789abcdef0

# 4. The CSI driver is registered with the API server
kubectl get csidriver
# NAME                    ATTACHREQUIRED   MAXSTORAGE_PERNODE
# ebs.csi.aws.com         true             0

# 5. Test with a PVC
kubectl apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-pvc
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: "<storageclass>"
  resources:
    requests:
      storage: 1Gi
EOF

# 6. Check the PVC's status
kubectl get pvc test-pvc -w
# STATUS should transition from Pending to Bound

Failure modes

The common provisioner failure modes:

• Controller plugin down: new PVCs Pending. Existing PVCs continue to function (the controller plugin is not on the data path).
• Node plugin down on one node: volume mounts on that node fail. Other nodes are unaffected.
• Node plugin down on all nodes: all volume mounts fail. PVCs are still bound but Pods cannot mount.
• Provisioner throttled by backend: PVCs Pending (e.g., AWS API throttling). The provisioner logs show the throttling.
• Provisioner RBAC misconfigured: PVCs Pending with “permission denied” in the logs.
• Provisioner parameters invalid: PVCs Pending with the provisioner’s validation error in the logs.

Quiz

Production discipline

• The provisioner is a SLO dependency. Its latency and reliability affect every PVC on its StorageClass.
• Verify the controller plugin is running. A controller failure blocks new PVCs.
• Verify the node plugin runs on every node. A node plugin failure blocks mounts on the affected node.
• Monitor provisioner metrics. CreateVolume latency, error rate, throttling. The provisioner’s health is observable.
• Test the provisioner with a PVC. A regular validation that creates a test PVC and verifies it binds within the expected time.