Deploying and operating CSI drivers in production

A CSI driver in production is not a single Pod; it is a collection of components: controller plugin, node plugin, RBAC, secrets, and possibly a snapshot controller. This lesson walks the deployment and the operational discipline.

The components

A production CSI driver deployment includes:

flowchart TB
    subgraph "kube-system namespace"
        CP[Controller plugin Deployment<br/>2 replicas]
        NP[Node plugin DaemonSet<br/>1 per node]
        SC[Snapshot controller Deployment]
        CRB[ServiceAccount + ClusterRole + ClusterRoleBinding]
    end
    subgraph "API server"
        CSR[CSIDriver object]
        CRDS[CRDs: VolumeSnapshotClass, VolumeSnapshot, VolumeSnapshotContent]
    end
    subgraph "Storage backend"
        BE[EBS / Ceph / NFS / etc.]
    end
    CP --> BE
    NP --> BE
    SC --> CRDS
    CP -.uses.-> CRB
    NP -.uses.-> CRB

Each component:

• Controller plugin: Deployment, 2+ replicas with leader election.
• Node plugin: DaemonSet, 1 Pod per node.
• Snapshot controller: Deployment for VolumeSnapshot / VolumeSnapshotContent (separate from the CSI driver).
• CSIDriver object: registers the driver with the API server.
• RBAC: ServiceAccount, ClusterRole, ClusterRoleBinding for the controller and node plugins.

Deploying the controller plugin

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ebs-csi-controller
  namespace: kube-system
spec:
  replicas: 2
  selector:
    matchLabels:
      app: ebs-csi-controller
  template:
    metadata:
      labels:
        app: ebs-csi-controller
    spec:
      serviceAccountName: ebs-csi-controller-sa
      containers:
      - name: csi-provisioner
        image: registry.k8s.io/sig-storage/csi-provisioner:v5.0.1
        args:
        - --csi-address=$(ADDRESS)
        - --feature-gates=HonorPVReclaimPolicy=true
        env:
        - name: ADDRESS
          value: /var/lib/csi/sockets/pluginproxy/csi.sock
      - name: csi-snapshotter
        image: registry.k8s.io/sig-storage/csi-snapshotter:v8.0.1
        args:
        - --csi-address=$(ADDRESS)
      - name: ebs-plugin
        image: amazon/aws-ebs-csi-driver:v1.30.0
        args:
        - --endpoint=$(CSI_ENDPOINT)
        - --leader-election=true
        env:
        - name: CSI_ENDPOINT
          value: unix:///var/lib/csi/sockets/pluginproxy/csi.sock
      volumes:
      - name: socket-dir
        emptyDir: {}

The pattern:

• The CSI driver container (ebs-plugin) implements the CSI spec.
• The external sidecars (csi-provisioner, csi-snapshotter) handle Kubernetes-specific watching logic.
• The sidecars communicate with the driver via a Unix socket.

Deploying the node plugin

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: ebs-csi-node
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: ebs-csi-node
  template:
    metadata:
      labels:
        app: ebs-csi-node
    spec:
      hostNetwork: true
      serviceAccountName: ebs-csi-node-sa
      containers:
      - name: csi-node-driver-registrar
        image: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.10.0
        args:
        - --csi-address=$(ADDRESS)
        - --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)
        env:
        - name: ADDRESS
          value: /csi/csi.sock
        - name: DRIVER_REG_SOCK_PATH
          value: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
      - name: ebs-plugin
        image: amazon/aws-ebs-csi-driver:v1.30.0
        args:
        - --endpoint=$(CSI_ENDPOINT)
        env:
        - name: CSI_ENDPOINT
          value: unix:///csi/csi.sock
        securityContext:
          privileged: true
        volumeMounts:
        - name: kubelet-dir
          mountPath: /var/lib/kubelet
          mountPropagation: Bidirectional
      volumes:
      - name: socket-dir
        hostPath:
          path: /var/lib/kubelet/plugins/ebs.csi.aws.com
          type: DirectoryOrCreate
      - name: kubelet-dir
        hostPath:
          path: /var/lib/kubelet
          type: Directory

The pattern:

• The CSI driver container implements the spec.
• The csi-node-driver-registrar sidecar registers the driver with the kubelet (creates the socket, updates the CSIDriver object).
• The driver needs privileged: true and access to /var/lib/kubelet.

The CSIDriver object

The CSIDriver object registers the driver with the API server:

apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:
  name: ebs.csi.aws.com
spec:
  attachRequired: true
  podInfoOnMount: false
  fsGroupPolicy: File
  storageCapacity: false
  volumeLifecycleModes:
  - Persistent
  - Ephemeral

Without the CSIDriver object, the kubelet does not know about the driver; the node plugin’s NodeGetInfo is not called; volume mounts fail.

RBAC

The controller and node plugins need RBAC:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: ebs-csi-controller-sa
  namespace: kube-system

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: ebs-csi-controller-role
rules:
- apiGroups: [""]
  resources: ["persistentvolumes"]
  verbs: ["get", "list", "watch", "create", "delete", "patch"]
- apiGroups: [""]
  resources: ["persistentvolumeclaims"]
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources: ["events"]
  verbs: ["list", "watch", "create"]
- apiGroups: ["storage.k8s.io"]
  resources: ["csinodes", "csidrivers"]
  verbs: ["get", "list", "watch"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: ebs-csi-controller-binding
subjects:
- kind: ServiceAccount
  name: ebs-csi-controller-sa
  namespace: kube-system
roleRef:
  kind: ClusterRole
  name: ebs-csi-controller-role
  apiGroup: rbac.authorization.k8s.io

The node plugin needs a separate RBAC for the csi-node-driver-registrar.

Secrets for credentials

Most CSI drivers need credentials to call the storage backend. The standard pattern:

apiVersion: v1
kind: Secret
metadata:
  name: ebs-csi-credentials
  namespace: kube-system
type: Opaque
stringData:
  access-key: <aws-access-key>
  secret-key: <aws-secret-key>

The CSI driver mounts the Secret and uses the credentials to call the backend API. In cloud-provider clusters, the driver can use IAM roles for service accounts (IRSA) to avoid storing credentials in the cluster.

The verification

After deployment, verify:

# 1. The CSIDriver is registered
kubectl get csidriver

# 2. The controller plugin is running
kubectl -n kube-system get pods -l app=ebs-csi-controller

# 3. The node plugin is running on every node
kubectl -n kube-system get pods -l app=ebs-csi-node -o wide

# 4. The CSINode object exists for every node
kubectl get csinode -o yaml

# 5. A test PVC binds
kubectl apply -f - <<EOF
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: test-pvc
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: ebs-sc
  resources:
    requests:
      storage: 1Gi
EOF
kubectl get pvc test-pvc -w

# 6. A test Pod mounts the PVC
kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: test-pod
spec:
  containers:
  - name: test
    image: busybox
    command: ["sleep", "3600"]
    volumeMounts:
    - name: data
      mountPath: /data
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: test-pvc
EOF

The upgrade discipline

A CSI driver upgrade is a cluster-wide change:

1. Read the release notes: breaking changes, new parameters, deprecated operations.
2. Test on a staging cluster: the upgrade may affect existing volumes.
3. Upgrade the controller plugin first: the controller is the entry point for new PVCs.
4. Roll the node plugins: the node plugin is a DaemonSet; rolling restart affects mount operations.
5. Verify existing volumes are unaffected: a PVC bound before the upgrade should still work after.
6. Monitor for errors: check CSI plugin logs for unexpected errors.

Quiz

Production discipline

• A CSI driver is multiple components. Controller, node plugin, RBAC, secrets, possibly snapshot controller.
• The CSIDriver object is mandatory. Without it, the kubelet does not know about the driver.
• Use the external sidecar pattern. The Kubernetes-CSI organization maintains the sidecars; reuse them.
• Test the upgrade on staging first. A CSI driver upgrade can break running workloads.
• Verify the deployment cluster-wide. The controller runs as a Deployment (a few replicas); the node plugin runs as a DaemonSet (one per node).
• Document the deployment in the cluster bootstrap. Every CSI driver in the cluster must be documented.