The CSI node plugin — Stage, Publish, Format, Mount

The CSI node plugin is the per-node operator that formats, mounts, and unmounts volumes. This lesson walks the operations, the staging path, the bind-mount into the Pod, and the production discipline.

Node plugin responsibilities

The node plugin implements the CSI Node Service:

sequenceDiagram
    participant K as kubelet
    participant NP as CSI node plugin
    participant BE as Block device
    participant S as Staging path
    participant P as Pod target path
    Note over K,NP: Staging
    K->>NP: NodeStageVolume (volumeHandle, stagingPath)
    NP->>BE: format (if needed) and mount to stagingPath
    BE-->>NP: mounted at stagingPath
    NP-->>K: success
    Note over K,NP: Publishing
    K->>NP: NodePublishVolume (stagingPath, targetPath)
    NP->>S: bind-mount to targetPath
    NP-->>K: success
    Note over K,NP: Unpublishing (reversed)
    K->>NP: NodeUnpublishVolume (targetPath)
    NP->>P: unmount
    Note over K,NP: Unstaging (reversed)
    K->>NP: NodeUnstageVolume (stagingPath)
    NP->>S: unmount

The staging path

The staging path is a node-local directory where the CSI node plugin formats and mounts the volume:

/var/lib/kubelet/plugins/<csi-driver>.csi.k8s.io/
  pvc-<pvc-uid>/
    mount          # the mounted filesystem

The staging path is shared between NodeStageVolume (which mounts the block device here) and NodePublishVolume (which bind-mounts from here to the Pod). The two-step process allows multiple Pods on the same node to share the same volume (one stage, multiple publish).

NodeStageVolume in detail

NodeStageVolume is the operation that formats and mounts the volume to the staging path:

message NodeStageVolumeRequest {
  string volume_handle = 1;       // from PV
  string publish_context = 2;     // from ControllerPublishVolume
  string staging_target_path = 3; // the staging path
  VolumeCapability volume_capability = 4;
  map<string, string> secrets = 5;
}

The node plugin:

1. Receives the request from the kubelet.
2. Identifies the block device (from publish_context, e.g., /dev/xvdba).
3. Formats the device if it is not yet formatted (e.g., mkfs.ext4).
4. Mounts the device to the staging path.
5. Returns success.

If the device is already formatted (idempotent), the plugin skips formatting and mounts.

NodePublishVolume in detail

NodePublishVolume is the operation that bind-mounts the staging path to the Pod’s target path:

message NodePublishVolumeRequest {
  string volume_handle = 1;
  string publish_context = 2;
  string staging_target_path = 3;
  string target_path = 4;        // Pod's mountPath
  VolumeCapability volume_capability = 5;
  bool readonly = 6;
}

The node plugin:

1. Receives the request from the kubelet.
2. Bind-mounts the staging path to the target path (e.g., /var/lib/kubelet/pods/<pod-uid>/volumes/...).
3. The kubelet then bind-mounts the target path into the container’s mount namespace.

The bind-mount chain:

block device
  -> staging path (/var/lib/kubelet/plugins/<csi>/pvc-<uid>/mount)
    -> kubelet target path (/var/lib/kubelet/pods/<pod-uid>/volumes/...)
      -> container mount path (/var/lib/app)

NodeExpandVolume in detail

NodeExpandVolume resizes the filesystem after the controller has expanded the underlying volume:

message NodeExpandVolumeRequest {
  string volume_handle = 1;
  string volume_path = 2;
  CapacityRange capacity_range = 3;
  string staging_target_path = 4;
}

The node plugin:

1. Receives the request from the kubelet.
2. Resizes the filesystem (e.g., resize2fs, xfs_growfs).
3. Returns success.

For online expansion, the Pod is running during the resize. For offline expansion, the Pod is stopped.

The DaemonSet deployment

The node plugin is a DaemonSet:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: ebs-csi-node
  namespace: kube-system
spec:
  template:
    spec:
      hostNetwork: true    # <-- required for node plugin
      containers:
      - name: ebs-csi-node
        image: amazon/aws-ebs-csi-driver:latest
        securityContext:
          privileged: true   # <-- required for mount operations
          mountPropagation: Bidirectional

The node plugin needs:

• hostNetwork: true: to communicate with the kubelet on localhost.
• privileged: true: to perform mount operations that require CAP_SYS_ADMIN.
• mountPropagation: Bidirectional: to allow mount propagation into the Pod.

The kubelet-CSI communication

The kubelet communicates with the CSI node plugin via a Unix domain socket:

/var/lib/kubelet/plugins/<csi-driver>.csi.k8s.io/
  csi.sock

The kubelet opens a connection to the socket, makes gRPC calls (NodeStageVolume, NodePublishVolume), and closes the connection. The CSI node plugin must be running and the socket must be present for the kubelet to mount volumes.

The node plugin verification

# Substitute your own driver's short name (ebs.csi.aws.com -> ebs):
CSI_DRIVER=ebs

# 1. The node plugin is running on every node
kubectl -n kube-system get pods -l app="$CSI_DRIVER",role=node -o wide
# READY   STATUS   NODE
# 1/1     Running  node-1
# 1/1     Running  node-2
# 1/1     Running  node-3

# 2. The socket is present
ls /var/lib/kubelet/plugins/"$CSI_DRIVER".csi.k8s.io/csi.sock

# 3. The CSIDriver object is registered
kubectl get csidriver
# NAME                  ATTACHREQUIRED   MAXSTORAGE_PERNODE
# ebs.csi.aws.com       true             0

# 4. Test mount a PVC
kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: csi-test
spec:
  containers:
  - name: test
    image: busybox
    command: ["sleep", "3600"]
    volumeMounts:
    - name: data
      mountPath: /data
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: test-pvc
EOF

Quiz

Production discipline

• The node plugin runs on every node. A DaemonSet ensures one Pod per node.
• privileged: true and hostNetwork: true are required. The node plugin needs elevated privileges for mount operations.
• The socket is the diagnostic goldmine. A missing socket blocks mounts; verify it on every node.
• Monitor NodeStageVolume and NodePublishVolume latency. Mount failures are user-visible; latency spikes are early signals.
• Test mount on every node. A new node may be missing the node plugin if the DaemonSet did not roll.