Attach vs mount vs format — the three CSI operations and their separation

Three operations take a volume from “exists on the backend” to “filesystem mounted in a Pod”: attach, mount, and format. Each has a specific responsibility; each happens at a specific layer; each can fail independently. This lesson walks the three and the production discipline for diagnosing failures at each stage.

The three operations

sequenceDiagram
    participant CP as CSI controller plugin
    participant BE as Backend
    participant NP as CSI node plugin
    participant N as Node (kernel)
    participant P as Pod
    Note over CP,BE: Attach (controller-level)
    CP->>BE: attach volume to node
    BE-->>CP: device path (/dev/xvdba)
    Note over NP,N: Format (node-level, first time only)
    NP->>N: mkfs.ext4 /dev/xvdba
    Note over NP,N: Mount (node-level)
    NP->>N: mount /dev/xvdba /var/lib/kubelet/plugins/.../mount
    Note over NP,N: Publish (node-level)
    NP->>N: bind-mount stagingPath -> podPath
    Note over P,N: Pod sees the filesystem
    P->>N: read/write at /var/lib/app

Attach: the controller-level operation

Attach is ControllerPublishVolume (and its inverse, ControllerUnpublishVolume). It is called by the kube-controller-manager, executed by the CSI controller plugin, and makes the volume visible to a node at the hardware level.

message ControllerPublishVolumeRequest {
  string volume_handle = 1;
  string node_id = 2;
  VolumeCapability volume_capability = 3;
  bool readonly = 4;
  map<string, string> secrets = 5;
  string volume_context = 6;
}

What attach does:

• EBS: ec2:AttachVolume API call; the EBS volume is attached to the EC2 instance as /dev/xvdba (or similar).
• GCE PD: disks.attach API call; the PD is attached to the GCE instance.
• Ceph RBD: rbd map command; the RBD image is mapped to a block device on the node.

The result is a device path on the node, accessible via the kernel’s block layer. The device is not yet mounted or formatted.

Format: the node-level operation

Format is part of NodeStageVolume. It happens on the node, executed by the CSI node plugin, and prepares the filesystem on the block device.

What format does:

• Check if the device has a filesystem (e.g., via blkid).
• If not, create a filesystem (e.g., mkfs.ext4, mkfs.xfs).
• The fsType is determined by the VolumeCapability’s FilesystemVolume.fs_type.

Format happens once per volume (the first time the volume is staged on any node). Subsequent NodeStageVolume calls are idempotent: the format is skipped if the filesystem already exists.

# Manual format (for understanding)
mkfs.ext4 /dev/xvdba
# Filesystem UUID: ...

Mount: the node-level operation

Mount is the second part of NodeStageVolume (after format). It happens on the node, executed by the CSI node plugin, and makes the filesystem accessible at the staging path.

What mount does:

• Mount the block device to the staging path (e.g., /var/lib/kubelet/plugins/<csi-driver>/pvc-<pvc-uid>/mount).
• Use the mount options from VolumeCapability.FilesystemVolume.mount_flags.
• Verify the mount succeeded.

After mount, the staging path contains the filesystem’s contents. The Pod’s bind-mount (from NodePublishVolume) makes the contents visible inside the container.

# Manual mount (for understanding)
mount -t ext4 -o noatime /dev/xvdba /var/lib/kubelet/plugins/.../mount

The ordering matters

The operations must happen in order:

1. Create (controller): the volume exists on the backend.
2. Attach (controller): the volume is attached to the node.
3. Stage (node, includes format + mount): the volume is formatted and mounted to the staging path.
4. Publish (node): the volume is bind-mounted to the Pod’s target path.

Each operation’s failure blocks the next:

• Create fails: no volume to attach.
• Attach fails: no device to format/mount.
• Stage fails: no filesystem to bind-mount.
• Publish fails: Pod cannot access the filesystem.

Failure modes at each stage

The diagnostic ladder is: PV/PVC events -> controller plugin logs -> node plugin logs -> node kernel logs (dmesg) -> cloud-provider metrics.

Volume mode Block vs Filesystem

The volumeMode in the PVC affects which operations happen:

• Filesystem (default): format + mount + bind-mount. The Pod sees a directory.
• Block: attach only, no format, no mount, no bind-mount. The Pod sees a raw block device.

For Block volumes:

• No mkfs is run; the device is exposed raw.
• The Pod’s volumeDevices (not volumeMounts) reference the volume.
• The application manages the filesystem (e.g., a database using a custom block-based storage engine).

Quiz

Production discipline

• Each operation has its own failure mode. The diagnostic ladder is attach -> format -> mount -> publish.
• Format is idempotent. It checks for an existing filesystem before formatting.
• Block mode bypasses format and mount. The Pod sees a raw device.
• Mount options are part of the StorageClass. Specify them in the StorageClass for the workload.
• Test format behavior on existing volumes. A CSI driver that always formats will overwrite data.