The kubelet volume manager — how volumes become Pod mounts

The kubelet is the per-node agent that turns a Pod’s PVC into a mounted filesystem. This lesson walks the kubelet’s volume manager — the reconciliation loop, the on-disk state, and the diagnostic patterns for volume attach and mount failures.

The reconciliation loop

The kubelet’s volume manager is a reconciliation loop similar to the controllers in the control plane:

flowchart LR
    A[Watch API server for Pods scheduled to this node] --> B{Pod has volumes?}
    B -->|yes| C[For each volume]
    C --> D[Is volume attached?]
    D -->|no| E[Call CSI NodePublishVolume: attach + mount]
    D -->|yes| F[Is volume mounted?]
    F -->|no| G[Call CSI NodePublishVolume: mount only]
    F -->|yes| H[Volume ready]
    H --> I[Update Pod status: volumes ready]

The loop runs continuously, watching for Pods that are scheduled to the node. When a Pod with volumes is detected, the kubelet calls the CSI node plugin to attach (if needed) and mount the volume.

The on-disk state

The kubelet stores volume metadata on the node:

/var/lib/kubelet/pods/<pod-uid>/
  containers/
  volumes/
    kubernetes.io~csi/
      pvc-<pvc-uid>/
        mount          # the mounted filesystem
        vol_data.json  # metadata about the volume

The vol_data.json file contains:

{
  "spec": {
    "volumeSource": {
      "csi": {
        "driver": "ebs.csi.aws.com",
        "volumeHandle": "vol-0123456789abcdef0",
        "fsType": "ext4"
      }
    }
  },
  "attachmentMetadata": {
    "devicePath": "/dev/xvdba"
  }
}

This is the kubelet’s record of what is attached, what is mounted, and what the CSI driver returned. When the kubelet restarts, it reads this file to re-establish the volume state.

# Substitute your own value before running:
POD=postgres-0

# Inspect the kubelet's volume state for a Pod
POD_UID=$(kubectl get pod "$POD" -o jsonpath='{.metadata.uid}')
ls -la "/var/lib/kubelet/pods/$POD_UID/volumes/"
cat "/var/lib/kubelet/pods/$POD_UID"/volumes/kubernetes.io~csi/pvc-*/vol_data.json

The attach sequence

For a Pod with a PVC, the kubelet’s sequence is:

1. Pod scheduled to the node.
2. kubelet sees the Pod in its watch.
3. kubelet calls CSI NodeStageVolume — formats the block device (if needed) and mounts it to a staging path (typically /var/lib/kubelet/plugins/<csi-driver>/pvc-<pvc-uid>/mount).
4. kubelet calls CSI NodePublishVolume — bind-mounts the staging path into the Pod’s volume mount path.
5. kubelet reports volume status to the API server.

The CSI node plugin implements steps 3 and 4. The kubelet implements step 5 and the orchestration.

# Substitute your own value before running:
CSI=ebs-csi

# Trace the CSI node plugin's logs
kubectl -n kube-system logs -l "app=$CSI,role=node" --tail=50

# Look for NodeStageVolume and NodePublishVolume calls
# The logs show the volumeHandle, the staging path, and the mount path

Failure modes at the kubelet level

The kubelet’s volume subsystem has several failure modes:

• Volume stuck in attaching: the CSI node plugin returns an error; the kubelet retries with backoff. Inspect the CSI node plugin logs.
• Volume stuck in mounting: the filesystem fails to mount (wrong fsType, missing device). Inspect the kubelet logs.
• Volume mounted but not visible to the Pod: the bind-mount into the Pod’s namespace failed. Inspect the kubelet’s container runtime logs.
• Volume fails to detach: when the Pod is deleted, the kubelet calls NodeUnpublishVolume. A failure leaves the volume mounted on the node. Inspect the kubelet logs.

# Kubelet volume errors
journalctl -u kubelet | grep -i 'volume\|csi\|mount'

# Container runtime errors
journalctl -u containerd | grep -i 'volume\|mount'

The volume manager’s idempotency

The kubelet’s volume manager is idempotent: if a Pod is already mounted, calling CSI again is a no-op. This is important because the kubelet restarts, the node reboots, and the CSI node plugin can fail and restart — the volume manager must converge to the desired state.

The idempotency is implemented by the CSI spec: each CSI call (NodeStageVolume, NodePublishVolume) is documented as idempotent. The CSI driver implements idempotency; the kubelet assumes idempotency.

Quiz

Production discipline

• The kubelet’s on-disk state is the diagnostic starting point. /var/lib/kubelet/pods/<pod-uid>/volumes/ contains the volume metadata.
• The reconciliation loop is idempotent. A kubelet restart, a CSI node plugin restart, or a node reboot converges to the desired state.
• Two-step mount: stage then publish. The kubelet stages to a node-local path, then publishes to the Pod. This separation allows multi-Pod sharing.
• CSI node plugin logs are the diagnostic for attach and mount failures. The kubelet orchestrates; the CSI driver implements.