Mount failures — wrong fsType, missing secrets, and the kubelet-level diagnostic

Mount failures are the most common volume incident at the node level: the volume is attached, but the kubelet cannot mount it into the Pod. The cause is one of several categories: fsType, secret, format, stale state, or permission. This lesson walks the diagnostic.

The mount operation

Mount is NodeStageVolume (format + mount to staging path) followed by NodePublishVolume (bind-mount to Pod’s target path). The kubelet calls both; the CSI node plugin executes them.

sequenceDiagram
    participant K as kubelet
    participant NP as CSI node plugin
    participant D as Block device
    participant S as Staging path
    participant P as Pod target path
    K->>NP: NodeStageVolume (volumeHandle, stagingPath)
    NP->>D: format (if needed) and mount to stagingPath
    K->>NP: NodePublishVolume (stagingPath, targetPath)
    NP->>P: bind-mount

The mount can fail at any stage: format, mount to staging, or bind-mount to Pod.

Cause 1: wrong fsType

The PVC’s volumeMode and the StorageClass’s fsType may be inconsistent. For example:

• A CSI driver expects ext4 but the volume was formatted with xfs by another tool.
• The fsType parameter in the StorageClass is wrong for the workload.

# Substitute your own value before running:
POD=postgres-0

kubectl describe pod "$POD"
# Events:
#   Warning  FailedMount  ... mount failed: fsType is not supported
#   Warning  FailedMount  ... wrong fs type, bad superblock

The fix: verify the fsType matches the actual filesystem on the volume. The CSI driver’s NodeGetCapabilities returns supported fsTypes; the StorageClass’s fsType parameter must match.

Cause 2: missing secret

Some CSI drivers need credentials (e.g., Ceph, NFS with Kerberos). The credentials are passed as a Secret reference in the StorageClass:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: cephfs
provisioner: cephfs.csi.ceph.com
parameters:
  clusterName: my-cluster
  fsName: myfs
mountOptions:
- _netdev
reclaimPolicy: Retain
volumeBindingMode: Immediate
# The Secret is referenced via the StorageClass parameters
# or via the PV's `csi.volumeAttributes`

If the Secret is missing or has the wrong content, the mount fails:

# Substitute your own value before running:
POD=postgres-0

kubectl describe pod "$POD"
# Events:
#   Warning  FailedMount  ... failed to get secret: secrets "ceph-secret" not found

The fix: create the Secret with the right content. The CSI driver’s documentation specifies the required keys.

Cause 3: format error

The CSI node plugin attempts to format the device, but the format fails:

# Substitute your own value before running:
CSI_NODE_POD=csi-cephfsplugin-7x2kd

kubectl logs -n kube-system "$CSI_NODE_POD"
# mkfs.ext4: /dev/xvdba contains a mounted filesystem

The causes:

• The device is already mounted (from a previous failed attempt).
• The device has a filesystem that the CSI driver cannot re-format.
• The device has bad blocks.

The fix: ensure the device is not mounted before staging; verify the CSI driver’s format idempotency.

Cause 4: stale mount

A previous mount is left in the kernel’s mount table. The new mount fails because the device is already mounted:

mount | grep /dev/xvdba
# /dev/xvdba on /var/lib/kubelet/plugins/.../mount type ext4 (rw,relatime)

The fix: unmount the stale mount:

umount /var/lib/kubelet/plugins/.../mount

Production rule: investigate why the stale mount exists; the kubelet should unmount on Pod termination.

Cause 5: permission denied

The kubelet or the CSI node plugin does not have permission to mount:

# Substitute your own value before running:
CSI_NODE_POD=csi-cephfsplugin-7x2kd

kubectl logs -n kube-system "$CSI_NODE_POD"
# mount: permission denied

The causes:

• The CSI node plugin is not running as privileged: true.
• The hostPath mount of /var/lib/kubelet is not configured correctly.
• SELinux or AppArmor is blocking the mount.

The fix: ensure the CSI node plugin’s security context allows mounts; verify the hostPath configuration.

The diagnostic ladder

flowchart TD
    A[Mount failure] --> B[kubectl describe pod]
    B --> C{kubelet events?}
    C -->|FailedMount| D[kubelet logs]
    C -->|FailedAttach| E[CSI controller logs]
    D --> F{Error message?}
    F -->|fsType| G[Fix fsType]
    F -->|secret| H[Create Secret]
    F -->|permission| I[Fix security context]
    F -->|stale mount| J[Unmount stale]
    F -->|format| K[Verify format]
    E --> L{Backend error?}
    L -->|yes| M[Fix backend]

The production fix

For each cause:

Quiz

Production discipline

• Mount failures surface in three layers. The diagnostic checks kubelet -> CSI node plugin -> kernel.
• Wrong fsType is the most common. The StorageClass’s fsType must match the device.
• Missing secret blocks authentication. Verify Secrets exist and have the right content.
• Stale mounts are a smell. Investigate why the kubelet left stale state.
• Permission requires privileged: true. The CSI node plugin’s security context must allow mounts.