high riskdata loss risk~30 min

Runbook: Troubleshoot CSI Mount Failure

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Confirm the PVC is bound: kubectl get pvc <name> -n <ns> reports Bound and a non-empty VOLUME
· Capture the PV: kubectl get pv <pv-name> -o yaml
· Capture the StorageClass: kubectl get storageclass <sc> -o yaml
· Capture the CSI driver pods: kubectl -n <csi-namespace> get pods -o wide -l app=<csi-driver>
· Capture the Pod events: kubectl describe pod <pod> -n <ns> | sed -n "/Events:/,$p"
· Capture the node kubelet journal: kubectl get pod <pod> -n <ns> -o jsonpath='{.spec.nodeName}' | xargs -I{} ssh {} sudo journalctl -u kubelet --since '10 min ago' | grep -i mount

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Identify the mount stage that failed: attach (volume not visible to the node), format (volume not yet formatted), mount (volume not mounted into the Pod), or filesystem access (volume mounted but read-only)
2Read the Pod events and kubelet journal to localise the failure
3For attach failure: confirm the CSI controller (csi-<driver>-controller) is healthy and that the volume exists in the storage backend
4For format failure: confirm the volume is new or has been previously formatted; the CSI driver handles fsGroup and permissions
5For mount failure: confirm the CSI node plugin (csi-<driver>-node) is Running on the affected node, and the volume is attached at the device path
6For filesystem access failure: confirm the Pod securityContext.fsGroup matches the volume permissions
7Apply the smallest fix: restart the CSI node plugin on the stuck node, fix the storage backend, re-attach the volume, or correct the fsGroup
8Verify the Pod mounts the volume: kubectl exec <pod> -- ls /<mount-path> returns the expected contents
9Confirm the application can read and write to the mount

4 · Verification

Confirm the procedure actually fixed the problem.

✓kubectl get pod <pod> -n <ns> reports Running and the container is not stuck in ContainerCreating
✓kubectl describe pod <pod> -n <ns> shows the volume mounted at the expected path
✓kubectl exec <pod> -n <ns> -- mount | grep <pv> shows the volume mounted
✓kubectl exec <pod> -n <ns> -- ls -la /<mount-path> shows expected contents
✓kubectl exec <pod> -n <ns> -- touch /<mount-path>/write-test && rm /<mount-path>/write-test succeeds
✓CSI controller and node plugin Pods are Running and Ready
✓No new Warning events in the last 5 minutes

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶If the mount fix required detaching and re-attaching the volume, the application should reconnect automatically; verify by reading the application log
↶If the fix involved changing fsGroup, the Pod must be deleted and recreated (the kubelet sets fsGroup only at first mount)
↶If the CSI driver upgrade caused the failure, roll back the CSI driver DaemonSet to the previous version (with explicit approval)
↶Do not delete the PV unless you are sure it is detached from every node; a deleted PV with the volume still attached can leave an orphaned device on the node
↶Capture the failing kubelet and CSI driver logs to off-cluster storage before any rollback that may lose them

6 · Escalation

When the runbook isn't enough, contact:

· Volume attach fails and the storage backend does not show the volume: the volume was deleted in the backend; escalate to storage ownership — this is a data-loss-risk situation
· CSI node plugin CrashLoopBackOff on the affected node: the driver cannot communicate with the storage backend; escalate to platform/storage ownership
· Volume is attached but cannot be mounted: filesystem corruption on the volume; escalate to storage ownership before any operation that might write to the volume
· Multi-attach error (RWX volume attached to two nodes with conflicting access modes): the storage backend cannot serve the volume to both; detach from one node and let the scheduler relocate
· CSI driver version skew with the cluster minor version: the driver does not support the cluster CSI version; escalate to platform and align versions

A mount failure can lose data if the operator formats the wrong device, detaches the wrong volume, or deletes a PV that is still attached. The runbook is conservative: never delete a PV; never format a device unless the volume is documented as new; never detach without confirming no Pod is using it.

1. Identify the failing stage

Stage	Symptom in Pod events	Where to look
Attach	`FailedAttachVolume: ...`	CSI controller logs, storage backend
Mount	`FailedMount: ... mount failed`	kubelet journal, CSI node plugin logs
Format	`Volume is not formatted` or `format failed`	CSI node plugin logs
fsGroup	Pod stuck in `ContainerCreating`	kubelet journal, security context

Read-only / SafeIdentify the failing stage


kubectl describe pod <pod> -n <ns> | sed -n '/Events:/,$p' | head -40

NODE=$(kubectl get pod <pod> -n <ns> -o jsonpath='{.spec.nodeName}')
ssh "$NODE" -- sudo journalctl -u kubelet --since "10 min ago" | grep -iE 'mount|attach|csi' | tail -40

2. Attach stage

Read-only / SafeAttach stage


kubectl get volumeattachments -o wide | grep <pv>
kubectl get volumeattachment <va-name> -o yaml

# On the node
NODE=$(kubectl get pod <pod> -n <ns> -o jsonpath='{.spec.nodeName}')
ssh "$NODE" -- lsblk
ssh "$NODE" -- sudo dmesg | tail -20

# CSI controller logs
kubectl -n <csi-namespace> logs deploy/<csi-driver>-controller --tail=200 | grep -iE 'attach|error' | tail

If the device is not visible on the node, the attach failed. Common causes:

CSI controller cannot reach the storage backend
Storage backend does not have the volume (volume was deleted)
The volume is in use elsewhere and the backend blocks the attach

3. Mount stage

Read-only / SafeMount stage


# Kubelet journal around the mount syscall
NODE=$(kubectl get pod <pod> -n <ns> -o jsonpath='{.spec.nodeName}')
ssh "$NODE" -- sudo journalctl -u kubelet --since "10 min ago" | grep -iE 'mount|csi' | tail -30

# CSI node plugin on the node
kubectl -n <csi-namespace> get pods -o wide -l app=<csi-driver>-node | grep "$NODE"
kubectl -n <csi-namespace> logs ds/<csi-driver>-node --tail=200 | tail -50

# Is the device mounted on the node?
ssh "$NODE" -- mount | grep -E '<pv|ext4|xfs'

If the CSI node plugin Pod is CrashLoopBackOff on the node, restart it. If the device is not mounted but the attach succeeded, the mount plugin cannot find the device path.

4. fsGroup and permissions

Read-only / SafefsGroup and permissions


# Pod's requested fsGroup
kubectl get pod <pod> -n <ns> -o jsonpath='{.spec.securityContext.fsGroup}'
# Volume's permissions (after mount)
kubectl exec <pod> -n <ns> -- stat -c '%U:%G %a' /<mount-path>

The CSI driver sets fsGroup at first mount. If the volume was mounted by a Pod with a different fsGroup, the permissions may not match the current Pod’s expectations. Recreating the Pod (kubectl delete pod) forces kubelet to re-apply fsGroup.

5. Read-only mount

Read-only / SafeRead-only mount


kubectl describe pod <pod> -n <ns> | grep -E 'ReadOnly|RW'
kubectl exec <pod> -n <ns> -- touch /<mount-path>/test 2>&1 || true
kubectl get pv <pv-name> -o jsonpath='{.spec.accessModes}'

A volume may be mounted read-only if:

The PVC’s accessModes includes only ReadOnlyMany
The PV was mounted once and the storage backend marked it read-only after a failed write
The CSI driver is degraded

6. Apply the smallest fix

Read-only / SafeApply the smallest fix


# A. Restart the CSI node plugin on the affected node
NODE=$(kubectl get pod <pod> -n <ns> -o jsonpath='{.spec.nodeName}')
kubectl -n <csi-namespace> delete pod -l app=<csi-driver>-node --field-selector spec.nodeName="$NODE"
kubectl -n <csi-namespace> rollout status ds/<csi-driver>-node --timeout=5m

# B. Recreate the Pod to re-trigger attach/mount
kubectl delete pod <pod> -n <ns> --force --grace-period=0
kubectl wait --for=condition=Ready pod -l app=<name> -n <ns> --timeout=120s

# C. Cordon the node if it cannot recover (avoid scheduling new Pods)
kubectl cordon "$NODE"

# D. Re-attach the volume manually (advanced)
kubectl get volumeattachment -o yaml | head
kubectl delete volumeattachment <va-name>  # do this only if the device is detached on the node
# The controller will re-create it

7. Verify

Read-only / SafeVerify


kubectl describe pod <pod> -n <ns> | grep -A2 'Mounts:'
kubectl exec <pod> -n <ns> -- ls -la /<mount-path>
kubectl exec <pod> -n <ns> -- sh -c 'touch /<mount-path>/.write-test && rm /<mount-path>/.write-test && echo writable'

Common pitfalls

Symptom	Cause	Action
Attach times out repeatedly	CSI controller cannot reach storage backend	Check controller logs; check network/firewall
Mount fails with `wrong fs type`	Volume was formatted on a different filesystem	Do not reformat; escalate to storage ownership
Pod stuck in `ContainerCreating` after fsGroup change	fsGroup is applied only at first mount	Recreate the Pod
Mount fails only after node reboot	CSI node plugin not running after reboot	Restart the CSI node plugin
Read-only volume that should be RW	Storage backend has marked the volume RO	Investigate backend; do not chmod the mount

A mount failure is a storage failure, not a Pod failure. The fix is in the CSI driver, the storage backend, or the volume configuration; the Pod is the symptom.