Diagnosing a volume that will not mount

Why this matters in production

A pod stuck ContainerCreating with a volume error has a small set of possible causes, and the event message narrows it immediately.

Reading the event

NAME=acme
kubectl describe pod ${NAME} | sed -n '/Events:/,$p'

The message names the phase and usually the component.

Following the failure

# Substitute your own value before running:
POD=web-5f9c7d8b6c-2xk9p

# which node?
kubectl get pod "$POD" -o jsonpath='{.spec.nodeName}'

# the node plugin on that node
NODE=$(kubectl get pod "$POD" -o jsonpath='{.spec.nodeName}')
kubectl -n ceph-csi get pods -o wide --field-selector spec.nodeName="$NODE"

# its logs -- PLUGIN_POD is the csi-rbdplugin pod from the listing above
PLUGIN_POD=csi-rbdplugin-4xk7z
kubectl -n ceph-csi logs "$PLUGIN_POD" -c csi-rbdplugin --tail=100

# and on the node itself
dmesg -T | grep -i rbd | tail -20

The kernel log on the node carries the krbd errors, which the CSI logs do not always surface clearly.

The common causes

# 1. can the node reach Ceph?
PLUGIN_POD=plugin_pod
IMAGE=vm-disk-01
PV=pv
USERID=userid
kubectl -n ceph-csi exec ${PLUGIN_POD} -- ceph -s --connect-timeout 5

# 2. does the image exist?
rbd -p k8s-rbd ls | grep ${IMAGE}

# 3. are the features supported?
rbd info k8s-rbd/${IMAGE} | grep features
uname -r

# 4. is it attached elsewhere?
rbd status k8s-rbd/${IMAGE}
kubectl get volumeattachment | grep ${PV}

# 5. is the secret correct?
kubectl -n ceph-csi get secret csi-rbd-secret -o jsonpath='{.data.userID}' | base64 -d
ceph auth get client.${USERID}

Five checks covering the substantial majority of mount failures.

The stale attachment case

IMAGE=vm-disk-01
rbd status k8s-rbd/${IMAGE}
# Watchers:
#   watcher=10.0.1.42:0/1234567 client.4567 cookie=1

A node that failed without cleanly unmapping leaves a watcher, and another node cannot acquire the exclusive lock:

# after confirming the old node is genuinely gone
IMAGE=vm-disk-01
OLD_NODE_ADDR=10.20.0.11
NAME=acme
rbd status k8s-rbd/${IMAGE}
ceph osd blocklist add ${OLD_NODE_ADDR}
kubectl delete volumeattachment ${NAME}

Blocklisting the stale client is what releases the lock, and it must only be done after confirming the node is not still writing.

Restoring the pod

# after fixing the cause
NAME=acme
kubectl delete pod ${NAME}

# Substitute your own value before running:
POD=web-5f9c7d8b6c-2xk9p

# verify
kubectl get pod "$POD" -w
kubectl exec "$POD" -- df -h /data

Quiz

Production discipline

Read the node’s dmesg alongside the CSI plugin logs — krbd errors land in the kernel log and are often the only place the actual reason appears. Confirm a node is genuinely down before blocklisting its client to release a lock; doing so to a live node severs it mid-write.

Cross-course references

• Kubernetes: stale volume attachments after node failure are a general CSI problem
• Linux: a lock held by a crashed process needs the same careful release