CSI mount failure

Reported symptoms

Four things are wrong this morning and none of them look related.

• Platform. postgres-2, rescheduled last night after a routine node drain, has been in ContainerCreating for twenty-three minutes.
• Data. The nightly backup CronJob has produced no completed Pod for two nights. Nobody noticed until the report failed to arrive.
• Search. A Deployment rollout is stuck at 3 of 5. The three Pods that started are on node-02, node-08 and node-14.
• Storage. Asked to look at the backend, the storage team reports every volume attached, healthy, and serving IO to the workloads that already have it.

Two engineers have independently concluded that particular nodes are flaky, because deleting a stuck Pod fixes it about half the time. One of them has started draining node-19.

Meanwhile the CSI driver is, by every check the platform runs, in perfect health: forty node plugin Pods Running and Ready, the controller plugin healthy, and PVCs binding within a second of being created.

The questions worth asking at this point:

• If the volumes are attached, which part of the storage path is left?
• Why does deleting a Pod fix it sometimes?
• What happened at 06:40 this morning?

Evidence provided

$ kubectl describe pod postgres-2 -n platform | sed -n '/Events/,$p'

Events:
Type     Reason       Age                    From               Message
----     ------       ----                   ----               -------
Normal   Scheduled    24m                    default-scheduler  Successfully assigned platform/postgres-2 to node-19
Warning  FailedMount  2m17s (x9 over 22m)    kubelet            Unable to attach or mount volumes: unmounted volumes=[data], unattached volumes=[], timed out waiting for the condition

$ kubectl get volumeattachment -o custom-columns=NAME:.metadata.name,PV:.spec.source.persistentVolumeName,NODE:.spec.nodeName,ATTACHED:.status.attached | grep node-19

csi-3f81a4c9  pvc-a17c5b2e-9d44-4c1a-b6f2-71e0c8d3a952  node-19  true

That single line eliminates a great deal. The PVC is Bound, so provisioning worked. The VolumeAttachment is attached, so ControllerPublishVolume worked and the device is present on the node. Everything left is on the node, between the kubelet and the node plugin.

$ kubectl get pods -n kube-system -l app=ebs-csi-node -o wide --sort-by=.metadata.creationTimestamp | awk 'NR==1 || NR%6==2'

NAME                  READY   STATUS    RESTARTS   AGE   NODE
ebs-csi-node-x4k2p    3/3     Running   0          26d   node-02
ebs-csi-node-r7wqm    3/3     Running   0          26d   node-14
ebs-csi-node-9tzvb    3/3     Running   0          3h1m  node-19
ebs-csi-node-c5mdh    3/3     Running   0          2h48m node-27
ebs-csi-node-k8plf    3/3     Running   0          2h11m node-33
ebs-csi-node-j2xrn    3/3     Running   0          97m   node-41

$ kubectl rollout history daemonset/ebs-csi-node -n kube-system

daemonset.apps/ebs-csi-node
REVISION  CHANGE-CAUSE
1         <none>
2         driver upgrade v1.30.0 -> v1.31.0

$ kubectl exec -n kube-system ebs-csi-node-9tzvb -c ebs-plugin -- grep -c 'ebs.csi.aws.com' /proc/mounts

3

$ kubectl debug node/node-19 -it --image=busybox -- grep -c 'ebs.csi.aws.com' /host/proc/1/mountinfo

0

Work the evidence before reading on

Nothing here has failed. Every component reports success, and every one of those reports is honest.

1. Line up the list of stuck workloads against the list of node plugin Pod ages. The correlation is the diagnosis; write it down before you form an opinion about nodes.
2. The volume is mounted inside the plugin container and not on the host. Which of the two processes that need to see that mount is inside the container?
3. Deleting a stuck Pod helps about half the time. Given the rollout is still in progress, what determines which half?

Before continuing: the engineer draining node-19 is trying to move those workloads somewhere healthy. What does the drain actually do to this incident?

Root cause

1. The plugin mounted the volume into its own namespace

A container has its own mount namespace. That is the point of a container.

The CSI node plugin therefore has a problem no other workload has: the mounts it creates have to be visible to somebody else — to the kubelet, which asked for them, and to the container runtime, which has to bind the path into the workload container. Both are on the host.

The mechanism that solves this is mountPropagation: Bidirectional on the volumeMount for the host /var/lib/kubelet directory. It is the setting that lets a mount made inside the container travel outward to the host.

Revision 2 of the DaemonSet does not have it:

        volumeMounts:
        - name: kubelet-dir
          mountPath: /var/lib/kubelet
          # mountPropagation: Bidirectional   <- present in revision 1

Absent, the field defaults to None, which means private. So NodeStageVolume formats and mounts the device, NodePublishVolume bind-mounts it to the target path, both return success to the kubelet, and all of it happens in a namespace nobody else can see. The plugin is not broken. It is working perfectly, in a room with the door shut.

2. The kubelet has no vocabulary for this

The kubelet asked for a mount and got a success. It then looks at the target path, finds nothing mounted, and does the only thing it can: waits, retries, and eventually reports timed out waiting for the condition.

That message is accurate and useless. It names no component, carries no backend error, and reads exactly like a slow storage array. It is the reason three teams filed three tickets against three different subsystems.

3. A DaemonSet rolls one node at a time, which disguises the change

ebs-csi-node updates with a rolling strategy, so at 09:15 the fleet is split: nodes the DaemonSet has reached are broken for new mounts, nodes it has not reached are fine, and the boundary moves during the incident.

Every confusing observation follows from that split.

• Deleting a stuck Pod works when the scheduler happens to place it on a node the rollout has not reached yet, and does nothing when it does not. That is the “half the time”.
• The three Search Pods that started are on node-02, node-08 and node-14, which are simply three of the nodes still on revision 1.
• Workloads already running are unaffected, because their mounts were made before the roll and live on the host where they always did.

None of this is a property of any node. It is a property of the rollout order, and it is why “these nodes are flaky” survived as an explanation for three hours.

Resolution

1. Stop the drain. It is converting working nodes into broken ones, and it is the most urgent thing in the incident.
2. Write the correlation down: stuck workloads against node plugin Pod age. One table, two columns, and the three tickets become one.
3. Roll the DaemonSet back with kubectl rollout undo daemonset/ebs-csi-node -n kube-system. Revision 1 carries the propagation setting, and the mechanism that spread the fault heals it in the same order.
4. Cordon the nodes still on revision 2 while the rollback works through them, and uncordon each one as its plugin Pod returns to revision 1. This bounds the incident instead of chasing it.
5. Leave the stuck Pods alone. The kubelet retries the mount on its own backoff and will succeed once the plugin under it is correct; a Pod you delete tells you nothing about whether the node recovered.
6. Verify one formerly broken node properly before declaring the fleet fixed: the mount has to appear in the host namespace, not merely in the plugin container.
7. Fix the upstream manifest afterwards, not during. The upgrade stays held at revision 1 until somebody has read the volumeMounts stanza of revision 2 and explained how the line was lost.
8. Check what the two missed backup nights cost before closing. The CronJob failed silently and that is a second finding, not a footnote.

Verification

1. The DaemonSet reports desired, current, ready and up-to-date all equal to the node count, on revision 1.
2. On a node that was in the broken set, the CSI mount for a test volume appears in the host mount namespace. This is the check that can fail and it is the one that matters.
3. The Pods that were stuck reached Running without being deleted. Note which ones, because any Pod that was deleted during the incident has to be re-tested.
4. A short-lived Pod with a fresh PVC mounts and writes on each node that was broken, tested one node at a time. A random node proves nothing when the fault followed a rollout order.
5. The backup CronJob produced a completed Pod on its next scheduled run, and the resulting artefact is the right size.
6. No VolumeAttachment was force-removed and no StatefulSet Pod was force-deleted during the incident. If either happened, the affected volume needs a filesystem check before it is trusted.
7. The drained node is back in service and carrying workloads again.

Prevention

• Make the upgrade gate mount something. Every check that ran during this upgrade asked whether the plugin Pods were Ready. They were Ready on all forty nodes, throughout, while serving no mounts at all. A post-upgrade gate that schedules a short-lived Pod with a fresh PVC onto every node is the only test that exercises the path that broke.
• Read the volumeMounts stanza on every driver upgrade. Diff the rendered manifest against the running one rather than trusting the chart version. The field that disappeared is one line in a block that otherwise looked identical.
• Enforce it as policy. A CSI node plugin whose /var/lib/kubelet mount is not Bidirectional is misconfigured in every case; there is no tuning argument on the other side. That makes it a good candidate for an admission policy or a CI check rather than a review convention.
• Alert on ContainerCreating, grouped by node. A Pod stuck in ContainerCreating past a few minutes is worth an alert on its own; the grouping by node is what turns four tickets into one diagnosis, and it is free.
• Alert on a CronJob that produces no completed Pod. Two nights of missing backups went unnoticed because the job never ran, and a monitor that watches job failures does not see a job that failed to start.
• Retire “that node is flaky”. It is a hypothesis with an obvious test — does the fault follow the node or the workload — and it was accepted here for three hours without anyone running it.