CSI driver crash and force-detach — the storage runtime failures

A CSI driver is two separate things, and which half has failed decides what you see. With the controller plugin down, new PVCs sit Pending and no volume attaches anywhere in the cluster; with the node plugin down on one node, volumes attach normally but every Pod scheduled there stops in ContainerCreating. Pods whose volumes are already mounted keep serving throughout either failure, which is why a broken CSI driver usually goes unnoticed until the next rollout tries to move a workload.

The CSI architecture

The CSI driver is the cluster’s storage runtime. The CSI driver is composed of:

• Controller plugin. A Deployment or StatefulSet that handles CreateVolume, DeleteVolume, Attach, Detach.
• Node plugin. A DaemonSet that handles mount, unmount.

flowchart LR
    A[API server] --> B[external-provisioner]
    A --> C[external-attacher]
    B --> D[CSI controller plugin]
    C --> D
    D --> E[Storage backend]
    F[CSI node plugin] --> G[Mount the volume]

The CSI driver is the cluster’s storage bridge.

The diagnostic

The canonical diagnostic:

# Substitute your own values before running:
CSI=ebs-csi
NS=production
PVC=postgres-data-postgres-0

# 1. Check the controller plugin
kubectl get pods -n kube-system -l "app=${CSI}-controller"

# 2. Check the node plugin
kubectl get pods -n kube-system -l "app=${CSI}-node"

# 3. Check the CSI driver logs
kubectl logs -n kube-system -l "app=${CSI}-controller" --tail=200
kubectl logs -n kube-system -l "app=${CSI}-node" --tail=200

# 4. Check the storage backend
# (backend-specific)

# 5. Check the events
kubectl get events -n "$NS" --field-selector involvedObject.name="$PVC"

The diagnostic is the CSI driver, the storage backend, and the events.

Common failures

• Controller plugin crashing. The controller plugin is in CrashLoopBackOff. The remediation is to restart the controller plugin.
• Node plugin crashing. The node plugin is in CrashLoopBackOff. The remediation is to restart the node plugin.
• Storage backend unreachable. The storage backend is unreachable. The remediation is to fix the network.
• Force-detach. The volume is stuck in detach; the remediation is to force-detach.

flowchart TD
    A[CSI failure] --> B{Controller plugin crashing?}
    B -->|Yes| C[Restart the controller plugin]
    B -->|No| D{Node plugin crashing?}
    D -->|Yes| E[Restart the node plugin]
    D---|No| F{Backend unreachable?}
    F -->|Yes| G[Fix the network]
    F -->|No| H{Force-detach?}
    H -->|Yes| I[Force-detach the volume]
    H -->|No| J[Unknown]

The force-detach

A force-detach is the cluster’s escape hatch for a stuck volume. The force-detach is the action that allows the volume to be detached even if the original attacher is not present.

# Substitute your own values before running:
NS=production
PVC=postgres-data-postgres-0

# Force-detach a volume
kubectl edit pvc "$PVC" -n "$NS"
# Add the annotation: "volume.kubernetes.io/force-detach: true"

The force-detach is the cluster’s last resort.

The remediation

The remediation depends on the cause:

# Substitute your own values before running:
CSI=ebs-csi
NS=production
PVC=postgres-data-postgres-0

# Option 1: Restart the controller plugin
kubectl rollout restart "deployment/${CSI}-controller" -n kube-system

# Option 2: Restart the node plugin
kubectl rollout restart "daemonset/${CSI}-node" -n kube-system

# Option 3: Fix the network
# (network-specific)

# Option 4: Force-detach the volume
kubectl edit pvc "$PVC" -n "$NS"

The remediation is the storage recovery.

Production discipline

A CSI driver failure is the cluster’s hypothesis. The discipline is to walk the 11-step methodology applied to the storage layer, identify the cause, apply the remediation. The storage is the cluster’s data; the remediation is the storage recovery.

• Check the CSI driver. The CSI driver is the cluster’s storage runtime.
• Check the controller plugin. The controller plugin is the cluster’s volume manager.
• Check the node plugin. The node plugin is the cluster’s mount manager.

Quiz