CSI driver failures and their blast radius

Why this matters in production

The CSI components have very different blast radii, and restarting the wrong one during an incident makes things worse.

The blast radius

The critical observation: existing mounts survive a node plugin restart. The mount is in the kernel; the plugin is not in the data path once the volume is mounted.

kubectl -n ceph-csi get pods -o wide
kubectl -n ceph-csi get deploy,ds

Recovering the provisioner

kubectl -n ceph-csi logs deploy/csi-rbdplugin-provisioner -c csi-rbdplugin --tail=100
kubectl -n ceph-csi rollout restart deploy/csi-rbdplugin-provisioner
kubectl -n ceph-csi rollout status deploy/csi-rbdplugin-provisioner

Restarting the provisioner is safe: it holds no state that matters, and in-progress operations are retried by the sidecars.

Recovering a node plugin

NODE=stor-04
NODE=${NODE}
kubectl -n ceph-csi delete pod -l app=csi-rbdplugin \
  --field-selector spec.nodeName=$NODE

Restarting a node plugin is safe for existing mounts and interrupts any mount or unmount in progress, which is retried.

# verify it re-registers
kubectl get csinode $NODE -o jsonpath='{.spec.drivers[*].name}'

When a restart is unsafe

# check for in-progress operations first
kubectl get pvc --all-namespaces | grep -v Bound
kubectl get volumeattachment | grep -v true

Most restarts are safe; checking for in-progress operations takes seconds and covers the exceptions.

Configuring for resilience

# provisioner: more than one replica with leader election
replicaCount: 3

# node plugin: appropriate tolerations so it runs everywhere
tolerations:
  - operator: Exists

# priority so it is not evicted under pressure
priorityClassName: system-node-critical

The priority class matters: a node plugin evicted under memory pressure takes every mount operation on that node with it, and it is exactly the component that should not be evicted.

kubectl -n ceph-csi get pods -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.spec.priorityClassName}{"\n"}{end}'

Quiz

Production discipline

Give the CSI node plugin system-node-critical priority — it is evicted under memory pressure otherwise, which is exactly when workloads are rescheduling and need volumes. Restart CSI components freely for diagnosis; existing mounts are unaffected because the plugin is not in the I/O path.

Cross-course references

• Kubernetes: critical system components need priority classes to survive pressure
• Linux: control-plane daemons that establish state can restart without disturbing it