Storage incident runbook — the complete diagnostic and recovery

A storage incident is one of: PVC Pending, attach failure, mount failure, topology mismatch, expansion failure, or snapshot failure. This lesson provides a complete runbook for the diagnostic, the recovery, and the prevention.

The diagnostic ladder

The diagnostic is the same for every storage incident; the cause is per-incident:

flowchart TD
    A[Storage incident] --> B[kubectl describe pvc]
    B --> C{kubectl describe pod}
    C --> D{journalctl -u kubelet}
    D --> E{CSI plugin logs}
    E --> F{Backend metrics}
    F --> G{Identify the cause}

Each layer reveals a different cause:

• kubectl describe pvc: events for Pending PVCs.
• kubectl describe pod: events for FailedMount, FailedAttach.
• journalctl -u kubelet: kubelet’s view of the mount.
• CSI plugin logs: the gRPC calls and the backend responses.
• Backend metrics: AWS CloudWatch, GCP Monitoring, etc.

Runbook: PVC Pending

## PVC Pending

### Symptoms
- PVC in Pending state
- Pod that mounts the PVC is also Pending (waiting for
  volume)

### First 5 minutes
1. `kubectl describe pvc <pvc-name>` — check events
2. `kubectl describe pod <pod-name>` — check Pod events
3. Identify the cause from the event message:
   - "no storage class is set" — PVC has no
     storageClassName; default StorageClass is missing
   - "storageclass not found" — StorageClass does not
     exist
   - "failed to provision volume" — provisioner error
   - "exceeded quota" — backend quota exceeded
   - "no topology constraints" — topology mismatch

### Diagnostic
1. If "no storage class is set": configure a default
   StorageClass, or specify storageClassName in the PVC
2. If "storageclass not found": create the StorageClass
3. If "failed to provision": check CSI controller logs
4. If "exceeded quota": reduce request or request quota
5. If "no topology constraints": verify
   allowedTopologies matches the cluster's zones

### Recovery
- Create the missing StorageClass
- Fix the provisioner (restart, fix IAM, fix parameters)
- Reduce the request or request quota increase
- Update allowedTopologies

### Validation
- The PVC transitions from Pending to Bound
- The Pod's events show "Successfully assigned" and the
  Pod starts

Runbook: FailedAttach

## Volume Attach Failure

### Symptoms
- Pod in ContainerCreating with FailedAttach events
- Volume not visible on the node (ls /dev/xvd* shows no
  new device)

### First 5 minutes
1. `kubectl describe pod <pod-name>` — check events
2. `journalctl -u kubelet -n <pod-uid>` — check kubelet
   logs
3. `kubectl -n kube-system logs <csi-controller-pod>` —
   check controller logs
4. Verify the volume is attached to the instance: AWS
   console, `aws ec2 describe-volumes`

### Diagnostic
1. If "max volume count per instance": instance has hit
   its EBS limit
2. If AZ mismatch: volume is in a different AZ than the
   instance
3. If "permission denied": IAM permissions are missing
4. If "InvalidInstanceType": instance type does not
   support the volume type
5. If multipath issues: device appears at the wrong path

### Recovery
- Use a larger instance type or reduce PVC count per
  node
- Set `volumeBindingMode: WaitForFirstConsumer`
- Fix IAM permissions
- Use a different instance type
- Configure multipath correctly

### Validation
- The volume is visible on the node (ls /dev/xvd*)
- The Pod's events show successful attach
- The Pod starts and reads/writes to the volume

Runbook: FailedMount

## Volume Mount Failure

### Symptoms
- Pod in ContainerCreating with FailedMount events
- Volume is attached but Pod cannot mount

### First 5 minutes
1. `kubectl describe pod <pod-name>` — read the
   FailedMount message and the volume handle
2. `journalctl -u kubelet -n <pod-uid>` — check the mount
   syscall and any fsck output
3. `kubectl -n kube-system logs <csi-node-pod>` — check
   node plugin logs
4. `dmesg | tail -20` on the node — check kernel logs

### Diagnostic
1. If "wrong fs type": StorageClass's fsType does not
   match the device
2. If "failed to get secret": Secret is missing or has
   wrong content
3. If "permission denied": CSI node plugin is not
   privileged
4. If "device busy": stale mount; unmount first
5. If "mkfs failed": format error; investigate

### Recovery
- Update the StorageClass's fsType
- Create the missing Secret with the right content
- Ensure CSI node plugin has `privileged: true`
- Unmount the stale mount
- Investigate the format error (CSI driver issue)

### Validation
- The volume mounts successfully
- The Pod starts and reads/writes to the volume
- df -h shows the expected capacity

Runbook: Topology Pending

## Topology Pending

### Symptoms
- PVC Pending with topology-related events
- Pod cannot be scheduled

### First 5 minutes
1. `kubectl describe pvc <pvc-name>` — read the topology
   constraint named in the events
2. `kubectl get nodes --show-labels | grep topology` —
   verify cluster's topology labels
3. `kubectl get storageclass <name> -o yaml` — verify
   allowedTopologies

### Diagnostic
1. If no allowedTopologies: add them to the StorageClass
2. If AZ mismatch: add the missing zone to
   allowedTopologies
3. If regional mismatch: align the StorageClass's region
4. If node labels missing: add the standard topology
   labels

### Recovery
- Update the StorageClass with allowedTopologies
- Match the cluster's zones and regions
- Apply the change

### Validation
- The PVC binds
- The Pod schedules to a node in the matching zone
- The volume is in the same AZ as the Pod

Runbook: Expansion Failure

## Volume Expansion Failure

### Symptoms
- PVC capacity edit is rejected or fails
- Filesystem does not grow to the new size

### First 5 minutes
1. `kubectl describe pvc <pvc-name>` — compare
   `status.capacity` against `spec.resources.requests`
2. `journalctl -u kubelet -n <pod-uid>` — check the
   kubelet's online filesystem resize
3. `kubectl -n kube-system logs <csi-controller-pod>` —
   check controller logs

### Diagnostic
1. If edit is rejected: StorageClass does not have
   allowVolumeExpansion: true
2. If Unimplemented: CSI driver does not support
   ControllerExpandVolume
3. If quota: backend quota exceeded
4. If "fs resize failed": filesystem cannot resize online

### Recovery
- Set `allowVolumeExpansion: true` on the StorageClass
- Update the CSI driver
- Reduce the request or request quota
- Stop the workload (offline expansion)

### Validation
- The PVC status shows the new capacity
- df -h inside the Pod shows the new size
- The application sees the new capacity

The prevention checklist

## Storage Incident Prevention

### Cluster bootstrap
- [ ] StorageClasses per workload class (db, web, batch)
- [ ] `WaitForFirstConsumer` for multi-AZ clusters
- [ ] `allowedTopologies` matches the cluster's zones
- [ ] `allowVolumeExpansion: true` on production SCs
- [ ] `Retain` for stateful data; `Delete` for ephemeral
- [ ] CSI driver deployed with 2+ replicas, leader election
- [ ] CSIDriver object registered

### Monitoring
- [ ] Alert on Pending PVC for > 5 minutes
- [ ] Alert on FailedMount/FailedAttach
- [ ] Alert on PVC capacity > 80%
- [ ] Alert on Released PVs
- [ ] Dashboard with PVC state, PV state, volume capacity

### Runbook
- [ ] Storage incident runbook (this lesson)
- [ ] PVC deletion RBAC (production requires approval)
- [ ] Snapshot schedule for stateful workloads
- [ ] Snapshot restore procedure tested

### Validation
- [ ] Test PVC creation on every StorageClass
- [ ] Test mount on every node
- [ ] Test expansion on a staging cluster
- [ ] Test snapshot creation and restore

Quiz

Production discipline

• The runbook is the on-call reference. Without it, the SRE re-derives the fix at 3 AM.
• The diagnostic ladder is standard. Every storage incident starts with kubectl describe pvc.
• The recovery is per-cause. Each failure mode has its own fix; the runbook has the fix for each.
• The prevention checklist prevents recurrence. A storage incident that recurs is a process failure, not a tool failure.
• Test the runbook with a simulated incident. A runbook that is never tested is not a runbook.