Volume expansion failures — when growing a PVC does not work

Volume expansion is the operation of growing a PVC’s capacity. The operation has multiple stages; each can fail independently. This lesson walks the failure modes and the diagnostic patterns.

The expansion lifecycle

sequenceDiagram
    participant U as User
    participant API as API server
    participant CSI as CSI controller
    participant K as kubelet
    U->>API: edit PVC: requests.storage 100Gi -> 200Gi
    API->>CSI: ControllerExpandVolume (volumeHandle, newSize)
    CSI-->>API: volume expanded
    API->>K: NodeExpandVolume (volumePath, newSize)
    K->>K: resize filesystem (resize2fs / xfs_growfs)
    K-->>API: filesystem resized
    API-->>U: PVC status: capacity 200Gi

The stages:

1. API server: accepts the edit (if allowVolumeExpansion: true on the StorageClass).
2. CSI controller: calls ControllerExpandVolume to grow the underlying volume.
3. kubelet: calls NodeExpandVolume to resize the filesystem.

Each stage has its own failure modes.

Cause 1: StorageClass does not allow expansion

The most common cause:

# StorageClass without allowVolumeExpansion
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: db-default
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
# allowVolumeExpansion: true   <-- missing

The user edits the PVC:

kubectl edit pvc data
# Change spec.resources.requests.storage from 100Gi to 200Gi

The API server rejects the edit:

# The PVC's spec.resources.requests.storage is unchanged
kubectl get pvc data -o jsonpath='{.spec.resources.requests.storage}'
# 100Gi

The fix: set allowVolumeExpansion: true on the StorageClass. Note that this change applies to new PVCs; existing PVCs are unaffected. To grow an existing PVC on a StorageClass without allowVolumeExpansion, you must recreate the PVC (with a snapshot and restore).

Cause 2: CSI driver does not support expansion

The StorageClass allows expansion; the API server accepts the edit; the CSI driver returns an error:

kubectl describe pvc data
# Events:
#   Warning  VolumeResizeFailed  ... rpc error: code = Unimplemented
#                                  desc = controller expand not supported

The fix: update the CSI driver to a version that supports ControllerExpandVolume. Or use a different StorageClass with a driver that supports expansion.

Cause 3: backend quota exceeded

The CSI driver attempts to expand the underlying volume (e.g., call ec2:ModifyVolume), but the backend’s quota is exceeded:

kubectl describe pvc data
# Events:
#   Warning  VolumeResizeFailed  ... exceeded quota: storage-gigabytes

The fix: reduce the PVC’s request, or request a quota increase from the cloud provider.

Cause 4: filesystem cannot resize online

The CSI controller succeeds; the kubelet attempts to resize the filesystem; the filesystem cannot be resized online:

kubectl describe pvc data
# Events:
#   Warning  VolumeResizeFailed  ... fs resize failed: device busy

The causes:

• The filesystem is busy (a process is using it).
• The filesystem does not support online resize (some older filesystems).
• The kernel’s online resize module is not loaded.

The fix: stop the workload (offline expansion), or use a filesystem that supports online resize (ext4, xfs).

Cause 5: new size smaller than current

The user accidentally reduced the PVC’s capacity request:

kubectl edit pvc data
# Change spec.resources.requests.storage from 200Gi to 100Gi

The API server rejects the edit:

# The PVC's spec.resources.requests.storage is unchanged
kubectl get pvc data -o jsonpath='{.spec.resources.requests.storage}'
# 200Gi

The fix: ensure the new request is greater than the current size. Volume expansion cannot shrink.

The diagnostic ladder

flowchart TD
    A[Expansion fails] --> B[kubectl describe pvc]
    B --> C{Event message?}
    C -->|allowVolumeExpansion| D[Fix StorageClass]
    C -->|Unimplemented| E[Update CSI driver]
    C -->|quota| F[Reduce or request quota]
    C -->|fs resize| G[Stop workload or use online-resize fs]
    C -->|smaller| H[Increase size]

The production fix

For each cause:

Quiz

Production discipline

• allowVolumeExpansion: true on production StorageClasses. Databases grow; the StorageClass must allow it.
• Validate expansion on staging. A failed expansion can leave the workload in a degraded state.
• Snapshot before expansion. There is no rollback; the snapshot is the safety net.
• Verify the application sees the new capacity. Some applications cache the size; a refresh is required.
• Use a filesystem that supports online resize. ext4 and xfs do; older filesystems may not.