Volume expansion — growing PVCs online and the capacity discipline

Volume expansion is the operation of growing a PVC’s capacity. The PVC’s spec.resources.requests.storage is edited; the CSI driver expands the underlying volume; the filesystem is resized. This lesson walks the requirements, the modes, and the production discipline.

How expansion works

sequenceDiagram
    participant U as User
    participant API as API server
    participant CSI as CSI driver
    participant N as Node (kubelet)
    U->>API: edit PVC: requests.storage 100Gi -> 200Gi
    API->>CSI: ExpandVolume (volumeHandle, newSize)
    CSI-->>API: volume expanded
    API->>N: resize filesystem on the node
    N-->>API: filesystem resized
    API-->>U: PVC status: capacity 200Gi

The expansion has two phases:

1. Backend expansion: the CSI driver calls the backend to grow the underlying volume.
2. Filesystem resize: the kubelet resizes the filesystem on the node (if the filesystem supports online resize).

Requirements

For expansion to work:

1. StorageClass: allowVolumeExpansion: true.
2. CSI driver: the driver must support ExpandVolume.
3. Filesystem: must support online or offline resize (ext4, xfs support online resize).

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: db-ssd
provisioner: ebs.csi.aws.com
allowVolumeExpansion: true    # <-- required
parameters:
  type: io2

Online vs offline expansion

Online expansion is the standard for modern CSI drivers (EBS CSI, GCE PD CSI, Ceph RBD CSI, Longhorn CSI). The Pod continues to run; the filesystem is resized in place.

Offline expansion requires:

1. The Pod is stopped (PVC unmounted).
2. The volume is expanded.
3. The Pod is restarted; the filesystem is resized on mount.

For databases, offline expansion is a maintenance window. For stateless workloads, online expansion is invisible.

The capacity discipline

The production discipline for capacity:

• Monitor PVC usage. A PVC that is > 80% full should trigger an expansion alert.
• Expand before the PVC is full. A database that fills the volume can have data corruption; the expansion should be proactive.
• Test the expansion procedure. A volume expansion that fails can leave the workload in a degraded state. Test on a staging cluster first.
• Document the expansion in the runbook. The procedure, the validation, the rollback.

# Substitute your own value before running:
POD=app-0                 # the Pod that mounts the `data` PVC

# Check the PVC's current usage
kubectl exec "$POD" -- df -h /var/lib/app
# Filesystem      Size  Used Avail Use% Mounted on
# /dev/xvdba       99G   85G   14G  86% /var/lib/app

# Expand the PVC
kubectl edit pvc data
# Change spec.resources.requests.storage from 100Gi to 200Gi

# Verify the expansion
kubectl get pvc data -o jsonpath='{.status.capacity.storage}'
# 200Gi

kubectl exec "$POD" -- df -h /var/lib/app
# Filesystem      Size  Used Avail Use% Mounted on
# /dev/xvdba      198G   85G  113G  43% /var/lib/app

The expansion failure modes

Common expansion failure modes:

• StorageClass does not have allowVolumeExpansion: true: the API server rejects the edit.
• CSI driver does not support ExpandVolume: the API server accepts the edit; the CSI driver returns an error; the PVC status shows the error.
• Filesystem does not support online resize: the CSI driver succeeds; the kubelet fails to resize the filesystem; the Pod’s mount shows the old size.
• Backend quota exceeded: the CSI driver fails (e.g., AWS EBS volume size limit).

# Check the expansion status
kubectl describe pvc data
# Events:
#   Normal   ExternalExpanding   ...   resize volume requested
#   Normal   Resizing            ...   filesystem resize completed
#   Warning  VolumeResizeFailed  ...   failed to resize filesystem

Quiz

Production discipline

• allowVolumeExpansion: true on production StorageClasses. Databases grow; the StorageClass must allow it.
• Monitor PVC usage. > 80% full triggers an alert; expand before the workload fails.
• Test the expansion on a staging cluster. A failed expansion can leave the workload in a degraded state.
• Snapshot before expansion. There is no rollback path for a growth operation; the snapshot is the safety net.
• Verify the application sees the new capacity. Some applications cache the size at startup and need a refresh.