Volume expansion and quota — the storage growth

A database running out of disk is a deadline, and editing the PVC’s spec.resources.requests.storage is only the request — the volume grows when the CSI driver and the filesystem agree to grow it. It fails silently in four ways: a StorageClass with allowVolumeExpansion set to false, a driver that does not implement expansion, a filesystem that needs the Pod restarted before it will resize, and a namespace quota with no room for the increase. Knowing which one you have matters under time pressure, because the request cannot be withdrawn: a PVC can be expanded but never shrunk.

The volume expansion

The volume expansion is the workload’s request for more storage. The expansion is performed by:

1. The user updating the PVC’s spec.resources.requests.storage.
2. The CSI driver resizing the volume.
3. The kubelet remounting the filesystem with the new size.

flowchart TD
    A[User updates PVC] --> B[CSI driver resizes volume]
    B --> C[Kubelet remounts filesystem]

The expansion is the cluster’s storage growth.

The diagnostic

The canonical diagnostic:

# Substitute your own values before running:
NS=production
PVC=postgres-data-postgres-0
SC=gp3-encrypted
CSI_APP=ebs-csi-controller

# 1. Check the PVC
kubectl get pvc -n "$NS"
kubectl describe pvc "$PVC" -n "$NS"

# 2. Check the PV
kubectl get pv -o yaml | grep -A5 capacity

# 3. Check the StorageClass
kubectl get storageclass
kubectl describe storageclass "$SC"

# 4. Check the CSI driver
kubectl logs -n kube-system -l app="$CSI_APP" --tail=200

# 5. Check the events
kubectl get events -n "$NS" --field-selector involvedObject.name="$PVC"

The diagnostic is the PVC, the PV, the StorageClass, and the CSI driver.

Common failures

• StorageClass does not allow expansion. The StorageClass’s allowVolumeExpansion is false. The remediation is to update the StorageClass.
• CSI driver does not support expansion. The CSI driver does not support expansion. The remediation is to replace the CSI driver.
• Pod is using the volume. The volume is in use and the expansion is offline. The remediation is to restart the Pod.
• Quota exceeded. The namespace’s ResourceQuota is exceeded. The remediation is to increase the quota.

flowchart TD
    A[Expansion failing] --> B{Allow expansion?}
    B -->|No| C[Update the StorageClass]
    B -->|Yes| D{CSI driver supports?}
    D---|No| E[Replace the CSI driver]
    D---|Yes| F{Pod using volume?}
    F -->|Yes| G[Restart the Pod]
    F -->|No| H{Quota exceeded?}
    H -->|Yes| I[Increase the quota]
    H -->|No| J[Unknown]

The online vs offline expansion

The online expansion is performed while the Pod is running. The offline expansion requires the Pod to be restarted.

# Substitute your own values before running:
NS=production
PVC=postgres-data-postgres-0
POD=postgres-0

# Online expansion
kubectl edit pvc "$PVC" -n "$NS"
# Update spec.resources.requests.storage

# Offline expansion (for filesystems that require it)
kubectl delete pod "$POD" -n "$NS"
kubectl edit pvc "$PVC" -n "$NS"
# Update spec.resources.requests.storage

The online expansion is preferred.

The remediation

The remediation depends on the cause:

# Substitute your own values before running:
NS=production
SC=gp3-encrypted
POD=postgres-0
QUOTA=storage-quota

# Option 1: Update the StorageClass
kubectl patch storageclass "$SC" -p '{"allowVolumeExpansion":true}'

# Option 2: Restart the Pod
kubectl delete pod "$POD" -n "$NS"

# Option 3: Increase the quota
kubectl patch resourcequota "$QUOTA" -n "$NS" -p '{"spec":{"hard":{"requests.storage":"100Gi"}}}'

# Option 4: Replace the CSI driver
# (CSI-specific)

The remediation is the storage expansion recovery.

Production discipline

A volume expansion failure is the cluster’s hypothesis. The discipline is to walk the 11-step methodology applied to the storage layer, identify the cause, apply the remediation. The storage is the cluster’s data; the remediation is the storage expansion recovery.

• Check the StorageClass. The StorageClass is the workload’s storage class.
• Check the CSI driver. The CSI driver is the cluster’s storage runtime.
• Check the quota. The quota is the namespace’s resource budget.

Quiz