StorageClass anti-patterns — misconfigurations that cause incidents

StorageClass anti-patterns are silent until the incident: a PVC is Pending because the binding mode is wrong; a database loses data because the reclaim policy is Delete; a workload cannot expand because the StorageClass does not allow it. This lesson walks the common anti-patterns and the production discipline for avoiding them.

Anti-pattern 1: relying on the default StorageClass

A PVC without storageClassName binds to the cluster default. The default may be standard, but the workload requires db-ssd. The PVC binds to the wrong tier; the database runs on slow storage.

Fix: every production PVC specifies storageClassName explicitly. The default is for development only.

Anti-pattern 2: Immediate binding in multi-AZ

# WRONG (multi-AZ cluster)
volumeBindingMode: Immediate

The PV is created in an arbitrary AZ. Pods scheduled to a different AZ cannot mount the volume. The Pod is stuck in ContainerCreating.

Fix: WaitForFirstConsumer for multi-AZ clusters. The PV is created in the Pod’s AZ.

Anti-pattern 3: Delete reclaim on critical data

# WRONG (database)
reclaimPolicy: Delete

Accidental PVC deletion destroys the data. There is no recovery path.

Fix: Retain for stateful data. Have a tested snapshot restore as the safety net.

Anti-pattern 4: missing volume expansion

# WRONG (database that will grow)
allowVolumeExpansion: false  # or unset

A database that fills the PVC cannot expand. The PVC must be replaced (data migration), or the workload fails when the volume is full.

Fix: allowVolumeExpansion: true for any workload that may grow.

Anti-pattern 5: missing topology constraints

# WRONG (multi-AZ cluster without allowedTopologies)
volumeBindingMode: WaitForFirstConsumer
# allowedTopologies missing

The PV can be created in any AZ, including AZs that the Pod cannot reach. The PVC may be Pending because no matching topology is available.

flowchart LR
    A[Pod scheduled to us-east-1b] --> B[PVC: storageClassName: db-ssd]
    B --> C{PV created in?}
    C -->|us-east-1b| D[Mount succeeds]
    C -->|us-east-1a| E[Mount fails - cross-AZ not supported]
    C -->|us-east-1c| F[Mount fails - cross-AZ not supported]

Fix: specify allowedTopologies with the cluster’s AZs.

Anti-pattern 6: wrong provisioner parameters

# WRONG
parameters:
  type: gp3
  iops: "100000"   # exceeds gp3 max

The CSI driver rejects the parameters; the PVC is Pending with a parameter validation error.

Fix: validate the parameters against the backend’s documentation. Test the StorageClass with a test PVC.

Anti-pattern 7: missing mountOptions

# WRONG (database workload)
# No mountOptions

The default mount options may not be optimal for the workload (e.g., noatime is missing, hurting performance).

Fix: specify mountOptions for the workload (e.g., noatime, discard for SSDs).

Anti-pattern 8: one StorageClass for everything

# WRONG
kind: StorageClass
metadata:
  name: storage
provisioner: ebs.csi.aws.com
parameters:
  type: gp3

A single StorageClass for all workloads. Databases and batch processing share the same tier; performance is unpredictable.

Fix: per-workload StorageClasses. Databases get db-ssd; web apps get web-standard; batch processing gets batch-throughput.

Anti-pattern 9: changing StorageClass parameters without migration

# WRONG
# Edit a StorageClass to change the provisioner parameters
# All new PVCs use the new parameters; existing PVCs are unaffected
# The team assumes the change applies to existing PVCs

StorageClass parameters apply only to new PVCs provisioned after the change. Existing PVCs (and their PVs) retain the old parameters. The operator who assumes the change applies to all PVCs is surprised.

Fix: understand that StorageClass parameters are a template for new PVCs. Existing PVCs are not retroactively re-parameterized. For an existing PVC to use new parameters, a new PVC must be created.

The audit

# 1. PVCs relying on the default StorageClass
kubectl get pvc -A -o json | \
  jq '.items[] | select(.spec.storageClassName == null) |
    {name: .metadata.name, namespace: .metadata.namespace}'

# 2. StorageClasses without allowVolumeExpansion
kubectl get storageclass -o json | \
  jq '.items[] | select(.allowVolumeExpansion != true) |
    {name: .metadata.name, allowVolumeExpansion: .allowVolumeExpansion}'

# 3. StorageClasses with Delete reclaim
kubectl get storageclass -o json | \
  jq '.items[] | select(.reclaimPolicy == "Delete") |
    {name: .metadata.name, reclaimPolicy: .reclaimPolicy}'

# 4. StorageClasses with Immediate binding
kubectl get storageclass -o json | \
  jq '.items[] | select(.volumeBindingMode == "Immediate") |
    {name: .metadata.name, volumeBindingMode: .volumeBindingMode}'

# 5. Released PVs with Retain reclaim
kubectl get pv -o json | \
  jq '.items[] | select(.status.phase == "Released") |
    {name: .metadata.name, reclaimPolicy: .spec.persistentVolumeReclaimPolicy}'

The audit output is the action list. Each item is a remediation task.

The production StorageClass hierarchy

A production cluster has multiple StorageClasses, each tuned for a workload class:

# Database tier
metadata:
  name: db-high-iops
provisioner: ebs.csi.aws.com
parameters:
  type: io2
  iops: "10000"
reclaimPolicy: Retain
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer
allowedTopologies:
- matchLabelExpressions:
  - key: topology.kubernetes.io/zone
    values: ["us-east-1a", "us-east-1b", "us-east-1c"]

# Web tier
metadata:
  name: web-standard
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
reclaimPolicy: Delete
allowVolumeExpansion: true
volumeBindingMode: WaitForFirstConsumer

# Batch tier
metadata:
  name: batch-throughput
provisioner: ebs.csi.aws.com
parameters:
  type: st1
reclaimPolicy: Delete
allowVolumeExpansion: false
volumeBindingMode: WaitForFirstConsumer

Each StorageClass encodes the workload’s requirements. The hierarchy is documented in the cluster bootstrap; the audit verifies compliance.

Quiz

Production discipline

• Per-workload StorageClasses. One per workload tier (db, web, batch, etc.).
• WaitForFirstConsumer for multi-AZ. The PV follows the Pod.
• Retain for stateful data. The reclaim policy matches the backup strategy.
• allowVolumeExpansion: true for any workload that grows.
• allowedTopologies for multi-region or constrained AZs. The PV is in a topology the cluster can reach.
• Audit StorageClasses regularly. Catch the anti-patterns before the incidents.