Storage architecture — designing storage for production clusters

A production storage architecture is not “use the default StorageClass.” It is a layered design: per-workload StorageClasses, reclaim policies aligned to the backup strategy, multi-AZ awareness, and a documented operational discipline. This lesson walks the design.

The per-workload StorageClass hierarchy

A production cluster has multiple StorageClasses, each tuned for a workload class:

# Database tier: high IOPS, Retain
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: db-high-iops
provisioner: ebs.csi.aws.com
parameters:
  type: io2
  iops: "10000"
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Retain
allowVolumeExpansion: true

---
# Web tier: standard, Delete
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: web-standard
provisioner: ebs.csi.aws.com
parameters:
  type: gp3
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete
allowVolumeExpansion: true

---
# Batch tier: throughput-optimized, Delete
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: batch-throughput
provisioner: ebs.csi.aws.com
parameters:
  type: st1
volumeBindingMode: WaitForFirstConsumer
reclaimPolicy: Delete

---
# Shared content: NFS, RWX
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: shared-nfs
provisioner: nfs.csi.k8s.io
parameters:
  server: nfs.prod.example.com
  path: /exports
volumeBindingMode: Immediate
reclaimPolicy: Delete

The hierarchy is per-workload, not per-team or per-app. Each workload class has different performance, durability, and reclaim requirements.

Reclaim policies and the backup strategy

The reclaim policy is the most consequential decision in the storage architecture:

The reclaim policy must align with the backup strategy:

• Stateful databases with critical data: Retain. The PV is not deleted when the PVC is deleted; the operator decides when to reclaim.
• Stateless workloads with no critical data: Delete. The PV and underlying storage are deleted when the PVC is deleted.
• Ephemeral workloads (batch processing): Delete. The PV is deleted on completion.

A Delete StorageClass for a database is a data-loss waiting to happen. The operator who deletes the PVC deletes the data; there is no second chance.

Multi-AZ awareness

A multi-AZ cluster must consider AZ affinity:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: db-high-iops
provisioner: ebs.csi.aws.com
parameters:
  type: io2
volumeBindingMode: WaitForFirstConsumer  # <-- key
allowedTopologies:
- matchLabelExpressions:
  - key: topology.kubernetes.io/zone
    values: ["us-east-1a"]

With WaitForFirstConsumer, the PV is created in the same AZ as the Pod that consumes it. Without it, the PV may be created in a different AZ and the Pod cannot mount it.

For multi-AZ databases:

• Option A: one StorageClass per AZ, with explicit topology constraints. The user picks the AZ.
• Option B: one StorageClass with WaitForFirstConsumer and allowedTopologies for the cluster’s AZs. The PV follows the Pod.
• Option C: a stretched cluster (rare in Kubernetes) with shared storage across AZs.

The standard production pattern is Option B: one StorageClass with multi-AZ topology, the PV created in the Pod’s AZ.

Snapshot integration

The snapshot controller and CRDs allow CSI snapshots:

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshotClass
metadata:
  name: db-snapshot
driver: ebs.csi.aws.com
parameters:
  type: snap
deletionPolicy: Delete

The user creates a VolumeSnapshot of a PVC:

apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: db-snapshot-2026-08-16
spec:
  volumeSnapshotClassName: db-snapshot
  source:
    persistentVolumeClaimName: data-postgres-0

Snapshots are part of the storage architecture, not a separate concern. The backup strategy uses snapshots; the recovery procedure uses snapshots; the data-loss prevention discipline uses snapshots.

The data-loss prevention discipline

Production storage requires a documented discipline:

• Reclaim policy matches the backup strategy. Retain for stateful data; Delete for ephemeral.
• Snapshots are scheduled and tested. Snapshots that are never restored are not a backup.
• PV orphans are detected and reconciled. Released PVs indicate data that is at risk.
• Capacity is monitored. Volumes that are near full trigger alerts; expansion is automated where possible.
• Multi-AZ topology is enforced. Pods cannot mount cross-AZ volumes.
• The PVC deletion is gated by RBAC. Production PVC deletion requires explicit approval.

Quiz

Production discipline

• Per-workload StorageClasses are the production standard. The default is for development only.
• Reclaim policy matches the backup strategy. Retain for stateful data; Delete for stateless and ephemeral.
• WaitForFirstConsumer for multi-AZ correctness. The PV follows the Pod; cross-AZ mounts are avoided.
• Snapshots are scheduled and tested. Snapshots that are never restored are not a backup.
• Audit Released PVs regularly. A Released PV with Delete reclaim is a storage leak; with Retain reclaim it is awaiting operator decision.
• PVC deletion is gated by RBAC. Production PVC deletion requires explicit approval.