Stateful workload anti-patterns — the storage mistakes that cause data loss

Stateful workload anti-patterns are silent until the incident: a database Pod is rescheduled and the data is gone; a PVC is deleted and the volume is gone; a scaling operation breaks the replication topology. This lesson walks the common anti-patterns and the production discipline for avoiding them.

Anti-pattern 1: emptyDir for databases

The most common data-loss anti-pattern:

# WRONG
volumes:
- name: data
  emptyDir: {}

A database using emptyDir appears to work until the Pod is rescheduled. The data is on the node’s local filesystem; the new Pod has an empty emptyDir; the data is gone.

Fix: use PVC for stateful data. Deploy a CSI driver.

Anti-pattern 2: hostPath for state

# WRONG
volumes:
- name: data
  hostPath:
    path: /var/lib/data

hostPath ties the data to one node. A database using hostPath cannot move to another node; HA is impossible.

Fix: use PVC with a CSI driver. Even on bare metal, deploy Rook-Ceph, Longhorn, or OpenEBS.

Anti-pattern 3: missing backup strategy

A database with no snapshot schedule, no backup procedure, no tested restore:

# WRONG
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  template:
    spec:
      containers:
      - name: postgres
        image: postgres:16
        volumeMounts:
        - name: data
          mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      storageClassName: db-default
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 100Gi

The database runs; no backup exists; a disk failure destroys the data.

Fix: deploy an Operator (Cloud Native PG, Zalando); configure the backup policy (Barman, S3); test the restore.

Anti-pattern 4: scaling without an Operator

# WRONG (StatefulSet without Operator)
kubectl scale statefulset postgres --replicas=5

The StatefulSet scales to 5; the new Pods start; but the database does not include them in the replication topology. The application is broken.

Fix: deploy a database Operator that knows how to reconfigure replication when scaling.

Anti-pattern 5: crash-consistent snapshots

# WRONG (VolumeSnapshot without coordination)
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: data-postgres-snap
spec:
  source:
    persistentVolumeClaimName: data-postgres-0
  volumeSnapshotClassName: postgres-snap

The snapshot is created without coordinating with the database. The result may be a corrupted restore.

Fix: use an Operator that coordinates the snapshot with the database (pg_start_backup / pg_stop_backup).

Anti-pattern 6: Delete reclaim on critical data

# WRONG (database on StorageClass with Delete reclaim)
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: db-default
provisioner: ebs.csi.aws.com
reclaimPolicy: Delete
parameters:
  type: gp3

Accidental PVC deletion destroys the EBS volume. The data is gone.

Fix: Retain reclaim for stateful data.

Anti-pattern 7: PVC without explicit StorageClass

# WRONG
spec:
  accessModes: ["ReadWriteOnce"]
  resources:
    requests:
      storage: 100Gi
  # storageClassName: not specified

The PVC binds to the cluster default. The default may be wrong for the workload.

Fix: specify storageClassName explicitly in every production PVC.

Anti-pattern 8: no monitoring on the database

A database without monitoring is invisible until it fails. The production rule:

• RED metrics (rate, errors, duration) for the application.
• Database-specific metrics (connections, replication lag, query latency).
• Storage metrics (PVC capacity, IOPS, latency).

Fix: deploy a metrics exporter; configure alerts.

The audit

A cluster audit for stateful workload anti-patterns:

# 1. Pods using emptyDir for stateful data
kubectl get pods -A -o json | \
  jq '.items[] | select(.spec.volumes[]? | .emptyDir != null) |
    select(.metadata.labels.app != "ephemeral-app") |
    {name: .metadata.name, namespace: .metadata.namespace}'

# 2. Pods using hostPath
kubectl get pods -A -o json | \
  jq '.items[] | select(.spec.volumes[]? | .hostPath != null) |
    {name: .metadata.name, namespace: .metadata.namespace,
     hostPath: .spec.volumes[] | select(.hostPath != null) | .hostPath.path}'

# 3. PVCs without explicit StorageClass
kubectl get pvc -A -o json | \
  jq '.items[] | select(.spec.storageClassName == null) |
    {name: .metadata.name, namespace: .metadata.namespace}'

# 4. StatefulSets without an Operator (manual check)
# Look for CRDs that match the database
kubectl get crd | grep -E 'postgres|kafka|redis|mongo'

# 5. StorageClasses with Delete reclaim
kubectl get storageclass -o json | \
  jq '.items[] | select(.reclaimPolicy == "Delete") |
    {name: .metadata.name}'

The audit output is the action list.

Quiz

Production discipline

• Audit stateful workloads regularly. emptyDir, hostPath, missing backup, missing Operator — each is a silent data-loss incident.
• Deploy Operators for production stateful workloads. Cloud Native PG, Strimzi, ECK, etc.
• Layered backups with application-consistency. Snapshot + file + logical; tested regularly.
• Retain reclaim for stateful data. The reclaim policy matches the backup strategy.
• Monitor the database. RED metrics + database- specific metrics + storage metrics.