StatefulSets — when Deployments are not the right controller

A StatefulSet is a controller for workloads that need a stable identity — Pods that are individually addressable, have a predictable name, and carry per-Pod persistent storage. It is the right controller for databases, message brokers, and other clustered software where “Pod 0” matters. It is the wrong controller for stateless HTTP services, where a Deployment already does everything needed and is cheaper to operate.

What StatefulSets provide

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  serviceName: postgres-h
  replicas: 3
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
      - name: postgres
        image: postgres:16
        volumeMounts:
        - name: data
          mountPath: /var/lib/postgresql/data
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      resources:
        requests:
          storage: 100Gi

Compared to a Deployment, three new things:

1. serviceName — a headless Service required by the StatefulSet. Provides stable DNS for individual Pods (postgres-0.postgres-h.prod.svc.cluster.local).
2. volumeClaimTemplates — a template that creates one PVC per Pod. Pod postgres-0 gets PVC data-postgres-0; Pod postgres-1 gets data-postgres-1. The PVC is bound to the Pod by name, not by selector.
3. Ordered operations — Pods are created in ordinal order (0, 1, 2), and on deletion in reverse order. Scaling up adds one Pod at a time; scaling down removes the highest ordinal first.

flowchart LR
    A["StatefulSet: postgres<br/>replicas: 3"] --> B[postgres-0]
    A --> C[postgres-1]
    A --> D[postgres-2]
    B --> B1["PVC: data-postgres-0"]
    C --> C1["PVC: data-postgres-1"]
    D --> D1["PVC: data-postgres-2"]

Stable identity

A StatefulSet assigns each Pod a stable ordinal index. The Pod’s name is <statefulset-name>-<ordinal>. The hostname is <pod-name>.<service-name>.<namespace>.svc.cluster. local. The hostname does not change for the lifetime of the Pod.

This matters for clustered software that uses hostnames as cluster member identifiers: ZooKeeper ensemble members, PostgreSQL replicas identified by hostname, Kafka brokers that bind to specific advertised listeners. A Deployment’s random Pod name changes whenever a Pod is rescheduled; a StatefulSet’s ordinal stays attached to the same volume, even across node failures.

Ordered deployment

StatefulSets deploy Pods in ordinal order. Pod N is not created until Pod N-1 is Ready. The same rule applies to scaling: scale from 3 to 5 creates pod-3, waits for Ready, then creates pod-4.

sequenceDiagram
    participant K as StatefulSet controller
    participant P0 as postgres-0
    participant P1 as postgres-1
    participant P2 as postgres-2
    K->>P0: create
    P0->>K: Ready
    K->>P1: create
    P1->>K: Ready
    K->>P2: create
    P2->>K: Ready
    Note over K: scaling 3 -> 5
    K->>P2: Ready
    K->>P3: create (ordinal 3)
    P3->>K: Ready
    K->>P4: create (ordinal 4)
    P4->>K: Ready

The order is the source of most StatefulSet complexity. It allows the workload’s bootstrap logic to assume a known set of peers (e.g., the first replica becomes the primary, the rest join). It also means a stuck ordinal blocks every later ordinal — a single bad configuration prevents the StatefulSet from scaling out.

Ordered deletion

Deletion runs in reverse ordinal order: pod-2 is deleted before pod-1 is deleted before pod-0. The controller waits for the previous deletion to complete before proceeding. PVCs are not deleted automatically (default StorageClass reclaimPolicy: Delete applies, but the StatefulSet does not remove the PVC).

sequenceDiagram
    participant K as StatefulSet controller
    participant P0 as postgres-0
    participant P1 as postgres-1
    participant P2 as postgres-2
    K->>P2: delete
    P2->>K: gone
    K->>P1: delete
    P1->>K: gone
    K->>P0: delete
    P0->>K: gone
    Note over K,P0: PVCs retained by default

When StatefulSets are correct

StatefulSets exist for one reason: workloads that need stable identity and per-Pod persistent storage. The canonical examples:

• Replicated databases. PostgreSQL with a primary + replicas, MySQL Group Replication, MongoDB replica sets.
• Message brokers. Kafka brokers need stable advertised listeners; ZooKeeper ensembles need stable member IDs.
• Coordination stores. etcd clusters, Consul servers.
• Clustered caches. Redis Cluster nodes, Memcached shards.

The test for “do I need a StatefulSet?” is: does the workload need to know which Pod it is? If yes — StatefulSet. If the workload is happy to be “any of the running Pods,” a Deployment is sufficient.

When StatefulSets are wrong

StatefulSets are expensive to operate. The common anti-patterns:

• A stateless web service with a PVC because “we need a volume.” Use a Deployment with emptyDir or hostPath for ephemeral storage; use a Deployment with no volume for pure stateless.
• A StatefulSet with a single replica because the workload “needs persistence.” A single-replica StatefulSet has the cost of a database controller without the benefit of clustering. A Deployment + PVC, or a single-node database Deployment with a volumeClaimTemplate-less StatefulSet, is cheaper.
• A StatefulSet for “stability” without a need for ordered lifecycle. The Pod-name stability is a side-effect, not a goal. A Deployment with a stable app.kubernetes.io/name label and pod-template-hash is enough.

Headless Services

A StatefulSet requires a headless Service (clusterIP: None). The Service’s selector matches the StatefulSet’s Pods. The DNS provider (CoreDNS) returns one A record per Pod, with the Pod’s ordinal in the hostname:

postgres-0.postgres-h.prod.svc.cluster.local.  A  10.244.1.12
postgres-1.postgres-h.prod.svc.cluster.local.  A  10.244.2.34
postgres-2.postgres-h.prod.svc.cluster.local.  A  10.244.1.55

Clients that need a specific replica connect to that hostname. Clients that need any replica use a regular Service or a CNAME that round-robins across the A records.

Inspecting a StatefulSet

kubectl get statefulset postgres -n data -o wide
# NAME       READY   AGE
# postgres   3/3     30d

kubectl get pods -l app=postgres -n data
# NAME          READY   STATUS    AGE
# postgres-0    1/1     Running   30d
# postgres-1    1/1     Running   30d
# postgres-2    1/1     Running   30d

kubectl get pvc -n data -l app.kubernetes.io/name=postgres
# NAME              STATUS   VOLUME                                     CAPACITY
# data-postgres-0   Bound    pvc-7c8f2d8e-...                          100Gi
# data-postgres-1   Bound    pvc-3a9b4c1d-...                          100Gi
# data-postgres-2   Bound    pvc-5e6f8a2b-...                          100Gi

$ kubectl describe statefulset postgres -n data

Name:               postgres
Namespace:          data
Image(s):           postgres:16
Selector:           app=postgres
Replicas:           3 desired | 3 total
Volumes:
data:
  Type:       PersistentVolumeClaim (a reference to a PVC in the same namespace)
  ClaimName:  data-postgres-N (template generated per Pod)
...

Quiz

Production discipline

• The StatefulSet is a long-lived commitment. Deleting it does not delete the data. Treat it as a database cluster, not a Deployment.
• replicas is rarely the right number of replicas. A production database needs quorum (odd replicas), replication topology, and quorum-loss handling. The StatefulSet only guarantees the count.
• The order of operations matters. Scaling up a StatefulSet requires the cluster to accept the new member. Scaling down is destructive. Both are operations that require database-level commands in addition to the Kubernetes-level kubectl scale.
• Volume reclaim policy is a design decision. Most StatefulSet workloads want Retain for the StorageClass so PVs survive PVC deletion; some want Delete for clean teardown. The default StorageClass’s policy applies if the StorageClass does not specify.
• Test the cluster’s bootstrap path. A new StatefulSet member needs the cluster’s existing data. The pattern is application-specific: PostgreSQL uses pg_basebackup, Kafka uses replication, etc. Test it on a regular cadence so the runbook is current.

StatefulSets are the right tool for clustered stateful workloads. They are not a default replacement for Deployments. Most production workloads are stateless; the few that are stateful need a StatefulSet, and the operator who runs them understands the controller.