StatefulSet alone is not enough — why stateful workloads need an Operator

StatefulSet is a foundational primitive for stateful workloads in Kubernetes, but it is not enough on its own. This lesson walks what StatefulSet provides, what it does not, and why an Operator is the production standard for serious stateful workloads.

What StatefulSet provides

A StatefulSet provides:

flowchart LR
    A[StatefulSet spec] --> B[Stable identity: pod-0, pod-1, ...]
    A --> C[Ordered deployment]
    A --> D[Per-Pod PVCs from volumeClaimTemplates]
    A --> E[Per-Pod Service: Headless Service]

The standard production features:

• Stable network identity: each Pod has a stable hostname (e.g., postgres-0.postgres.prod.svc).
• Stable storage identity: each Pod has a stable PVC (e.g., data-postgres-0).
• Ordered deployment and scaling: Pods are created in order (0, 1, 2); termination is reverse order.
• Per-Pod PVCs: volumeClaimTemplates create one PVC per Pod.

What StatefulSet does not provide

A StatefulSet does not provide:

Backup and restore

StatefulSet does not know how to back up the application. It can trigger CSI snapshots (via VolumeSnapshot CRDs), but it cannot coordinate with the application for application-consistency.

# A VolumeSnapshot created from a StatefulSet's PVC
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: data-postgres-0-snap-20260816
spec:
  source:
    persistentVolumeClaimName: data-postgres-0
  volumeSnapshotClassName: postgres-snap

The snapshot is created, but it is crash-consistent. The database may be in the middle of a transaction. The restore may require WAL replay; some transactions may be lost.

Scaling logic

StatefulSet can scale up and down, but it does not understand application-level scaling:

• For PostgreSQL: streaming replication topology must be reconfigured when adding a replica.
• For Kafka: partition reassignment must be performed when adding a broker.
• For Cassandra: bootstrap must be performed when adding a node.

# StatefulSet scales from 3 to 5 replicas
kubectl scale statefulset postgres --replicas=5
# Pods postgres-3 and postgres-4 are created
# But the PostgreSQL primary does not know about the new replicas
# Streaming replication is not configured

Upgrade coordination

StatefulSet can roll out a new image, but it does not understand application upgrades:

• For PostgreSQL: a major version upgrade requires pg_upgrade; the database must be stopped during the upgrade.
• For Kafka: a major version upgrade requires a rolling restart with specific compatibility settings.
• For Elasticsearch: a rolling upgrade requires shard allocation awareness.

# StatefulSet rolls out a new image
kubectl rollout restart statefulset postgres
# Pods are restarted in order with the new image
# But PostgreSQL is not upgraded (pg_upgrade is not run)
# The database is still running the old version

Quorum management

StatefulSet does not understand quorum:

• For a 5-replica ZooKeeper: 3 nodes form a quorum; the StatefulSet can scale to 2 (losing quorum) or to 4 (still has quorum).
• For a 3-replica etcd: 2 nodes form a quorum; the StatefulSet can scale to 1 (losing quorum).

Monitoring and metrics

StatefulSet does not expose application metrics. A PostgreSQL StatefulSet has no built-in exporter for connection counts, replication lag, or query latency.

The Operator pattern

The Operator pattern fills the gaps. An Operator is a Kubernetes controller that:

• Watches custom resources (CRs) that represent the application.
• Reconciles the CR’s desired state with the actual state.
• Encodes application-specific knowledge (backup, restore, scaling, upgrade, quorum).

flowchart LR
    A[Custom Resource: PostgreSQL] --> B[PostgreSQL Operator]
    B --> C[StatefulSet: postgres]
    B --> D[Service: postgres-primary, postgres-replica]
    B --> E[VolumeSnapshot: postgres-snap]
    B --> F[Secret: postgres-credentials]
    B --> G[ConfigMap: postgres-config]

The Operator:

• Creates and manages the StatefulSet.
• Configures streaming replication when scaling.
• Triggers application-consistent snapshots.
• Coordinates major version upgrades.
• Exposes application metrics.

The standard production Operators

For stateful workloads:

• PostgreSQL: Zalando’s postgres-operator, Cloud Native PG, Crunchy Data.
• MySQL: Oracle MySQL Operator, Percona Operator for MySQL.
• MongoDB: MongoDB Kubernetes Operator, Percona Operator for MongoDB.
• Kafka: Strimzi, Confluent Operator.
• Redis: Redis Operator.
• Elasticsearch: Elastic Cloud on Kubernetes (ECK), OpenSearch Operator.
• etcd: etcd-operator (less common; many clusters use the control-plane etcd directly).
• ZooKeeper: ZooKeeper Operator.

Each operator encodes the application-specific knowledge for its database.

The production pattern

sequenceDiagram
    participant U as User
    participant O as Operator
    participant S as StatefulSet
    participant DB as Database
    U->>O: create PostgreSQL CR
    O->>S: create StatefulSet with 3 replicas
    S->>DB: start postgres-0, postgres-1, postgres-2
    O->>DB: configure streaming replication
    Note over O,DB: operator has application knowledge
    U->>O: scale PostgreSQL to 5 replicas
    O->>S: scale StatefulSet to 5
    S->>DB: start postgres-3, postgres-4
    O->>DB: reconfigure replication to include new replicas
    U->>O: take backup
    O->>DB: pg_start_backup
    O->>S: trigger VolumeSnapshot
    O->>DB: pg_stop_backup

The Operator:

• Knows the application’s backup procedure.
• Knows the application’s scaling procedure.
• Knows the application’s upgrade procedure.
• Coordinates with the application for consistency.

Quiz

Production discipline

• StatefulSet alone is not enough. It provides the foundation, not the application knowledge.
• Operators are the production standard. Cloud Native PG, Zalando, Strimzi, ECK, etc.
• Migration to an Operator is a project. Plan the data migration, the cutover, the rollback.
• Test the Operator’s features. Backup, restore, scaling, upgrade. Each is a runbook entry.
• The Operator is part of the cluster bootstrap. Document it; maintain it; upgrade it.