StatefulSet operations — scaling, rolling out, and the deletion sequence

A StatefulSet is a long-lived commitment. Day-2 operations on it are not “edit and observe.” Each operation is a sequence of application-level and Kubernetes-level steps. This lesson covers scaling, rollout, and deletion in production.

Scaling up

Scaling up adds a new ordinal. For a clustered workload, the new ordinal must join the existing cluster:

sequenceDiagram
    participant K as StatefulSet
    participant N as New Pod (ordinal N)
    participant C as Existing cluster
    participant OP as Operator
    OP->>K: kubectl scale statefulset postgres --replicas=4
    K->>N: create postgres-3
    N->>C: bootstrap join
    C->>N: cluster member
    N->>K: Ready
    Note over C: cluster now has 4 members

The StatefulSet controller creates postgres-3. The Pod runs the application’s bootstrap logic, which connects to the cluster as a new member. For PostgreSQL this is pg_basebackup from the primary; for Kafka this is kafka-reassign-partitions to move partitions to the new broker; for ZooKeeper this is the new member joining the ensemble.

Scaling down

Scaling down deletes the highest ordinal first. The application-level cleanup runs before the Pod is terminated:

sequenceDiagram
    participant OP as Operator
    participant A as Application CLI
    participant K as StatefulSet
    participant P as Old Pod (ordinal 3)
    OP->>A: decluster postgres-3
    A->>P: remove from cluster
    OP->>K: kubectl scale statefulset postgres --replicas=3
    K->>P: delete
    P->>K: Terminating
    P->>K: gone

For PostgreSQL: stop the replica (pg_ctl stop), remove the replication slot, drop the replica from the cluster topology. For Kafka: reassign partitions away from the broker, then remove the broker from the cluster (kafka-configs.sh --alter --delete-config).

The StatefulSet’s PVC for the removed ordinal is not deleted. It remains as Available or Bound depending on the StorageClass. Re-scaling back up to 4 will rebind the existing PVC; the data is preserved.

Rolling out a new template

The partition field drives staged rollouts:

spec:
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      partition: 3

With replicas: 3, partition: 3 means no ordinals are updated. partition: 2 updates only ordinal 2. partition: 0 updates all ordinals.

The operational pattern:

flowchart LR
    A["partition: 3<br/>no ordinals updated"] --> B["partition: 2<br/>update postgres-2"]
    B --> C{postgres-2 healthy?}
    C -->|yes| D["partition: 1<br/>update postgres-1"]
    C -->|no| E["rollback: partition: 3<br/>revert postgres-2"]
    D --> F{postgres-1 healthy?}
    F -->|yes| G["partition: 0<br/>update postgres-0"]
    F -->|no| E
    G --> H{postgres-0 healthy?}
    H -->|yes| I[rollout complete]
    H -->|no| J["emergency: full rollback"]

For a Postgres primary + replicas, the primary (ordinal 0) updates last. This requires the application to support replica rolling restarts (most do) and a primary switch procedure for the operator to perform when postgres-0 is finally updated.

# Update image
kubectl set image statefulset/postgres \
  postgres=postgres:16.2 -n data

# Stagger rollout
kubectl patch statefulset postgres -n data --type=merge \
  -p '{"spec":{"updateStrategy":{"rollingUpdate":{"partition":2}}}}'
kubectl rollout status statefulset/postgres -n data

# After observation
kubectl patch statefulset postgres -n data --type=merge \
  -p '{"spec":{"updateStrategy":{"rollingUpdate":{"partition":1}}}}'
kubectl rollout status statefulset/postgres -n data

# Final update
kubectl patch statefulset postgres -n data --type=merge \
  -p '{"spec":{"updateStrategy":{"rollingUpdate":{"partition":0}}}}'
kubectl rollout status statefulset/postgres -n data

Rolling back

Rollback is kubectl apply with the old template and partition: 0. The controller reconciles and reverts every ordinal to the old template.

git log --oneline -- src/statefulset/postgres.yaml
# abc1234 bump postgres to 16.2 (current)
# def5678 postgres 16.1 baseline
git checkout def5678 -- src/statefulset/postgres.yaml
kubectl apply -f src/statefulset/postgres.yaml
kubectl rollout status statefulset/postgres -n data

The PVCs are not touched. The application-level rollback runs: PostgreSQL replicas re-sync from the primary, Kafka reassigns partitions, ZooKeeper re-elects leaders.

Deleting a StatefulSet

Deletion is destructive in three ways:

1. Pods are deleted in reverse ordinal order.
2. PVCs are not deleted by default. They remain as Bound to the deleted Pods (or Released if kubectl delete pvc is run separately).
3. The PV’s reclaim policy decides what happens to the underlying storage. Retain keeps it; Delete deletes it.

kubectl delete statefulset postgres -n data
# statefulset.apps "postgres" deleted

kubectl get pvc -n data -l app.kubernetes.io/name=postgres
# NAME              STATUS   VOLUME                            CAPACITY
# data-postgres-0   Bound    pvc-7c8f2d8e-...                 100Gi
# data-postgres-1   Bound    pvc-3a9b4c1d-...                 100Gi
# data-postgres-2   Bound    pvc-5e6f8a2b-...                 100Gi

For a complete cleanup:

kubectl delete statefulset postgres -n data
kubectl delete pvc -n data -l app.kubernetes.io/name=postgres
# PVs with reclaimPolicy: Retain remain as Released; clean them up separately
kubectl get pv | grep Released

Recovering from a failed scaling event

A scale-up that creates a new ordinal but fails to bootstrap the new replica:

flowchart TB
    A[Scale 3 to 4] --> B[postgres-3 created]
    B --> C{Application<br/>bootstrap?}
    C -->|fails| D[Pod Running but not Ready]
    D --> E{Investigate logs}
    E --> F[Application-level fix]
    F --> G[bootstrap manually]
    G --> H[Pod becomes Ready]
    H --> I["StatefulSet 4/4"]

The StatefulSet’s scale-up completes when the Pod is Ready — but Ready depends on the application reporting ready. Often the application needs a manual nudge (pg_basebackup in PostgreSQL, manual rebalance in Elasticsearch).

The risk: leaving the StatefulSet in a “scaling” state for hours. The Partition field doesn’t help; the new ordinal exists and the cluster believes it should be Ready.

Quiz

Production discipline

• The StatefulSet manifest is a long-lived contract. Treat changes as code review-level events; the runbook for scaling is part of the manifest’s README.
• Rollouts are staged by default. Never set partition: 0 and observe; the only safe default is starting from a high partition and walking down.
• Application-level operations run alongside Kubernetes-level ones. Scaling without the application’s bootstrap is incomplete; deleting without the application’s decommission is destructive.
• Deletion is irreversible at the data layer. A StorageClass with reclaimPolicy: Delete plus a kubectl delete pvc deletes the underlying volume. Verify the cluster is empty (or backed up) before deleting.
• Snapshot policy is part of the StatefulSet design. The VolumeSnapshot schedule, the snapshot’s sourceVolumeMode, and the application’s crash-consistency requirements are co-designed.

StatefulSet operations are not Deployment operations. The operator who treats them as Deployments will lose data. The operator who treats them as database operations has the discipline to keep them running.