Ordered deployment and the partition field — controlling rollout sequence

The ordered lifecycle is the StatefulSet’s most distinctive feature and its most common operational hazard. Pods are created 0, 1, 2, ... and deleted n-1, n-2, ..., 0. The controller waits for each Pod to be Ready before creating the next. A bad configuration that prevents one ordinal from becoming Ready blocks every later ordinal, and the StatefulSet appears “stuck” with no obvious cause.

Ordered creation

sequenceDiagram
    participant K as StatefulSet controller
    participant P0 as postgres-0
    participant P1 as postgres-1
    participant P2 as postgres-2
    Note over K: replicas: 0 -> 3
    K->>P0: create
    P0->>K: Pending
    P0->>K: ContainerCreating
    P0->>K: Running, Ready=True
    K->>P1: create
    P1->>K: Pending
    P1->>K: ContainerCreating
    P1->>K: Running, Ready=True
    K->>P2: create
    P2->>K: Running, Ready=True
    Note over K: replicas: 3 ready

Each Pod waits for the previous ordinal to be Ready. The controller does not parallelise creation across ordinals.

The reason is correctness: clustered software’s bootstrap logic typically relies on the first ordinal being the primary (or some other seed) and the rest joining the cluster. If the controller spawned all ordinals simultaneously, the “first to start” race would be non-deterministic.

Ordered deletion

Deletion runs in reverse ordinal order: highest ordinal first.

sequenceDiagram
    participant K as StatefulSet controller
    participant P0 as postgres-0
    participant P1 as postgres-1
    participant P2 as postgres-2
    Note over K: replicas: 3 -> 0
    K->>P2: delete
    P2->>K: Terminating
    P2->>K: gone
    K->>P1: delete
    P1->>K: gone
    K->>P0: delete
    P0->>K: gone
    Note over K: 0 ready

Reverse order means the cluster’s primary (typically ordinal 0) is the last to go. The replicas are removed first, the primary last. This gives the workload a chance to drain the cluster cleanly.

The partition field for staged rollouts

spec.updateStrategy.rollingUpdate.partition controls how many Pods get the new template during a rolling update. With partition: 1 and replicas: 3, only ordinals ≥ 1 are updated; ordinal 0 keeps the old template.

spec:
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      partition: 1

The use case: a release engineer wants to update postgres-2 first (a non-primary replica), observe it, then update postgres-1, then postgres-0 (the primary) last. The partition lets the operator sequence the rollout by hand:

flowchart LR
    A["partition: 3<br/>no ordinals updated"] --> B["partition: 2<br/>only postgres-2"]
    B --> C["partition: 1<br/>postgres-2 and postgres-1"]
    C --> D["partition: 0<br/>all ordinals updated"]

$ kubectl patch statefulset postgres -n data --type=merge -p '{"spec":{"updateStrategy":{"rollingUpdate":{"partition":2}}}}'

statefulset.apps/postgres patched

$ kubectl rollout status statefulset/postgres -n data

Waiting for ordinals [2] to be ready
waiting for statefulset rolling update to complete 2 out of 3 new pods updated...

A partition smaller than the previous step is a rollback operation (re-apply the old template to the previously updated ordinals).

Failure modes

Stuck ordinal

A Pod that never becomes Ready blocks every subsequent ordinal. Causes:

flowchart TB
    A[Stuck ordinal] --> B{Image pull<br/>failure?}
    A --> C{Readiness probe<br/>failing?}
    A --> D{PVC not bound?}
    A --> E{Init container<br/>hangs?}
    A --> F{Resource quota<br/>exhausted?}

Each one is diagnosable with kubectl describe pod and kubectl logs. A failure to find any of these after the usual checks usually points to application-level bootstrap failure — the container is Running but not Ready because the application’s own readiness check (e.g., the PostgreSQL cluster replication check) is failing.

Headless Service missing

The StatefulSet references a headless Service in serviceName. If that Service is missing, the StatefulSet’s Pods fail DNS resolution and the application’s bootstrap cannot find peers. The Pods appear Running but stuck.

Wrong PVC access mode

A StatefulSet’s PVC with ReadWriteOnce cannot be mounted on two nodes. If the StatefulSet tries to schedule ordinals that land on the same node, the second Pod’s volume mount fails. This is rare with podAntiAffinity but does happen with small clusters.

Update with bad template

A template change that breaks the application — wrong env, broken readiness probe, missing ConfigMap — affects only the updated ordinals. With partition: 0, all ordinals update. The operator must kubectl edit the StatefulSet to revert, but the controller will not apply the old template automatically; the operator must kubectl apply the manifest with the old template and observe the rollout revert.

Why ordered is necessary

The cluster bootstrap is the answer. A replicated database cluster needs a known starting topology:

• PostgreSQL primary + replicas. Replica 1 connects to the primary (postgres-0); replica 2 joins the cluster. If both replicas start simultaneously, neither knows which is primary.
• ZooKeeper ensemble. Each member needs the full member list. Orderly startup ensures the first member sees a single-member ensemble, the second sees {1, 2}, the third sees {1, 2, 3}. The leader election is consistent at each step.
• Kafka brokers. Each broker’s cluster.id is shared across the ensemble, but the broker’s broker.id is ordinal-specific. Sequential startup avoids racing on topic-partition leadership.

A Deployment, by contrast, has no ordering and is designed for stateless workloads. The choice of controller is the choice of semantics.

Inspecting the order

kubectl get pods -l app=postgres -n data -o custom-columns=\
  NAME:.metadata.name,ORDINAL:.metadata.labels.statefulset\.kubernetes\.io/pod-name,READY:.status.conditions[?(@.type=="Ready")].status,AGE:.metadata.creationTimestamp

flowchart LR
    A["StatefulSet controller<br/>reconcile loop"] --> B{desired replicas<br/>match current?}
    B -->|no, scale up| C["Create next ordinal<br/>wait for Ready"]
    B -->|no, scale down| D["Delete highest ordinal<br/>wait for gone"]
    B -->|yes| E{updateStrategy<br/>needs update?}
    E -->|yes| F["Update ordinals<br/>≥ partition"]
    E -->|no| G[Idle until next reconcile]

Quiz

Production discipline

• The partition field is your rollout knob. Use it for every production StatefulSet release; do not update all ordinals at once.
• Test the bootstrap path. A new ordinal must join the cluster from cold. Run kubectl scale statefulset postgres --replicas=4 in staging; verify the new replica joins.
• Runbook the rollback. kubectl apply -f with the old template and partition: 0 is the rollback path. The partition knob is the difference between a canary and a full outage.
• Monitor Ready replicas per ordinal. kube_statefulset _replicas and kube_statefulset_status_replicas_current are the metrics; alerts should fire on status_replicas_ready != spec_replicas.
• Avoid ordering-sensitive configuration. A broken readiness probe is worse on a StatefulSet than on a Deployment because the stuck ordinal blocks scaling.

Ordered deployment is the StatefulSet’s defining feature. Operators who understand it can run Postgres, Kafka, and ZooKeeper in production. Operators who do not understand it will lose data when the ordinal they did not realise mattered gets stuck.