Stable identity — ordinals, headless Services, and predictable DNS

The defining feature of a StatefulSet is stable network identity. Each Pod has an ordinal index that survives rescheduling, and that ordinal is encoded in the Pod’s hostname. A headless Service publishes the hostname-to-IP mapping as DNS A records. Clustered software uses these hostnames as member identifiers — replace the Pod, the hostname stays.

The hostname format

A StatefulSet Pod’s hostname is:

<pod-name>.<service-name>.<namespace>.svc.cluster.local

With serviceName: postgres-h and replicas: 3, the Pods are:

postgres-0.postgres-h.prod.svc.cluster.local
postgres-1.postgres-h.prod.svc.cluster.local
postgres-2.postgres-h.prod.svc.cluster.local

Each hostname resolves to a single Pod IP. When postgres-0 is rescheduled onto a different node, its hostname resolves to its new IP. The hostname does not change.

flowchart LR
    H1["postgres-0<br/>hostname"] --> A1[IP 10.244.1.12]
    H2["postgres-1<br/>hostname"] --> A2[IP 10.244.2.34]
    H3["postgres-2<br/>hostname"] --> A3[IP 10.244.1.55]
    A1 -.->|reschedule| H1

The headless Service

A headless Service is a Service with clusterIP: None. The Service has no ClusterIP; it does not load-balance. The DNS provider (CoreDNS) returns one A record per matched Pod instead of one record pointing at the Service’s ClusterIP.

apiVersion: v1
kind: Service
metadata:
  name: postgres-h
spec:
  clusterIP: None
  selector:
    app: postgres
  ports:
  - port: 5432
    name: postgres

The selector must match the StatefulSet’s Pod template labels. The StatefulSet’s serviceName field references this Service by name.

flowchart TB
    S["postgres-h<br/>headless Service<br/>clusterIP: None"]
    S -->|selector| P0["postgres-0<br/>A record"]
    S --> P1["postgres-1<br/>A record"]
    S --> P2["postgres-2<br/>A record"]

A regular Service with clusterIP non-None returns a single A record pointing at the ClusterIP, and the dataplane (kube-proxy or CNI) load-balances. A headless Service skips the ClusterIP and returns multiple A records — one per Pod.

DNS resolution in detail

kubectl run -it --rm --restart=Never --image=busybox:1.37 dns-test -- \
  nslookup postgres-0.postgres-h.prod.svc.cluster.local

Output:

Name:      postgres-0.postgres-h.prod.svc.cluster.local
Address:   10.244.1.12

And for the Service’s bare name (postgres-h.prod.svc.cluster.local):

Name:      postgres-h.prod.svc.cluster.local
Address:   10.244.1.12
Address:   10.244.2.34
Address:   10.244.1.55

The bare Service name resolves to all the StatefulSet’s Pod IPs. A client connecting to that name gets a random IP (typically round-robin, depending on the resolver). A client connecting to a specific ordinal gets that Pod’s IP.

$ kubectl get service postgres-h -n prod -o yaml

apiVersion: v1
kind: Service
metadata:
name: postgres-h
namespace: prod
spec:
clusterIP: None
ports:
- port: 5432
  name: postgres
selector:
  app: postgres

Why clustered software needs this

Replicated databases and message brokers require that each member has a stable identifier known to the cluster. Three examples:

• PostgreSQL with logical replication. The replica connects to the primary by hostname. If the primary is rescheduled, the new Pod has a new IP but the same hostname — the replica reconnects without cluster reconfiguration.
• ZooKeeper ensemble. Each ensemble member has a stable myid (an integer) and a stable advertised address (hostname + port). A new ensemble member can join without the others needing to learn a new address.
• Kafka brokers. Each broker has a stable broker.id and an advertised listener. StatefulSet ordinal maps to broker.id; the hostname is the advertised listener.

If the identifier changes on every restart, the cluster re-runs its discovery protocol on every Pod reschedule. Worse, in some software the identifier is persisted in the on-disk data directory; changing the identifier on reschedule causes the broker to refuse to start (it thinks it has been cloned from another broker).

Pod-name stability across rescheduling

sequenceDiagram
    participant K as StatefulSet
    participant N1 as node-1
    participant N2 as node-2
    K->>N1: create postgres-0
    N1->>K: Pod Running
    Note over N1: node-1 fails
    K->>N2: recreate postgres-0
    N2->>K: Pod Running
    Note over N2: hostname still<br/>postgres-0.postgres-h.prod

The Pod’s name (postgres-0) is the same. Its PVC (data-postgres-0) is rebound to the new Pod. The DNS A record is updated to the new IP. From the cluster’s perspective, the same broker with a new IP, not a new broker.

Failure modes

A misconfigured headless Service breaks the StatefulSet. The common cases:

flowchart TB
    A[StatefulSet Pod cannot resolve peers] --> B{Headless Service<br/>exists?}
    B -->|no| C["Create Service with clusterIP: None"]
    B -->|yes| D{Selector<br/>matches Pods?}
    D -->|no| E[Update selector to match Pod template labels]
    D -->|yes| F{DNS provider<br/>has records?}
    F -->|no| G[Check CoreDNS pods, kube-dns service]
    F -->|yes| H{Pod's<br/>/etc/resolv.conf<br/>correct?}
    H -->|no| I[Check Pod dnsPolicy, kubelet --cluster-dns]
    H -->|yes| J["Check pod-network namespace<br/>and search path"]

Service selector mismatch

The headless Service’s selector does not match the StatefulSet’s Pod template labels. Result: no DNS records; the StatefulSet Pods are “not in the Service.” The operator sees a working StatefulSet but the Pods cannot resolve each other.

Service name vs StatefulSet serviceName

The Service is named postgres-svc but the StatefulSet’s serviceName: postgres-h references a Service that does not exist. The API server does not validate this until the StatefulSet is reconciled; the controller logs Service not found.

CoreDNS down

A failed CoreDNS deployment means no DNS resolution at all, including for StatefulSet Pods. The Pods become “Running but unreachable.” The StatefulSet status is healthy but the workload inside the Pods is failing.

Wrong namespace

The StatefulSet is in prod but the Service is in data. The DNS is namespaced. Cross-namespace resolution requires a suffix or an explicit FQDN.

Why Deployments cannot do this

A Deployment’s Pods have names like web-7c8d9b1f8-abcd. The hash suffix changes when the Pod template changes; the Pod’s identity changes. There is no stable ordinal; there is no per-Pod DNS record. The Service routes to all matching Pods through kube-proxy load-balancing; individual Pods are not addressable by name.

A Deployment is correct when clients connect to “any Pod.” A StatefulSet is correct when clients need a specific Pod.

Quiz

Production discipline

• The headless Service is part of the StatefulSet contract. A change to its name, selector, or namespace is a StatefulSet-breaking change.
• Verify DNS after every cluster DNS change. CoreDNS upgrades and ConfigMap changes can break headless Services silently; smoke tests that resolve postgres-0.postgres-h in a fresh Pod catch this.
• Document the hostname-to-role mapping. A 3-replica PostgreSQL StatefulSet has postgres-0 as the primary by convention; the bootstrap Job that promotes it must be discovered by the operator.
• Don’t use a StatefulSet for “any Pod” addressing. A regular Service is sufficient; the headless Service is for specific-Pod addressing.
• Treat the StatefulSet’s hostname as the broker identity. A backup tool that captures the data directory by ordinal is restoring the right replica; one that captures it by IP is restoring whatever happens to be on that IP.

Stable identity is what StatefulSets exist to provide. The headless Service is the implementation. Operators who run StatefulSets verify the DNS contract, not just the Pod status.