Headless services and Service topology — direct Pod addressing

A headless service is a Service with clusterIP: None. The Service has no ClusterIP; the DNS resolver returns the Pod IPs directly. The use cases are StatefulSets where the client needs to address each Pod individually, and peer-to-peer protocols where every Pod is a peer. This lesson walks the mechanism, the DNS records, and the operational discipline.

The headless service manifest

apiVersion: v1
kind: Service
metadata:
  name: billing
spec:
  clusterIP: None
  selector:
    app: billing
  ports:
    - name: http
      port: 80
      targetPort: 8080

The clusterIP: None field marks the Service as headless. The selector must match the Pods; the DNS resolver returns the Pod IPs.

The DNS records

A headless service’s DNS records are different from a ClusterIP service:

• ClusterIP: the DNS returns a single A record with the ClusterIP. The kube-proxy DNATs to a Pod.
• Headless: the DNS returns multiple A records, one per Pod IP. The client connects to a Pod directly.

# A pod in the cluster to run the lookup from:
POD=client-5f9c7d8b6c-2xk9p

kubectl exec "$POD" -- nslookup billing.prod-app.svc.cluster.local

For a ClusterIP service:

Name:    billing.prod-app.svc.cluster.local
Address: 10.96.0.10

For a headless service:

Name:    billing.prod-app.svc.cluster.local
Address: 10.244.1.5
Address: 10.244.1.6
Address: 10.244.2.5

The headless service returns the Pod IPs; the client connects to one of them directly.

The traffic flow

The traffic flow for a headless service:

sequenceDiagram
    autonumber
    participant C as Client
    participant DNS as CoreDNS
    participant P1 as Pod 1
    participant P2 as Pod 2
    C->>DNS: billing.prod-app.svc.cluster.local
    DNS-->>C: 10.244.1.5, 10.244.1.6, 10.244.2.5
    C->>P1: TCP 10.244.1.5:8080
    P1->>P1: handle request
    P1->>C: response

The client resolves the Service’s name to a list of Pod IPs. The client chooses one (randomly, or by some client-side logic) and connects to it directly. The kube-proxy is not involved.

The use cases

The headless service’s use cases:

• StatefulSets: the client needs to address each Pod individually (e.g., database replicas, message brokers).
• Peer-to-peer protocols: every Pod is a peer (e.g., Cassandra, Kafka, ZooKeeper).
• DNS-based service discovery: the application does its own load balancing based on the Pod IPs.

The headless service is the right choice for applications that need direct Pod addressing.

The Service topology

The Service supports a topologyKeys field that restricts the traffic to Pods in the same zone, region, or other topology:

apiVersion: v1
kind: Service
metadata:
  name: billing
spec:
  topologyKeys:
    - "topology.kubernetes.io/zone"
    - "*"
  selector:
    app: billing
  ports:
    - port: 80
      targetPort: 8080

The topologyKeys field is used for headless services to prefer local Pods. The first key in the list is the preferred topology; the fallback is "*" (any Pod).

The headless service and StatefulSets

StatefulSets use headless services to provide a stable DNS name for each Pod:

apiVersion: v1
kind: StatefulSet
metadata:
  name: db
spec:
  serviceName: db
  replicas: 3
  selector:
    matchLabels:
      app: db
  template:
    metadata:
      labels:
        app: db
    spec:
      containers:
        - name: db
          image: postgres:16

The Pod’s DNS name is db-0.db.prod-app.svc.cluster.local, db-1.db.prod-app.svc.cluster.local, etc. The DNS resolver returns the Pod IP for each name.

The failure modes

The headless service’s failure modes:

• Selector mismatch: the headless service has no Pod IPs. The fix is to verify the selector.
• Pod not Ready: the Pod is not in the DNS RR. The fix is to verify the readiness probe.
• DNS resolver issue: the DNS resolver does not return the Pod IPs. The fix is to verify the CoreDNS configuration.
• Client load balancing: the client does not distribute the traffic evenly. The fix is to use a client-side load balancer.

The operational discipline

The headless service’s operational discipline:

• Document the headless service’s selector. The cluster operator must understand which Pods the Service returns.
• Audit the headless service’s DNS records. The DNS records are the source of truth.
• Test the headless service in staging. The client must connect to the Pods correctly.
• Monitor the headless service’s traffic. The traffic is direct; the kube-proxy metrics are not available.
• Document the headless service’s design. The headless service is the cluster’s DNS abstraction for direct Pod addressing.

Quiz

Production discipline

• Headless services return Pod IPs directly. The kube-proxy is not involved.
• Use headless services for StatefulSets. The DNS names are stable per Pod.
• Document the headless service’s selector. The cluster operator must understand which Pods the Service returns.
• Audit the headless service’s DNS records. The DNS records are the source of truth.
• Test the headless service in staging. The client must connect to the Pods correctly.
• Monitor the headless service’s traffic. The traffic is direct; the kube-proxy metrics are not available.
• Document the headless service’s design. The headless service is the cluster’s DNS abstraction for direct Pod addressing.
• Verify the client’s load balancing. The client is responsible for load balancing; the headless service does not load balance.