DNS → Service → EndpointSlice → Pod IP → application port — the canonical flow

The canonical flow is DNS → Service → EndpointSlice → Pod IP → application port. The diagnostic is the layer that fails. The discipline is to walk the flow in order, from the application’s request to the cluster’s network. The flow is the network’s hypothesis; the diagnostic is the operator’s test.

The five layers

The canonical flow has five layers:

1. DNS. The application resolves the Service’s DNS name to the Service’s ClusterIP.
2. Service. The application sends the request to the Service’s ClusterIP; the Service routes to the Pod IP.
3. EndpointSlice. The Service’s EndpointSlice lists the Pod IPs that the Service routes to.
4. Pod IP. The request reaches the Pod’s IP address.
5. Application port. The Pod’s application listens on the target port.

flowchart LR
    A[Application] --> B[DNS lookup]
    B --> C[Service ClusterIP]
    C --> D[EndpointSlice]
    D --> E[Pod IP]
    E --> F[Application port]
    F --> G[Workload response]

The five layers are the diagnostic stack. The operator who walks the flow in order is the operator who finds the failure mode.

Layer 1: DNS

The DNS layer is the first layer. The application calls getent hosts billing.prod.svc.cluster.local or its language equivalent. The DNS resolver queries the CoreDNS Pod in the kube-system namespace. The CoreDNS Pod returns the Service’s ClusterIP.

kubectl exec -it billing-7d8f-abcde -n prod -- nslookup billing.prod.svc.cluster.local

A real, broken DNS response:

Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.svc.cluster.local

** server can't find billing.prod.svc.cluster.local: NXDOMAIN

The NXDOMAIN is the cluster’s hypothesis: the DNS resolver could not find the Service’s DNS record. The cause is one of:

• The Service does not exist in the namespace.
• The CoreDNS Pod is failing.
• The DNS query is wrong (typo, wrong namespace).

Layer 2: Service

The Service layer is the second layer. The Service’s spec.clusterIP is a virtual IP that the kube-proxy watches. The kube-proxy programs iptables (or IPVS) rules on every node to route the ClusterIP to the Pod IPs.

kubectl get service billing -n prod -o yaml

A real Service spec:

apiVersion: v1
kind: Service
metadata:
  name: billing
  namespace: prod
spec:
  selector:
    app: billing
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP
  clusterIP: 10.96.45.123

The diagnostic is the spec.selector and the spec.ports. A Service with an empty selector has no endpoints; a Service with a wrong port does not route to the Pod’s port.

Layer 3: EndpointSlice

The EndpointSlice layer is the third layer. The EndpointSlice controller watches all Pods and populates the Service’s EndpointSlice with the Pods that match the Service’s selector.

kubectl get endpointslices -n prod -l kubernetes.io/service-name=billing

A real EndpointSlice:

addressType: IPv4
endpoints:
- addresses:
  - 10.244.5.23
  - 10.244.7.10
  conditions:
    ready: true
  targetRef:
    kind: Pod
    name: billing-7d8f-abcde
- addresses:
  - 10.244.5.24
  - 10.244.7.11
  conditions:
    ready: true
  targetRef:
    kind: Pod
    name: billing-7d8f-def01
ports:
- port: 8080
  protocol: TCP

The diagnostic is the endpoints array. An empty array means no Pods match the selector. A Pod with ready: false is excluded from the Service’s routing.

Layer 4: Pod IP

The Pod IP layer is the fourth layer. The Pod IP is the IP address of the Pod’s network namespace. The CNI plugin assigns the IP from the Pod CIDR.

kubectl get pod billing-7d8f-abcde -n prod -o jsonpath='{.status.podIP}'

The output is the Pod’s IP. The diagnostic is the IP address and the route from the source Pod to the destination Pod.

The route is the CNI’s responsibility. The CNI programs the Pod’s routing table and the node’s routing table; the route is the path that the request takes.

Layer 5: Application port

The application port layer is the fifth layer. The Pod’s application listens on a port (e.g., 8080). The Service’s targetPort is the port that the Service routes to.

kubectl exec -it billing-7d8f-abcde -n prod -- ss -tlnp

A real application listening:

State      Recv-Q  Send-Q   Local Address:Port   Peer Address:Port
LISTEN     0       128              *:8080              *:*

The diagnostic is the listening port. An application that is not listening on the target port is one that the Service cannot route to.

The diagnostic command sequence

The canonical diagnostic command sequence:

# Substitute your own values before running:
NS=prod
SVC=billing
SOURCE_POD=web-5f9c7d8b6c-2xk9p      # the client Pod that cannot reach the Service
BACKEND_POD=billing-7d8f-abcde       # a Pod the Service is meant to select

# Layer 1: DNS
kubectl exec -it "$SOURCE_POD" -n "$NS" -- nslookup "$SVC.$NS.svc.cluster.local"

# Layer 2: Service
kubectl get service "$SVC" -n "$NS" -o yaml

# Layer 3: EndpointSlice
kubectl get endpointslices -n "$NS" -l "kubernetes.io/service-name=$SVC"

# Layer 4: Pod IP
kubectl get pod "$BACKEND_POD" -n "$NS" -o jsonpath='{.status.podIP}'

# Layer 5: Application port
kubectl exec -it "$BACKEND_POD" -n "$NS" -- ss -tlnp

The command sequence is the canonical flow. Each layer that answers correctly eliminates itself and everything before it; the first that does not is the failure mode.

Production discipline

The canonical flow is the cluster’s hypothesis. The discipline is to walk the flow in order, gather evidence at each layer, identify the failure mode. The flow is the same for every Service failure; the depth scales with the complexity.

• Five layers. DNS, Service, EndpointSlice, Pod IP, application port.
• Read-only diagnostic. The diagnostic commands do not change the cluster’s state.

Quiz