DNS for Services — the cluster DNS, the records, and the search path

DNS for Services is the standard discovery pattern in Kubernetes. The cluster DNS service (CoreDNS) serves A, AAAA, and SRV records for every Service. The Pod has a DNS policy that determines the search path and the upstream resolvers. This lesson walks the DNS records, the Pod’s DNS policy, and the operational discipline.

The DNS records

The cluster DNS service serves the following records for a Service:

# Substitute your own value before running:
POD=web-5f9c7d8b6c-2xk9p

kubectl exec "$POD" -- nslookup billing.prod-app.svc.cluster.local

Name:    billing.prod-app.svc.cluster.local
Address: 10.96.0.10

This is an A record. The DNS service also serves:

• AAAA (IPv6) records when the cluster is dual-stack.
• SRV records for the ports: _http._tcp.billing.prod-app.svc.cluster.local.
• PTR records for reverse DNS.

The DNS records are automatically maintained by the DNS service. When a Service is created, the DNS service adds the records; when the Service is deleted, the DNS service removes the records.

sequenceDiagram
    autonumber
    participant C as Pod
    participant DNS as CoreDNS
    participant API as Kubernetes API
    DNS->>API: watch Services
    API-->>DNS: Service billing added
    DNS->>DNS: serve A record 10.96.0.10
    C->>DNS: billing.prod-app.svc.cluster.local
    DNS-->>C: 10.96.0.10

The DNS service is the single source of truth for the cluster’s DNS records. The record is generated from the Service’s spec.

The SRV record

The SRV record for a Service includes the port:

# Substitute your own value before running:
POD=web-5f9c7d8b6c-2xk9p

kubectl exec "$POD" -- nslookup -query=srv _http._tcp.billing.prod-app.svc.cluster.local

_http._tcp.billing.prod-app.svc.cluster.local
    service = 10 50 80 billing.prod-app.svc.cluster.local

The format is priority weight port target. The priority is 10, the weight is 50, the port is 80, the target is the Service’s DNS name.

The SRV record is useful for clients that support SRV records (e.g., Consul, some databases). Most clients use the A record.

The Pod DNS policy

The Pod has a DNS policy that determines how DNS queries are resolved:

apiVersion: v1
kind: Pod
metadata:
  name: my-app
spec:
  dnsPolicy: ClusterFirst
  containers:
    - name: app
      image: my-app

The DNS policies:

• Default: inherits the node’s DNS policy.
• ClusterFirst: uses the cluster DNS service for cluster-local names; forwards other queries to the upstream.
• ClusterFirstWithHostNet: uses the cluster DNS service for cluster-local names, even for Pods with hostNetwork.
• Default: uses the node’s resolv.conf.

The ClusterFirst policy is the default for Pods. The cluster operator must verify the policy is correct.

The search path

The Pod’s resolv.conf contains the search path:

# Substitute your own value before running:
POD=web-5f9c7d8b6c-2xk9p

kubectl exec "$POD" -- cat /etc/resolv.conf

nameserver 10.96.0.10
search prod-app.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

The search path is generated by the kubelet from the Pod’s DNS policy and the cluster’s DNS domain. The ndots:5 option is the threshold for the search path: queries with fewer than 5 dots are searched first.

The ndots:5 value is the source of a common performance issue: queries with fewer than 5 dots (e.g., billing) are searched against the search path first, then the absolute name. The cluster operator can lower the ndots to improve performance.

The nodelocal DNS cache

The nodelocal DNS cache is a DaemonSet that runs a DNS cache on every node. The cache intercepts DNS queries from Pods and forwards to the cluster DNS service.

flowchart LR
    A[Pod] -->|DNS query| B[nodelocal cache]
    B -->|cache miss| C[CoreDNS]
    C -->|answer| B
    B -->|answer| A
    B -->|cache hit| A

The nodelocal DNS cache reduces the latency of DNS queries and reduces the load on the cluster DNS service. The cache is a performance optimization; the cluster operator must verify the cache is healthy.

The upstream resolvers

The cluster DNS service forwards queries for non-cluster-local names to the upstream resolvers. The upstream is configured in the cluster DNS service’s Corefile:

.:53 {
    errors
    health
    kubernetes cluster.local in-addr.arpa ip6.arpa {
        pods insecure
        fallthrough in-addr.arpa ip6.arpa
    }
    forward . /etc/resolv.conf
    cache 30
    loop
    reload
    loadbalance
}

The forward . /etc/resolv.conf line forwards non-cluster-local queries to the upstream resolvers in the node’s resolv.conf. The cluster operator can configure the upstream resolvers to use a specific DNS service.

The failure modes

The DNS for Services’ failure modes:

• DNS service down: the cluster DNS service is unavailable. The fix is to verify the DNS service’s health.
• DNS records stale: the records are not updated in time. The fix is to verify the DNS controller’s cache.
• DNS policy misconfigured: the Pod uses the Default policy. The fix is to set the ClusterFirst policy.
• Search path wrong: the Pod’s search path is incorrect. The fix is to verify the DNS policy.
• Upstream resolver down: the upstream DNS is unavailable. The fix is to verify the upstream.
• NDOTS too high: the queries with fewer than 5 dots are slow. The fix is to lower the ndots.

The operational discipline

The DNS for Services’ operational discipline:

• Use ClusterFirst for all Pods. The Default policy is a fallback.
• Audit the DNS policy. The cluster operator must enumerate every Pod’s DNS policy.
• Monitor the DNS service’s health. The DNS service’s latency is a leading indicator.
• Test the DNS in staging. The DNS must work for the workload.
• Document the DNS configuration. The DNS is the cluster’s discovery primitive; the documentation is the reference.
• Plan the DNS’s evolution. The DNS service must scale with the cluster.

Quiz

Production discipline

• Use ClusterFirst for all Pods. The Default policy is a fallback.
• Audit the DNS policy. The cluster operator must enumerate every Pod’s DNS policy.
• Monitor the DNS service’s health. The DNS service’s latency is a leading indicator.
• Test the DNS in staging. The DNS must work for the workload.
• Document the DNS configuration. The DNS is the cluster’s discovery primitive; the documentation is the reference.
• Plan the DNS’s evolution. The DNS service must scale with the cluster.
• Tune the ndots. The ndots:5 default is a performance issue for short queries.
• Use the nodelocal DNS cache for performance. The cache reduces latency and load on the cluster DNS.