DNS and service discovery — CoreDNS, the resolv.conf, and the FQDN

DNS is the cluster’s service discovery. CoreDNS is the cluster’s DNS server; the Pod’s resolv.conf points to the DNS service; the FQDN is the Service’s name in the cluster. This lesson walks the DNS resolution, the CoreDNS plugin, the FQDN format, and the operational patterns.

The DNS resolution in a Pod

The DNS resolution in a Pod:

sequenceDiagram
    autonumber
    participant App as Application
    participant R as resolv.conf
    participant CD as CoreDNS
    participant API as API server

    App->>R: nslookup billing.prod-app.svc.cluster.local
    R->>R: search path: prod-app.svc.cluster.local
    R->>CD: query CoreDNS
    CD->>API: watch Service
    API-->>CD: Service billing in prod-app
    CD->>CD: build FQDN
    CD-->>R: response: 10.96.0.1
    R-->>App: 10.96.0.1

The DNS resolution is the cluster’s mechanism for discovering the Service’s IP. The Pod’s application resolves the Service’s name; the CoreDNS returns the Service’s IP.

The Pod’s resolv.conf

The Pod’s resolv.conf:

cat /etc/resolv.conf

search prod-app.svc.cluster.local svc.cluster.local cluster.local
nameserver 10.96.0.10
options ndots:5

The resolv.conf’s fields:

• search: the search path. The search path is the list of domains to append to the query.
• nameserver: the DNS server. The DNS server is the CoreDNS service.
• options ndots:5: the number of dots in the query before the search path is applied.

The Pod’s resolv.conf is the cluster’s DNS configuration. The kubelet configures the resolv.conf on the Pod’s startup.

The FQDN format

The FQDN format is the Service’s name in the cluster:

<service-name>.<namespace>.svc.cluster.local

The FQDN is the Service’s fully qualified domain name. The FQDN is composed of:

• <service-name>: the Service’s name.
• <namespace>: the Namespace.
• svc.cluster.local: the cluster’s DNS suffix.

The FQDN is the cluster’s DNS entry. The Pod’s application uses the FQDN to resolve the Service.

The search path

The search path is the list of domains appended to the query:

search prod-app.svc.cluster.local svc.cluster.local cluster.local

The search path is the cluster’s mechanism for shortening the FQDN. The Pod’s application can use the short name (billing) and the search path will append the domain.

The search path is configured by the kubelet. The kubelet reads the namespace from the Pod’s spec and configures the search path.

CoreDNS

CoreDNS is the cluster’s DNS server. CoreDNS is a single binary that runs as a Deployment in the cluster.

kubectl get pods -n kube-system -l k8s-app=kube-dns

NAME                       READY   STATUS    RESTARTS   AGE
coredns-7d5c5b9b9d-abcde   1/1     Running   0          5d
coredns-7d5c5b9b9d-fghij   1/1     Running   0          5d

CoreDNS is the cluster’s DNS server. The CoreDNS resolves the Service names to the Service IPs.

CoreDNS’s configuration:

apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
        }
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }

The Corefile is the CoreDNS configuration. The Corefile specifies the plugins, the forwarders, and the cache.

The Kubernetes plugin

The Kubernetes plugin is the CoreDNS plugin that watches the cluster’s Services and Pods. The plugin builds the DNS records for the Services and Pods.

The Kubernetes plugin’s fields:

• cluster.local: the cluster’s DNS suffix.
• in-addr.arpa: the reverse DNS suffix.
• ip6.arpa: the IPv6 reverse DNS suffix.
• pods insecure: the Pod’s DNS records.
• fallthrough: the fallthrough to the upstream DNS.

The Kubernetes plugin is the cluster’s DNS plugin. The plugin is the cluster’s mechanism for resolving the Service names.

The forward plugin

The forward plugin is the CoreDNS plugin that forwards the DNS queries to the upstream DNS servers:

forward . /etc/resolv.conf

The forward plugin forwards the DNS queries to the upstream DNS servers. The plugin is the cluster’s mechanism for resolving the external DNS names.

The forward plugin is configured by the CoreDNS config file. The plugin reads the upstream DNS servers from the cluster’s /etc/resolv.conf.

The cache plugin

The cache plugin is the CoreDNS plugin that caches the DNS responses.

cache 30

The cache plugin caches the DNS responses for 30 seconds. The cache is the cluster’s mechanism for reducing the DNS latency.

The cache plugin is the cluster’s mechanism for performance. The cache reduces the DNS latency; the cache reduces the upstream DNS load.

The nodelocal DNS cache

The nodelocal DNS cache is the cluster’s mechanism for caching the DNS responses on every node.

kubectl get pods -n kube-system -l k8s-app=node-local-dns

The nodelocal DNS cache is a DaemonSet that runs on every node. The nodelocal DNS cache is the cluster’s mechanism for reducing the DNS latency on the node.

The nodelocal DNS cache is faster than the CoreDNS cache because the cache is on the node. The cache reduces the DNS latency by 10x.

The nodelocal DNS cache is the cluster’s mechanism for performance. The production rule is to enable the nodelocal DNS cache for large clusters.

The Pod’s DNS policy

The Pod’s DNS policy:

spec:
  dnsPolicy: ClusterFirst

The Pod’s DNS policy is the cluster’s mechanism for configuring the DNS. The policies:

• ClusterFirst: the Pod’s DNS is configured to use the cluster’s DNS. The default for Pods.
• Default: the Pod’s DNS is configured to use the node’s DNS. The legacy default.
• ClusterFirstWithHostNet: the Pod’s DNS is configured to use the cluster’s DNS, but the Pod uses the host network.
• None: the Pod’s DNS is configured manually.

The Pod’s DNS policy is the cluster’s mechanism for DNS configuration. The production rule is to use ClusterFirst for all Pods.

The Pod’s DNS configuration

The Pod’s DNS configuration:

spec:
  dnsPolicy: ClusterFirst
  dnsConfig:
    nameservers:
      - 10.96.0.10
    searches:
      - prod-app.svc.cluster.local
      - svc.cluster.local
      - cluster.local
    options:
      - name: ndots
        value: "5"

The Pod’s DNS configuration is the cluster’s mechanism for overriding the DNS. The configuration is the Pod’s DNS configuration.

The DNS’s troubleshooting

The DNS’s troubleshooting:

# Substitute your own value before running:
POD=checkout-7d4f8c9b5-x2kqn

# Check the Pod's resolv.conf
kubectl exec "$POD" -- cat /etc/resolv.conf

# Check the DNS resolution
kubectl exec "$POD" -- nslookup billing.prod-app.svc.cluster.local

# Check the CoreDNS logs
kubectl logs -n kube-system -l k8s-app=kube-dns

The DNS’s troubleshooting is the operator’s diagnostic. The fix is to investigate the CoreDNS’s logs and the Pod’s resolv.conf.

Quiz

Production discipline

• DNS is the cluster’s service discovery. The CoreDNS is the cluster’s DNS server.
• The Pod’s resolv.conf is the cluster’s DNS configuration. The kubelet configures the resolv.conf on the Pod’s startup.
• The FQDN is the Service’s name in the cluster. The FQDN is the cluster’s DNS entry.
• The nodelocal DNS cache is the cluster’s mechanism for performance. The cache reduces the DNS latency.
• Audit the DNS at every release. The CoreDNS’s configuration should be version-controlled; the audit catches the failures.
• Monitor the DNS’s metrics. The CoreDNS’s metrics expose the DNS health; the operator should alert on the threshold.
• Test the DNS in non-production. A staging cluster that mirrors production is the right place to test the DNS.
• Document the DNS’s design. The DNS is the cluster’s service discovery; the documentation is the cluster’s networking reference.