DNS outage and mitigations — the cluster wide failure

A DNS outage is the cluster’s DNS resolver failing. The mitigations are NodeLocal DNSCache, ndots:2, and a fallback to upstream DNS. The discipline is the same canonical flow extended with the cluster-wide impact.

The DNS outage

A DNS outage is the CoreDNS Pods failing to answer queries. The cluster’s DNS resolution is broken. Every Service resolution in the cluster is failing.

flowchart TD
    A[Pod query] --> B{CoreDNS up?}
    B -->|Yes| C[Resolved]
    B -->|No| D[NXDOMAIN]
    D --> E[Application fails]

The DNS outage is the cluster’s hypothesis. The diagnostic is the CoreDNS Pods and the ConfigMap.

The mitigations

The mitigations are:

1. NodeLocal DNSCache. A DaemonSet that runs a DNS cache on every node. The cache reduces the CoreDNS query rate and provides a fallback if the CoreDNS is unreachable.
2. ndots:2. The Pod’s dnsConfig sets ndots:2, reducing the query rate from 5x to 1x.
3. Fallback to upstream DNS. The Pod’s resolv.conf includes the upstream DNS as a fallback.
4. Service mesh DNS. A service mesh (Istio, Linkerd) can provide DNS resolution independent of CoreDNS.

flowchart TD
    A[Pod query] --> B[NodeLocal DNSCache]
    B --> C{Cached?}
    C -->|Yes| D[Cache response]
    C -->|No| E[CoreDNS]
    E --> F{Resolved?}
    F -->|Yes| G[Response]
    F -->|No| H[Upstream DNS]
    H --> I[Response]

The mitigations are the cluster’s DNS resilience.

NodeLocal DNSCache

NodeLocal DNSCache is a DaemonSet that runs a DNS cache on every node. The cache listens on the node’s IP address (e.g., 169.254.20.10) and forwards queries to the CoreDNS.

# Install NodeLocal DNSCache
kubectl apply -f https://k8s.io/examples/admin/dns/nodelocaldns.yaml

# Verify the install
kubectl get pods -n kube-system -l k8s-app=node-local-dns

The Pod’s resolv.conf is patched to use the NodeLocal DNSCache as the nameserver.

The diagnostic

The canonical diagnostic for a DNS outage:

# Substitute your own value before running:
SOURCE_POD=netshoot-6b8d9c7f45-q2wxr

# 1. Check the CoreDNS Pods
kubectl get pods -n kube-system -l k8s-app=kube-dns -o wide

# 2. Check the CoreDNS logs
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=200

# 3. Check the NodeLocal DNSCache (if installed)
kubectl get pods -n kube-system -l k8s-app=node-local-dns -o wide

# 4. Test the DNS from inside a Pod
kubectl exec -it "$SOURCE_POD" -- nslookup billing.prod.svc.cluster.local

# 5. Test the fallback to upstream DNS
kubectl exec -it "$SOURCE_POD" -- nslookup example.com

The diagnostic is the CoreDNS Pods, the NodeLocal DNSCache, the logs, and the tests.

The recovery

The recovery for a DNS outage:

# Option 1: Restart the CoreDNS
kubectl rollout restart deployment/coredns -n kube-system

# Option 2: Scale the CoreDNS
kubectl scale deployment/coredns -n kube-system --replicas=4

# Option 3: Fix the ConfigMap
kubectl edit configmap coredns -n kube-system

# Option 4: Reinstall NodeLocal DNSCache
kubectl rollout restart daemonset/node-local-dns -n kube-system

The recovery is the DNS restoration.

Production discipline

A DNS outage is the cluster’s hypothesis. The discipline is to have NodeLocal DNSCache, ndots:2, and a fallback to upstream DNS. The DNS is the cluster’s name resolution; the mitigations are the resilience.

• Set ndots:2. The lower ndots reduce the query rate.
• Have a fallback. The Pod’s resolv.conf includes the upstream DNS as a fallback.

Quiz