DNS troubleshooting — diagnosing the cluster DNS

DNS troubleshooting verifies the CoreDNS Pods, the upstream, and the cache. The diagnostic tools are nslookup, dig, kubectl logs. This lesson walks the diagnostic flow, the failure modes, and the operational discipline.

The diagnostic flow

The diagnostic flow for DNS resolution:

flowchart TD
    A[DNS failed] --> B{nslookup works?}
    B -->|No| C[Check CoreDNS Pods]
    B -->|Yes| D{cluster.local works?}
    D -->|No| E[Check Corefile]
    D -->|Yes| F{upstream works?}
    F -->|No| G[Check upstream]
    F -->|Yes| H[Check cache]

The diagnostic flow is the operator’s guide. The cluster operator must follow the flow.

Step 1: Verify the CoreDNS Pods

The first step is to verify the CoreDNS Pods:

kubectl get pods -n kube-system -l k8s-app=kube-dns

The Pods must be Running and Ready.

Step 2: Verify the Corefile

The second step is to verify the Corefile:

kubectl -n kube-system get configmap coredns -o yaml

The Corefile must be valid. A syntax error produces a CoreDNS Pod that does not start.

Step 3: Verify the upstream

The third step is to verify the upstream:

kubectl exec -n kube-system coredns-xxx -- nslookup example.com

The upstream must be reachable. If the upstream is unreachable, the query fails.

Step 4: Verify the search path

The fourth step is to verify the search path:

# Substitute your own value before running:
POD=web-5f9c7d8b6c-2xk9p

kubectl exec "$POD" -- cat /etc/resolv.conf

nameserver 10.96.0.10
search prod-app.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

The search path must be correct. The ndots:5 option is the threshold for the search path.

Step 5: Verify the cache

The fifth step is to verify the cache:

# The metrics endpoint is on the CoreDNS Pod itself, so take its IP:
COREDNS_IP=$(kubectl get pod -n kube-system -l k8s-app=kube-dns \
  -o jsonpath='{.items[0].status.podIP}')

curl http://"$COREDNS_IP":9153/metrics | grep cache

The cache hit ratio is hits / (hits + misses). A low ratio is a sign of an undersized cache.

The failure modes

The DNS resolution’s failure modes:

• CoreDNS down: the DNS service is unavailable. The fix is to restart the Pods.
• Corefile invalid: the Corefile has a syntax error. The fix is to verify the Corefile.
• Upstream unreachable: the upstream resolvers are unavailable. The fix is to verify the upstream.
• Plugin crashed: a plugin is not running. The fix is to restart the Pods.
• ndots issue: the ndots is too high. The fix is to lower the ndots.
• Search path wrong: the search path is incorrect. The fix is to verify the DNS policy.

The operational discipline

The DNS troubleshooting’s operational discipline:

• Document the diagnostic flow. The flow is the cluster’s operational reference.
• Use the diagnostic tools. The tools are the operator’s primary tools.
• Follow the checklist. The checklist is the operator’s guide.
• Document the troubleshooting. The troubleshooting is the cluster’s operational reference.
• Train the operations team on the diagnostics. The diagnostics are the team’s tools.
• Set up alerts on the CoreDNS metrics. The alerts are the leading indicator of outages.

Quiz

Production discipline

• The diagnostic flow is systematic. The cluster operator must follow the flow.
• Document the diagnostic flow. The flow is the cluster’s operational reference.
• Use the diagnostic tools. The tools are the operator’s primary tools.
• Follow the checklist. The checklist is the operator’s guide.
• Document the troubleshooting. The troubleshooting is the cluster’s operational reference.
• Train the operations team on the diagnostics. The diagnostics are the team’s tools.
• Set up alerts on the CoreDNS metrics. The alerts are the leading indicator of outages.
• Document the troubleshooting in the runbook. The runbook is the cluster’s operational reference.