DNS troubleshooting flow — the diagnostic methodology

DNS troubleshooting in Kubernetes follows a diagnostic flow. This lesson walks the flow, the common failures, the tools, and the discipline.

The diagnostic flow

flowchart TD
    A[DNS not resolving] --> B{Pod can reach CoreDNS?}
    B -->|No| C["Check Pod network, service IP"]
    B -->|Yes| D{CoreDNS resolves cluster?}
    D -->|No| E["Check CoreDNS logs, Corefile"]
    D -->|Yes| F{Upstream resolves?}
    F -->|No| G[Check upstream connectivity]
    F -->|Yes| H{nodelocaldns works?}
    H -->|No| I[Check nodelocaldns config]
    H -->|Yes| J{Stub domains work?}
    J -->|No| K[Check stub config]
    J -->|Yes| L[DNS healthy]

The diagnostic flow:

1. Can the Pod reach CoreDNS? (10.96.0.10).
2. Does CoreDNS resolve cluster names? (kubernetes.default.svc.cluster.local).
3. Does the upstream resolve external names? (example.com).
4. Does nodelocaldns work? (169.254.20.10).
5. Do stub domains work? (.corp).

Each step has a specific failure mode.

Step 1: Pod can reach CoreDNS

kubectl exec -it myapp-pod -- nslookup kubernetes.default

If this fails, the Pod cannot reach CoreDNS. Possible causes:

• Pod network misconfigured. CNI not running on the node.
• CoreDNS Service IP unreachable. NetworkPolicy blocking DNS traffic.
• CoreDNS Pods are not Running. kubectl get pods -n kube-system -l k8s-app=kube-dns.

Step 2: CoreDNS resolves cluster names

kubectl exec -it myapp-pod -- nslookup kubernetes.default.svc.cluster.local

If this fails, CoreDNS is not resolving cluster names. Possible causes:

• Corefile error. Missing brace; plugin misconfiguration.
• kubernetes plugin not configured. Service records are not generated.
• Service doesn’t exist. The Service has no Endpoints.

Step 3: Upstream resolves

kubectl exec -it myapp-pod -- nslookup example.com

If this fails, CoreDNS cannot reach upstream. Possible causes:

• Forward misconfigured. Wrong upstream IP.
• Network egress blocked. NetworkPolicy blocking port 53.
• Upstream down. The upstream DNS is unreachable.

Step 4: nodelocaldns

kubectl exec -it myapp-pod -- nslookup kubernetes.default 169.254.20.10

If this fails, nodelocaldns is not working. Possible causes:

• nodelocaldns not installed. DaemonSet missing.
• nodelocaldns not running. Pods are CrashLoopBackOff.
• Pod dnsConfig not configured. Pod is querying CoreDNS, not nodelocaldns.

Step 5: Stub domains

kubectl exec -it myapp-pod -- nslookup billing.corp

If this fails, stub domains are misconfigured. Possible causes:

• Corefile missing forward for .corp. CoreDNS doesn’t know to forward.
• Pod stubDomains not set. (If using Pod-level config.)
• Corporate DNS unreachable.

The tools

flowchart LR
    A[Tools] --> B[nslookup]
    A --> C[dig]
    A --> D[kubectl exec]
    A --> E[CoreDNS logs]
    A --> F[CoreDNS metrics]

The tools:

• nslookup. Simple query tool; available in most containers.
• dig. Detailed query tool; available in busybox and most distros.
• kubectl exec. Run tools inside a Pod.
• CoreDNS logs. kubectl logs -n kube-system -l k8s-app=kube-dns.
• CoreDNS metrics. coredns_dns_request_* on :9153.

The common failures

flowchart LR
    A[Common failures] --> B[Corefile syntax error]
    A --> C[Upstream timeout]
    A --> D[ndots misconfiguration]
    A --> E[Stub domain leak]
    A --> F[Cache stale]
    A --> G[Service missing]

The common failures:

• Corefile syntax error. A missing brace breaks CoreDNS startup.
• Upstream timeout. The upstream DNS is slow or unreachable.
• ndots misconfiguration. Too many search path queries.
• Stub domain leak. A zone that’s not stubbed goes to public DNS.
• Cache stale. Cached responses are too old.
• Service missing. The Service has no Endpoints.

Quiz

The operational discipline

DNS troubleshooting in production rests on five non-negotiable elements:

• Walk the diagnostic flow methodically. Each step has a specific failure mode.
• Use dig for detail. nslookup for quick checks, dig for thorough diagnosis.
• Check CoreDNS logs. Errors are visible.
• Test changes in staging. A misconfigured Corefile affects every Pod.
• Document the troubleshooting runbook. Common failures and fixes.

DNS troubleshooting is a chain. The discipline is to walk the chain methodically.