CoreDNS troubleshooting — diagnosing the cluster DNS

CoreDNS troubleshooting requires checking the Pods, the Corefile, the upstreams, the cache, and the metrics. The diagnostic flow is: Pods -> Corefile -> Kubernetes plugin -> forward plugin -> cache -> metrics. This lesson walks the diagnostic flow and the operational discipline.

The diagnostic flow

The diagnostic flow for CoreDNS:

flowchart TD
    A[DNS resolution failed] --> B{CoreDNS Pods running?}
    B -->|No| C[Restart Pods]
    B -->|Yes| D{Corefile valid?}
    D -->|No| E[Fix Corefile]
    D -->|Yes| F{Kubernetes plugin watching?}
    F -->|No| G[Restart Pods]
    F -->|Yes| H{Upstream reachable?}
    H -->|No| I[Check network]
    H -->|Yes| J{Cache stale?}
    J -->|Yes| K[Reduce TTL]
    J -->|No| L[Check metrics]

The diagnostic flow is the operator’s guide. The cluster operator must understand the flow.

Step 1: Check the Pods

The first step is to verify the CoreDNS Pods are running:

kubectl get pods -n kube-system -l k8s-app=kube-dns

NAME                       READY   STATUS    RESTARTS   AGE
coredns-7d4f-abc           1/1     Running   0          12d
coredns-7d4f-def           1/1     Running   0          12d

The Pods must be Running and Ready. If the Pods are not Running, the fix is to investigate the Pod’s status.

Step 2: Check the Corefile

The second step is to verify the Corefile:

kubectl -n kube-system get configmap coredns -o yaml

The Corefile must be valid. A syntax error produces a CoreDNS Pod that does not start.

kubectl -n kube-system logs coredns-xxx

The logs show the Corefile errors. The fix is to correct the Corefile.

Step 3: Check the Kubernetes plugin

The third step is to verify the Kubernetes plugin is watching the API:

kubectl -n kube-system logs coredns-xxx | grep "kubernetes"

The logs show the plugin’s status. If the plugin is not watching, the fix is to restart the Pods.

Step 4: Check the upstreams

The fourth step is to verify the upstreams are reachable:

kubectl exec -n kube-system coredns-xxx -- nslookup example.com

If the upstream is reachable, the query is forwarded to the upstream. If the upstream is unreachable, the query fails.

Step 5: Check the cache

The fifth step is to verify the cache:

# The address comes from the running CoreDNS Pod:
COREDNS_IP=$(kubectl -n kube-system get pod -l k8s-app=kube-dns \
  -o jsonpath='{.items[0].status.podIP}')

curl "http://$COREDNS_IP:9153/metrics" | grep cache

coredns_cache_hits_total{server="dns://:53",zone="."} 5000
coredns_cache_misses_total{server="dns://:53",zone="."} 1000

The cache hit ratio is hits / (hits + misses). A low ratio is a sign of an undersized cache or a high churn rate.

Step 6: Check the metrics

The sixth step is to verify the metrics:

# The address comes from the running CoreDNS Pod:
COREDNS_IP=$(kubectl -n kube-system get pod -l k8s-app=kube-dns \
  -o jsonpath='{.items[0].status.podIP}')

curl "http://$COREDNS_IP:9153/metrics"

The metrics show the request rate, the response rate, the cache hit ratio, and the upstream latency. The cluster operator monitors these metrics.

The common failure modes

The common failure modes of CoreDNS:

• Pods down: the CoreDNS Pods are not running. The fix is to restart the Pods.
• Corefile invalid: the Corefile has a syntax error. The fix is to correct the Corefile.
• Kubernetes plugin not watching: the plugin’s watch loop is broken. The fix is to restart the Pods.
• Upstream unreachable: the upstream resolvers are unavailable. The fix is to verify the network.
• Cache stale: the cache returns stale records. The fix is to reduce the TTL.
• Memory pressure: the CoreDNS Pods are evicted. The fix is to increase the memory.

The operational discipline

The CoreDNS troubleshooting’s operational discipline:

• Document the diagnostic flow. The flow is the cluster’s operational reference.
• Run the diagnostics regularly. The diagnostics are the leading indicator.
• Monitor the metrics. The metrics are the leading indicator of degradation.
• Test the recovery in staging. The recovery must be tested before production.
• Document the troubleshooting. The troubleshooting is the cluster’s operational reference.
• Train the operations team on the diagnostics. The diagnostics are the team’s tools.

Quiz

Production discipline

• Document the diagnostic flow. The flow is the cluster’s operational reference.
• Run the diagnostics regularly. The diagnostics are the leading indicator.
• Monitor the metrics. The metrics are the leading indicator of degradation.
• Test the recovery in staging. The recovery must be tested before production.
• Document the troubleshooting. The troubleshooting is the cluster’s operational reference.
• Train the operations team on the diagnostics. The diagnostics are the team’s tools.
• Set up alerts on the metrics. The alerts are the leading indicator of outages.
• Document the troubleshooting in the runbook. The runbook is the cluster’s operational reference.
• Plan the troubleshooting’s evolution. The troubleshooting is the cluster’s operational reference; the operator must keep it current.