medium riskcluster affecting~30 min

Runbook: Troubleshoot CoreDNS

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Capture the CoreDNS Pod state: kubectl -n kube-system get pods -l k8s-app=kube-dns -o wide
· Capture the CoreDNS logs: kubectl -n kube-system logs -l k8s-app=kube-dns --tail=200 --timestamps
· Capture the CoreDNS ConfigMap: kubectl -n kube-system get configmap coredns -o yaml | tee /tmp/coredns-cm.yaml
· Capture the CoreDNS service endpoint: kubectl -n kube-system get svc kube-dns -o yaml and kubectl -n kube-system get endpointslices -l kubernetes.io/service-name=kube-dns
· Capture nodelocaldns state if installed: kubectl -n kube-system get pods -l k8s-app=node-local-dns
· Confirm a test Pod exists for DNS testing: kubectl run -it --rm --image=registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7 --restart=Never dnstest -- nslookup kubernetes.default

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Confirm CoreDNS is Running and Ready: kubectl -n kube-system get pods -l k8s-app=kube-dns
2Confirm the kube-dns Service has endpoints: kubectl -n kube-system get endpointslices -l kubernetes.io/service-name=kube-dns
3Capture the CoreDNS metrics: kubectl -n kube-system port-forward <coredns-pod> 9153:9153 & curl -s http://localhost:9153/metrics | grep -E "coredns_cache|coredns_dns"
4Classify the failure from the test query: NXDOMAIN, SERVFAIL, timeout, or unexpected answer
5For NXDOMAIN: check the search path and ndots in the test Pod (cat /etc/resolv.conf), and check the CoreDNS log for the upstream query
6For SERVFAIL: read the CoreDNS log for plugin errors (kubernetes, forward, cache, ready)
7For timeout: confirm the cluster DNS Service IP is reachable from the test Pod (kubectl exec dnstest -- nc -vz 10.96.0.10 53); check CoreDNS Pod CPU/memory
8For unexpected answer: confirm the ConfigMap kubernetes plugin is configured for the right cluster domain (default cluster.local)
9For upstream resolver issues: kubectl -n kube-system logs -l k8s-app=kube-dns --tail=200 for forward plugin errors
10Apply the smallest fix: correct the ConfigMap, increase replicas (scaling), restart the CoreDNS Pods to pick up a new ConfigMap, fix the upstream
11Verify from the test Pod: kubectl exec dnstest -- nslookup <service>.<ns>.svc.cluster.local and kubectl exec dnstest -- dig +short kubernetes.default

4 · Verification

Confirm the procedure actually fixed the problem.

✓kubectl -n kube-system get pods -l k8s-app=kube-dns reports every replica Running and Ready
✓kubectl -n kube-system get configmap coredns -o yaml parses correctly and the Corefile is valid (no syntax errors in the log)
✓From a test Pod, nslookup kubernetes.default returns the Service ClusterIP
✓From a test Pod, nslookup <service>.<ns>.svc.cluster.local returns the Service ClusterIP for the target Service
✓CoreDNS metrics show non-zero coredns_dns_responses_total and coredns_cache_hits_total
✓kubectl top pods -n kube-system -l k8s-app=kube-dns shows CPU and memory within limits
✓No SERVFAIL or plugin error in recent CoreDNS logs

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶If a ConfigMap change broke CoreDNS, kubectl apply the previous Corefile from Git and kubectl -n kube-system rollout restart deploy/coredns
↶If the upstream resolver change broke resolution, restore the previous forward block in the Corefile
↶If ndots or search path change broke resolution, restore the previous Pod DNSConfig
↶If scaling change (replicas up) caused resource pressure on nodes, scale down with explicit approval
↶Capture the failing Corefile and the test query to the incident record before any rollback

6 · Escalation

When the runbook isn't enough, contact:

· CoreDNS Pods are CrashLoopBackOff: the ConfigMap has a syntax error or a referenced plugin is missing; revert to the last known-good Corefile
· CoreDNS reports high error rate for kubernetes plugin: the API service is slow or unreachable; see kubernetes-rb-investigate-api-server-outage
· CoreDNS OOMKilled under load: scale out and tune cache TTL; escalate to capacity planning for the kube-system namespace
· Upstream DNS returns intermittent SERVFAIL: external resolver issue; use a conditional forward block to fall back to a secondary resolver
· nodelocaldns deployed but causing inconsistent resolution: disable nodelocaldns on a sample node and verify behaviour before rolling out a cluster-wide change

DNS failures look like application failures: every getent and every curl to a hostname times out or returns NXDOMAIN. The cause is in CoreDNS or its upstream, not in the workload.

1. Test from a known Pod

Read-only / SafeTest from a known Pod


cat /etc/resolv.conf
echo "---"
nslookup kubernetes.default
echo "---"
nslookup kube-dns.kube-system.svc.cluster.local
echo "---"
dig +short kubernetes.default
'

2. Check CoreDNS Pod health

Read-only / SafeCheck CoreDNS Pod health


kubectl -n kube-system describe deploy coredns | head -30
kubectl -n kube-system logs -l k8s-app=kube-dns --tail=200 --timestamps

If CoreDNS Pods are CrashLoopBackOff, the ConfigMap is broken (Corefile syntax error) or the cluster is short on memory. Do not advance to network debugging until the Pods are stable.

3. Check the Service and endpoints

Read-only / SafeCheck the Service and endpoints


kubectl -n kube-system get endpointslices -l kubernetes.io/service-name=kube-dns -o yaml
kubectl -n kube-system get pods -l k8s-app=kube-dns -o custom-columns=NAME:.metadata.name,IP:.status.podIP

If the Service has no endpoints, CoreDNS Pods are not Ready or do not match the Service selector (rare with stock CoreDNS, common after customisation).

4. Classify the failure

Read-only / SafeClassify the failure


echo "--- NXDOMAIN test (service that does not exist)"
nslookup does-not-exist.default.svc.cluster.local
echo "--- SERVFAIL test (random upstream)"
dig +short _test.example.com @10.96.0.10
echo "--- Timeout test"
time nslookup kubernetes.default
'

Failure	Where to look
NXDOMAIN for a service that should exist	Service selector, search path, ndots
SERVFAIL for any service	CoreDNS Pod logs, ConfigMap syntax
Timeout (no response)	Cluster DNS IP unreachable, CoreDNS CPU/memory exhausted
Works for `kubernetes.default` but not for app service	EndpointSlice or search path

5. CoreDNS logs

Read-only / SafeCoreDNS logs

kubectl -n kube-system logs -l k8s-app=kube-dns --tail=20 | grep -iE 'error|warn' || echo "no recent errors"

Common log patterns and what they mean:

[ERROR] plugin/errors: ... — a plugin returned an error (look at the plugin name)
[WARN] No DNS servers — forward plugin misconfigured
[INFO] ... - dropping request — rate-limited or stuck; check cache plugin
[ERROR] Failed to ... — typically a kubernetes plugin API call failed

6. Common fixes

Read-only / SafeCommon fixes


kubectl -n kube-system rollout restart deploy/coredns
kubectl -n kube-system rollout status deploy/coredns --timeout=5m

# B. Scale out under load
kubectl -n kube-system scale deploy/coredns --replicas=<n>
kubectl -n kube-system get pods -l k8s-app=kube-dns

# C. Fix the Corefile (must be valid CoreDNS syntax)
kubectl -n kube-system edit configmap coredns
# After edit:
kubectl -n kube-system rollout restart deploy/coredns

# D. Fix ndots/search path in a workload
# In the Pod spec:
#   spec:
#     dnsConfig:
#       options:
#       - name: ndots
#         value: "2"
#       searches:
#       - <ns>.svc.cluster.local
#       - svc.cluster.local
#       - cluster.local

7. Verify

Read-only / SafeVerify


nslookup kubernetes.default
nslookup <service>.<ns>.svc.cluster.local
dig +short <service>.<ns>.svc.cluster.local
'

# Cluster-side
kubectl -n kube-system get pods -l k8s-app=kube-dns -o custom-columns=NAME:.metadata.name,READY:.status.containerStatuses[0].ready
kubectl -n kube-system logs -l k8s-app=kube-dns --tail=20 | grep -iE 'error|warn' || echo "no recent errors""}

Common pitfalls

Symptom	Cause	Action
Works from kubelet but not from Pod	Pod `dnsPolicy: Default` or wrong `dnsConfig`	Fix the Pod spec
Works for some services, not others	CoreDNS cache stale or upstream SERVFAIL	Restart CoreDNS Pods; check upstream
Intermittent timeouts under load	CoreDNS undersized or rate-limited	Scale out; review `cache`/`loadbalance` plugins
NXDOMAIN for everything in a new namespace	CoreDNS cache TTL too high	Lower TTL in the Corefile
SERVFAIL for cluster.local	API server unreachable from CoreDNS	See `kubernetes-rb-investigate-api-server-outage`

A DNS issue is rarely a network issue at the cluster level. It is either a CoreDNS configuration problem, an upstream resolver issue, or a workload-level dnsConfig problem. The runbook distinguishes the three before changing anything.