Lab 14: Inspect and troubleshoot CoreDNS

Objective

“DNS is broken” is not a diagnosis. Between an application calling getaddrinfo() and an answer coming back there are six links, each owned by a different thing, each with its own cheap test:

The Pod’s /etc/resolv.conf — written once by the kubelet at Pod creation.
The search path and ndots — which decide how many queries one name becomes.
The kube-dns Service ClusterIP — an ordinary Service with an ordinary EndpointSlice.
The CoreDNS Pods behind it — an ordinary Deployment.
The Corefile plugin chain — where cluster-local and external names take different routes.
The upstream resolver the forward plugin points at.

By the end of this lab you will have read all six on a healthy cluster, watched one short name turn into four queries with CoreDNS’s own logging turned on, and broken the Corefile deliberately to find out what protects you and what does not.

The point is not to memorise a flowchart. It is to be able to say, within a minute, which of the six is wrong — because five of the six produce failures that look identical from inside the application.

Architecture

One lab namespace with three clients that differ only in their DNS configuration, plus the cluster DNS you already have.

dns-lab
├── deploy/web              2 x nginx:1.27.2, so there is a name to resolve
├── svc/web                 ClusterIP, port 80, named port http
├── pod/client              default dnsPolicy ClusterFirst   (the control)
├── pod/client-ndots1       dnsConfig ndots:1                (fewer queries)
└── pod/client-nodedns      dnsPolicy: Default               (the fault)

kube-system
├── svc/kube-dns            the ClusterIP every Pod resolver points at
├── deploy/coredns          the Pods behind that Service
└── cm/coredns              the Corefile

flowchart LR
    A[App calls getaddrinfo] --> B[Link 1 resolv.conf]
    B --> C[Link 2 search path and ndots]
    C --> D[Link 3 kube-dns ClusterIP]
    D --> E[Link 4 CoreDNS Pods]
    E --> F[Link 5 Corefile plugin chain]
    F --> G[kubernetes plugin: cluster.local]
    F --> H[forward plugin: everything else]
    H --> I[Link 6 upstream resolver]

Note where the chain forks. A cluster-local name is answered by the kubernetes plugin and never reaches the upstream; an external name walks past it to forward. That fork is the single most useful thing in the diagram, because it means “internal works, external does not” and “external works, internal does not” are two different faults with two different owners.

Requirements

A kubeadm cluster on Kubernetes 1.34.x, built as in Lab 01, with kubectl 1.34.x and cluster-admin on it, including write access to the coredns ConfigMap in kube-system.
Working cluster DNS right now. This lab inspects a healthy resolver before it disturbs anything. Confirm before you start: kubectl -n kube-system get pods -l k8s-app=kube-dns.
A working CNI, and outbound network access from the cluster for the upstream-resolution tasks. If the cluster has no route out, Task 5’s external lookups will fail for a reason that is real but is not the one being taught.
Ability to pull nginx:1.27.2 and nicolaka/netshoot. The lab names no other images.
Blast radius: one namespace, plus the coredns ConfigMap in kube-system. Tasks 3 and 7 modify that ConfigMap and revert it, and Task 7 ends with a rolling restart of the CoreDNS Deployment to prove the revert. Nothing else in kube-system is touched.

Scenario

At 09:40 a team reports that their new service “cannot reach the payments API”. By 09:55 three separate theories are on the table: DNS is down, the NetworkPolicy is wrong, and the payments service is down. Nobody has run a command.

Cluster DNS is the most over-diagnosed component in Kubernetes, because almost every failure anywhere in the stack presents to the application as a name that did not resolve or an address that did not answer. It is also the component people are most reluctant to inspect, because it lives in kube-system and touching it feels dangerous.

Both problems have the same fix: know exactly which six things you are allowed to read, know that five of them are read-only, and have done it once on a cluster where nothing was wrong.

Tasks

Task 1 — Build the namespace and three clients

kubectl create namespace dns-lab
kubectl label namespace dns-lab lab=dns-lab

cat > dns-lab.yaml <<'YAML'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  namespace: dns-lab
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: nginx
          image: nginx:1.27.2
          ports:
            - name: http
              containerPort: 80
          readinessProbe:
            httpGet:
              path: /
              port: http
            periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: web
  namespace: dns-lab
spec:
  selector:
    app: web
  ports:
    - name: http
      port: 80
      targetPort: http
      protocol: TCP
---
apiVersion: v1
kind: Pod
metadata:
  name: client
  namespace: dns-lab
spec:
  containers:
    - name: netshoot
      image: nicolaka/netshoot
      command: ["sleep", "infinity"]
---
apiVersion: v1
kind: Pod
metadata:
  name: client-ndots1
  namespace: dns-lab
spec:
  dnsConfig:
    options:
      - name: ndots
        value: "1"
  containers:
    - name: netshoot
      image: nicolaka/netshoot
      command: ["sleep", "infinity"]
---
apiVersion: v1
kind: Pod
metadata:
  name: client-nodedns
  namespace: dns-lab
spec:
  dnsPolicy: Default
  containers:
    - name: netshoot
      image: nicolaka/netshoot
      command: ["sleep", "infinity"]
YAML

kubectl apply -f dns-lab.yaml
kubectl -n dns-lab rollout status deploy/web --timeout=120s
kubectl -n dns-lab wait --for=condition=Ready pod/client pod/client-ndots1 pod/client-nodedns --timeout=120s

Three clients, identical except for their DNS settings. client is the control. client-ndots1 lowers ndots from the default 5 to 1. client-nodedns sets dnsPolicy: Default, whose name is one of the worst in the Kubernetes API — it does not mean “the default policy”. The default policy is ClusterFirst. Default means “inherit the node’s resolver”, which is a completely different thing and is the fault you will diagnose in Task 5.

Task 2 — Link 1 and Link 2: read the resolver from inside the Pod

kubectl -n dns-lab exec client -- cat /etc/resolv.conf

Read-only / Safeworkstation

$ kubectl -n dns-lab exec client -- cat /etc/resolv.conf

nameserver 10.96.0.10
search dns-lab.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

Illustrative output

Three lines, three separate facts, and each is worth confirming rather than assuming.

The nameserver must be the kube-dns Service ClusterIP. Check, do not assume — the Service is called kube-dns even though the Deployment behind it is called coredns, a leftover from the pre-CoreDNS era that trips up every newcomer:

kubectl -n kube-system get svc kube-dns -o jsonpath='{.spec.clusterIP}{"\n"}'

The search path is namespace-specific. The first entry contains dns-lab, which is why a short name resolves inside its own namespace and nowhere else. Compare against a Pod in another namespace and the first entry differs.

ndots:5 is the cluster default, and it is the reason Task 3 exists.

Now the fact that matters operationally: the kubelet wrote that file once, when the Pod’s sandbox was created, and nothing updates it afterwards.

kubectl -n dns-lab exec client-ndots1 -- cat /etc/resolv.conf
kubectl -n dns-lab exec client-nodedns -- cat /etc/resolv.conf

client-ndots1 shows options ndots:1. client-nodedns shows something else entirely — the node’s own resolver, with no cluster.local search domains at all. Keep that output; Task 5 uses it.

Task 3 — Turn on CoreDNS query logging and count the queries

Everything so far was read-only. This is the first change, so it gets the full discipline: back up, diff, apply, verify, and know the rollback before you start.

kubectl -n kube-system get configmap coredns -o yaml > coredns-backup.yaml
kubectl -n kube-system get configmap coredns -o jsonpath='{.data.Corefile}' > Corefile.orig

cat Corefile.orig

Read your own Corefile before changing it. You are looking for the errors plugin (the anchor for the edit), the kubernetes plugin and the zones it serves, the forward line and where it points, and the reload plugin — without reload, the change below does nothing until the Pods restart.

Add the log plugin directly after errors:

sed '/^[[:space:]]*errors$/a\        log' Corefile.orig > Corefile.log
diff Corefile.orig Corefile.log

Read the diff. One added line, correctly indented, inside the server block. If the diff shows anything else, stop and fix it here rather than in the cluster.

Configuration changeworkstation— Replaces the cluster DNS configuration; reverted at the end of this task

$ kubectl -n kube-system create configmap coredns --from-file=Corefile=Corefile.log --dry-run=client -o yaml | kubectl -n kube-system apply -f -

The change is not instant, and knowing why matters more than the change itself. The ConfigMap update has to propagate into the CoreDNS Pods’ mounted volume, which the kubelet syncs periodically — up to about a minute by default. Only then does the reload plugin, which polls the file on its own interval, notice the new content and restart the server internally. Two independent delays, neither of which you control, so a Corefile change that has not taken effect after ten seconds is normal, not broken.

kubectl -n kube-system logs -l k8s-app=kube-dns --tail=20

Once you see reload activity, start following the logs in one terminal:

kubectl -n kube-system logs -f -l k8s-app=kube-dns --tail=0

And in a second terminal, make exactly one request:

kubectl -n dns-lab exec client -- curl -s -o /dev/null -w '%{http_code}\n' http://web

Count the query lines. One curl to a one-word name produced four lookups — web.dns-lab.svc.cluster.local, web.svc.cluster.local, web.cluster.local, and finally web itself — because web has zero dots, which is fewer than ndots:5, so the resolver walks the search path before trying the name as given. A dual-stack client asking for A and AAAA doubles it again.

Now the same name, absolute:

kubectl -n dns-lab exec client -- curl -s -o /dev/null -w '%{http_code}\n' \
  http://web.dns-lab.svc.cluster.local./

One query. The trailing dot is what did it — not the length of the name. web.dns-lab.svc.cluster.local without the dot has four dots, still fewer than five, so it walks the entire search path too and only succeeds on the fourth attempt.

And the low-ndots client:

kubectl -n dns-lab exec client-ndots1 -- curl -s -o /dev/null -w '%{http_code}\n' \
  http://web.dns-lab.svc.cluster.local

With ndots:1, a name containing one or more dots is tried as absolute first, so this resolves in one query without any trailing dot. The trade-off is real: a bare short name still expands, but anything you rely on the search path to complete beyond one label now behaves differently.

kubectl -n kube-system create configmap coredns \
  --from-file=Corefile=Corefile.orig --dry-run=client -o yaml \
  | kubectl -n kube-system apply -f -

kubectl -n kube-system get configmap coredns -o jsonpath='{.data.Corefile}' \
  | diff - Corefile.orig && echo "Corefile restored"

Task 4 — Links 3 and 4: cluster DNS is an ordinary Service

There is a persistent instinct to treat cluster DNS as special infrastructure that can only be inspected with special tools. It cannot, and it is not.

kubectl -n kube-system get svc kube-dns
kubectl -n kube-system get endpointslices -l kubernetes.io/service-name=kube-dns
kubectl -n kube-system get pods -l k8s-app=kube-dns -o wide

Three commands you already know from Lab 12, pointed at kube-system. The numbers must agree: the count of ready endpoints in the slice equals the count of Ready CoreDNS Pods. When they disagree, you are looking at exactly the same class of fault as any other Service, and the same evidence settles it.

kubectl -n kube-system get endpointslices -l kubernetes.io/service-name=kube-dns \
  -o jsonpath='{range .items[*].endpoints[*]}{.addresses[0]}{" ready="}{.conditions.ready}{"\n"}{end}'

CoreDNS also reports on itself. The prometheus plugin exposes metrics on port 9153, reachable directly from any Pod:

COREDNS_IP=$(kubectl -n kube-system get pod -l k8s-app=kube-dns \
  -o jsonpath='{.items[0].status.podIP}')

kubectl -n dns-lab exec client -- \
  curl -s "http://${COREDNS_IP}:9153/metrics" | grep -E '^coredns_dns_(requests|responses)_total'

Read the rcode labels on coredns_dns_responses_total. You will find a large NXDOMAIN count sitting next to the NOERROR count, and it is not a fault.

That NXDOMAIN volume is the search path you measured in Task 3: every short name that eventually resolves produces up to three failed lookups first, and each one is a legitimate NXDOMAIN. A Kubernetes cluster with a healthy DNS service normally shows a NXDOMAIN ratio that would be alarming anywhere else. Alerting on absolute NXDOMAIN rate here produces a permanently firing alert; what is worth watching is a change in the ratio, which usually means someone deployed a client that is asking for a name that does not exist.

COREDNS_IP=$(kubectl -n kube-system get pod -l k8s-app=kube-dns \
  -o jsonpath='{.items[0].status.podIP}')

kubectl -n dns-lab exec client -- \
  curl -s "http://${COREDNS_IP}:9153/metrics" | grep -E '^coredns_cache_(hits|misses)_total'

Cache hits over hits plus misses is the cache hit ratio. A low ratio with a normal query rate means the queries are mostly unique — which, again, is often the search path generating names that will never be cached usefully.

Task 5 — Link 5: the fork in the plugin chain

Now use the third client. It has dnsPolicy: Default, which means it inherited the node’s resolver and knows nothing about the cluster.

Do not look at its manifest. Diagnose it from the symptoms:

kubectl -n dns-lab exec client-nodedns -- curl -s -o /dev/null -m 5 -w '%{http_code}\n' http://web
kubectl -n dns-lab exec client-nodedns -- getent hosts web.dns-lab.svc.cluster.local
kubectl -n dns-lab exec client-nodedns -- getent hosts kubernetes.io

Cluster-local names fail — both the short one and the fully qualified one — and the external name resolves fine. Meanwhile the control client resolves both.

That signature localises the fault immediately, and it is worth being explicit about why. If CoreDNS were down, everything would fail, for every Pod. If the kubernetes plugin were broken, cluster names would fail for every Pod while external ones worked. Here only one Pod is affected, and it is the fork in the chain that tells you which half: this client is not reaching the kubernetes plugin at all, because it is not reaching CoreDNS at all.

Confirm with the evidence you already collected in Task 2:

kubectl -n dns-lab exec client-nodedns -- cat /etc/resolv.conf
kubectl -n dns-lab exec client -- cat /etc/resolv.conf

Different nameserver, and no cluster.local search domains. The fault is in Link 1 for this Pod, not in CoreDNS. Nothing in kube-system needed touching, and the whole diagnosis was three read-only commands.

Now confirm the other half of the fork, on the control client:

kubectl -n dns-lab exec client -- getent hosts web.dns-lab.svc.cluster.local
kubectl -n dns-lab exec client -- getent hosts kubernetes.io

Both work. The first was answered by the kubernetes plugin from its watch on the API server; the second walked past it to forward, which sent it to whatever the node’s /etc/resolv.conf names. Two different halves of the same server, one of which depends on the node’s own DNS being correct — which is why “external name resolution broke for the whole cluster” is often a node-level or upstream-level problem that CoreDNS is merely reporting.

Task 6 — Pod records and SRV records

Two record types most operators never look at until they need them. Both depend on the Corefile, so check first:

grep -A 4 'kubernetes cluster.local' Corefile.orig

If the kubernetes block contains pods insecure, Pod A records are served. The name format substitutes dashes for the dots in the Pod IP:

POD_IP=$(kubectl -n dns-lab get pod -l app=web -o jsonpath='{.items[0].status.podIP}')
POD_DNS=$(echo "$POD_IP" | tr '.' '-')

kubectl -n dns-lab exec client -- \
  getent hosts "${POD_DNS}.dns-lab.pod.cluster.local"

Note what this record does not do: it is generated from the address, not from a watch on the Pod, so it resolves whether or not a Pod with that IP exists. That is what insecure in pods insecure is telling you.

SRV records carry the port, which is how a client can discover a port it was never told:

kubectl -n dns-lab exec client -- \
  dig +short SRV _http._tcp.web.dns-lab.svc.cluster.local

The four fields are priority, weight, port and target. The port comes from the Service’s named port — _http matches the name: http in the Service’s ports entry. An unnamed port has no SRV record, which is one more reason to name them.

Task 7 — Break the Corefile on purpose, and prove the rollback

The most common way to take out cluster DNS is a bad Corefile edit. This task finds out exactly how much protection you have.

cp Corefile.orig Corefile.broken
printf 'not-a-real-plugin\n' >> Corefile.broken
diff Corefile.orig Corefile.broken

Cluster-wide riskworkstation— Deliberately invalid cluster DNS configuration; reverted in this same task

$ kubectl -n kube-system create configmap coredns --from-file=Corefile=Corefile.broken --dry-run=client -o yaml | kubectl -n kube-system apply -f -

Wait for propagation, then look at two things in this order:

kubectl -n dns-lab exec client -- getent hosts web.dns-lab.svc.cluster.local
kubectl -n kube-system logs -l k8s-app=kube-dns --tail=30 | grep -iE 'error|reload'

DNS still works. The reload plugin parsed the new Corefile, found it invalid, refused to apply it, logged the failure, and kept serving the previously valid configuration. That is genuinely good design and it is why a fat-fingered Corefile edit does not usually cause an immediate outage.

Now understand precisely what it did not protect you from.

Restore, and then prove the restore:

kubectl -n kube-system create configmap coredns \
  --from-file=Corefile=Corefile.orig --dry-run=client -o yaml \
  | kubectl -n kube-system apply -f -

kubectl -n kube-system get configmap coredns -o jsonpath='{.data.Corefile}' \
  | diff - Corefile.orig && echo "Corefile restored"

A diff proving the ConfigMap matches your backup is necessary and not sufficient, because the thing you actually need to know is whether a CoreDNS Pod can start from it. There is exactly one way to find that out:

Service impact possibleworkstation— Rolling restart of cluster DNS; with more than one replica the Service stays up

$ kubectl -n kube-system rollout restart deployment/coredns

kubectl -n kube-system rollout status deployment/coredns --timeout=180s
kubectl -n kube-system get pods -l k8s-app=kube-dns
kubectl -n dns-lab exec client -- getent hosts web.dns-lab.svc.cluster.local

A Deployment rollout replaces Pods one at a time and waits for each to become Ready, so with two or more replicas the kube-dns Service keeps ready endpoints throughout and no query is lost. With a single replica there is a gap, which is one concrete reason cluster DNS should not run a single replica.

If the rollout stalls with a Pod in CrashLoopBackOff, the Corefile is still wrong and kubectl -n kube-system logs on that Pod will name the line. That outcome is exactly what this task exists to make visible while it is cheap.

Task 8 — The chain table

Fill this in from your own captures. It is the deliverable, and it is the thing worth keeping.

Link	What it is	Cheapest test	Failure signature
1	Pod `/etc/resolv.conf`	`kubectl exec POD -- cat /etc/resolv.conf`	One Pod fails; others fine
2	Search path and `ndots`	Same file, plus a query count from the `log` plugin	Slow resolution, or short names failing across namespaces
3	`kube-dns` ClusterIP	`kubectl -n kube-system get svc,endpointslices`	Whole cluster fails; no endpoints
4	CoreDNS Pods	`kubectl -n kube-system get pods -l k8s-app=kube-dns`	Whole cluster fails; Pods not Ready
5	Corefile plugin chain	Read the ConfigMap; the reload lines in the logs	One half fails: cluster-local or external
6	Upstream resolver	`getent hosts` for an external name from a Pod	External fails, cluster-local fine

Links 1 to 4 and 6 are read-only. Only link 5 requires touching anything, and only when the evidence has already pointed there.

Validation

Save as validate.sh and run it. It exits non-zero while anything is wrong, including a Corefile you forgot to restore.

#!/usr/bin/env bash
set -euo pipefail
NS=dns-lab

dnsip=$(kubectl -n kube-system get svc kube-dns -o jsonpath='{.spec.clusterIP}')
podns=$(kubectl -n "$NS" exec client -- \
  sh -c "awk '/^nameserver/ {print \$2; exit}' /etc/resolv.conf" | tr -d '\r')
if [ "$dnsip" != "$podns" ]; then
  echo "FAIL client resolver is $podns, kube-dns is $dnsip"
  exit 1
fi
echo "ok   client resolver points at kube-dns ($dnsip)"

pods=$(kubectl -n kube-system get pods -l k8s-app=kube-dns \
  -o jsonpath='{range .items[*]}{.status.conditions[?(@.type=="Ready")].status}{"\n"}{end}' \
  | grep -c True)
eps=$(kubectl -n kube-system get endpointslices \
  -l kubernetes.io/service-name=kube-dns \
  -o jsonpath='{range .items[*].endpoints[*]}{.conditions.ready}{"\n"}{end}' \
  | grep -c true)
if [ "$pods" -lt 1 ] || [ "$pods" -ne "$eps" ]; then
  echo "FAIL $pods ready CoreDNS Pods but $eps ready endpoints"
  exit 1
fi
echo "ok   $pods ready CoreDNS Pods, $eps ready endpoints"

kubectl -n kube-system get configmap coredns -o jsonpath='{.data.Corefile}' \
  | diff -q - Corefile.orig >/dev/null
echo "ok   Corefile matches the Task 3 backup"

if kubectl -n kube-system get configmap coredns -o jsonpath='{.data.Corefile}' \
  | grep -qE '^[[:space:]]*log$'; then
  echo "FAIL the log plugin is still enabled"
  exit 1
fi
echo "ok   the log plugin is off"

kubectl -n "$NS" exec client -- getent hosts web."$NS".svc.cluster.local >/dev/null
echo "ok   cluster-local name resolves from the control client"

kubectl -n "$NS" exec client -- getent hosts kubernetes.io >/dev/null
echo "ok   external name resolves from the control client"

if kubectl -n "$NS" exec client-nodedns -- \
  getent hosts web."$NS".svc.cluster.local >/dev/null 2>&1; then
  echo "FAIL client-nodedns resolved a cluster name; it should not"
  exit 1
fi
echo "ok   client-nodedns still cannot resolve cluster names, as designed"

Seven claims, and the last two are the ones people skip. Proving the fault Pod still fails is what confirms you diagnosed a real configuration difference rather than a transient. Proving the log plugin is off is what stops a debugging aid from becoming a permanent cost, which is the single most common way these sessions leave damage behind.

Expected Outcome

Corefile        identical to Corefile.orig, verified by diff
                and by a rollout restart that reached Ready
CoreDNS         all Pods Running and Ready; endpoint count matches Pod count
client          resolves cluster-local and external names
client-ndots1   resolves an FQDN in one query instead of four
client-nodedns  resolves external names only, by design

validate.sh exits 0, and you hold a completed chain table plus a query-log capture showing four lookups for one short name and one for the absolute form.

Troubleshooting

ImagePullBackOff on a client Pod. nicolaka/netshoot comes from Docker Hub and anonymous pulls are rate limited. Any image with a shell and a resolver works; this course also uses registry.k8s.io/e2e-test-images/jessie-dnsutils:1.7. Check what a substitute actually has before relying on it: kubectl -n dns-lab exec client -- sh -c 'command -v dig getent nslookup curl'.

The log plugin produces no output. Either the ConfigMap has not propagated yet — give it two minutes — or the Corefile has no reload plugin, in which case nothing rereads it until the Pods restart. Confirm with grep reload Corefile.orig.

The sed command did not insert the line. Your Corefile’s errors line may carry a trailing space or a block. Look at it, then edit Corefile.log by hand; the diff step exists exactly so this is caught before it reaches the cluster.

Every query in the log is duplicated. The client is dual-stack and asking for A and AAAA. That is normal, and it doubles the search-path cost measured in Task 3.

getent hosts returns nothing for an external name from every Pod. Look at Link 6, not at CoreDNS. The forward plugin usually points at /etc/resolv.conf, meaning the node’s resolver, so a node with broken upstream DNS presents as a cluster-wide external-resolution outage while cluster-local names keep working perfectly.

CoreDNS logs a loop detection error and the Pods crash. The loop plugin has found the forward target resolving back to CoreDNS itself. This happens when the node’s /etc/resolv.conf points at a local stub resolver that forwards to the cluster DNS. Fix the node’s resolver, or point forward at a real upstream address instead of /etc/resolv.conf.

The rollout restart never completes. A CoreDNS Pod in CrashLoopBackOff after a Corefile change is the latent failure from Task 7 arriving. Read kubectl -n kube-system logs POD — the parse error names the offending line — and restore from Corefile.orig.

client-nodedns resolves cluster names anyway. Then the node’s resolver happens to be able to reach cluster DNS, which some single-node setups produce. The teaching still holds; the search-path difference in /etc/resolv.conf is still visible and is still the evidence.

Cleanup

Two things to undo, and the order matters: cluster DNS first, the namespace second.

kubectl -n kube-system get configmap coredns -o jsonpath='{.data.Corefile}' \
  | diff - Corefile.orig && echo "Corefile already restored"

kubectl -n kube-system get pods -l k8s-app=kube-dns

If that diff shows anything, restore before going any further:

kubectl -n kube-system create configmap coredns \
  --from-file=Corefile=Corefile.orig --dry-run=client -o yaml \
  | kubectl -n kube-system apply -f -

Destructiveworkstation— Deletes the lab namespace and everything inside it

$ kubectl delete namespace dns-lab

kubectl get namespace dns-lab 2>&1 | grep -q NotFound && echo "namespace gone"
rm -f dns-lab.yaml Corefile.orig Corefile.log Corefile.broken \
      coredns-backup.yaml validate.sh

Nothing else was modified. No node, no CNI configuration, no kube-proxy setting, and no object in kube-system other than the coredns ConfigMap, which is back to its original content and has been proved to start cleanly.

Production notes

Every Corefile change needs the same four steps. Back up to a file, diff the proposed content, apply, and then verify with a rollout restart — because the reload plugin’s protection makes a broken Corefile invisible until something restarts. Verifying only that “DNS still works” verifies nothing about the config you just installed.

Cluster DNS needs the treatment you give any critical Deployment. More than one replica, anti-affinity so the replicas are not on one node, a PodDisruptionBudget that allows a drain but not a total outage, and resource requests so it is not the first thing evicted under node pressure. It is an ordinary Deployment, and that cuts both ways: nothing protects it unless you do.

The search path is a real cost, and there are three ways to reduce it. Use absolute names with a trailing dot in high-QPS clients; lower ndots for workloads that mostly talk to external names; or deploy a node-local DNS cache so the amplified queries are answered on the node instead of crossing the network. The first is free and the most reliable, and it is the one nobody does because it looks like a typo in a config file.

Do not alert on absolute NXDOMAIN rate. In a Kubernetes cluster it is structurally high and always will be. Alert on request rate, on the ratio of SERVFAIL responses, on upstream latency, and on CoreDNS Pod readiness. A SERVFAIL spike means the upstream is failing; an NXDOMAIN spike usually means somebody deployed a client asking for a name that does not exist.

Cluster DNS depends on node DNS, and most people do not know it. The default forward . /etc/resolv.conf makes the node’s own resolver an upstream dependency of the entire cluster. A change to the nodes’ DNS configuration — a new corporate resolver, a VPN, an image rebuild — is a change to cluster DNS, and it will present as an application problem.

Turning on query logging is a change window, not a debugging convenience. It is the right tool for “which names is this workload actually asking for”, and it must go out and come back in the same session with an owner watching. A log plugin left enabled on a busy cluster is a self-inflicted capacity incident.

Under the hood

The plugin chain is a chain of responsibility, and the order in the Corefile is not decorative — but it is also not the order they execute in. CoreDNS compiles the plugin chain in a fixed, built-in order determined by the plugin registry, not by the order you wrote them. Rearranging lines in the Corefile does not rearrange the chain, which is why a Corefile with the plugins in an odd order still behaves correctly and why “I moved cache above forward” changes nothing.

What the chain does mean is that each plugin either answers or passes the query on. The kubernetes plugin owns the zones listed on its line — usually cluster.local plus the reverse zones — and answers anything inside them from a watch it holds on the API server. A name outside those zones is not its business, so it declines and the query continues to forward.

fallthrough is the exception that makes reverse lookups work. Without it, the kubernetes plugin claiming in-addr.arpa would answer NXDOMAIN for every reverse lookup of an address it does not know about, including every external one. fallthrough in-addr.arpa ip6.arpa says “if I claim this zone but have no record, let the next plugin try anyway”.

Two more mechanisms worth knowing:

The loop plugin is a suicide switch, deliberately. At startup it sends itself a probe query, and if that query comes back around to CoreDNS, it halts the process instead of running. A DNS forwarding loop would otherwise amplify until it consumed the node, so failing to start is the safer outcome. When you see CoreDNS crash-looping immediately after a node’s /etc/resolv.conf changed, this is almost always why.

The kubernetes plugin needs RBAC. It watches Services, EndpointSlices and Namespaces through a ClusterRole bound to the CoreDNS ServiceAccount. Break that binding and CoreDNS starts, reports healthy, answers external names perfectly — and returns NXDOMAIN for every Service in the cluster, because its watch never populated. That is the purest example of why link 5’s fork matters: a healthy process, a healthy Service, a healthy endpoint list, and half the answers missing.

Finally, the health plugin serves /health on port 8080 and the ready plugin serves /ready on port 8181, which are what the CoreDNS Pod’s liveness and readiness probes use. ready returns success only once every plugin that declares readiness — including kubernetes, after its initial watch sync — is ready. That is why a restarted CoreDNS Pod takes a moment to become Ready rather than answering immediately with an empty cluster view.

What You Learned

“DNS is broken” names six possible faults, and five of them are read-only to diagnose. The resolver file, the search path, the Service, the Pods and the upstream can all be inspected without changing anything.
A Pod’s /etc/resolv.conf is written once and never updated. Cluster DNS configuration a Pod received at creation stays with it for life, so a change to the kube-dns ClusterIP strands every running Pod.
One short name is four queries. The search path plus ndots:5 turns web into four lookups, three of which are NXDOMAIN by design — which is why a Kubernetes cluster’s NXDOMAIN rate is structurally high and is not an alert.
A trailing dot, not a long name, is what skips the search path. web.dns-lab.svc.cluster.local has four dots, still under ndots:5, and walks the whole path anyway.
dnsPolicy: Default does not mean the default policy. It means the node’s resolver, and it is the reason hostNetwork Pods silently lose cluster-local name resolution while resolving external names perfectly.
The fork in the plugin chain is the fastest localiser you have. Cluster-local names come from the kubernetes plugin; everything else goes to forward and the node’s upstream. Which half failed tells you whose problem it is.
A rejected Corefile is a latent outage. reload refuses invalid config and keeps serving the old one, so DNS keeps working and the broken ConfigMap sits there until the next Pod restart. Only a rollout restart verifies a Corefile change.
Cluster DNS is an ordinary Service in front of an ordinary Deployment. The same three commands that debug any Service debug this one, and the same production disciplines — replicas, anti-affinity, a PDB, resource requests — are the ones protecting it.

Why this matters

DNS gets blamed for more Kubernetes incidents than it causes, and it causes plenty. The reason is that almost every failure in the stack reaches the application as a name that did not resolve or an address that did not answer, so “it’s DNS” is the cheapest available theory and it is unfalsifiable until somebody runs a command.

The six-link chain is what makes it falsifiable. Five of the six can be checked read-only, from a laptop, in under a minute, and each produces a different signature: one Pod versus all Pods, cluster names versus external names, no endpoints versus endpoints that are not ready. An operator who has walked the chain once on a working cluster can rule out four of the six before the incident channel has finished arguing.

And the sixth link — the Corefile — is where the real damage happens, not because it is hard to edit but because its safety mechanism hides the mistake. The protection that keeps a bad edit from taking DNS down immediately is exactly what lets it take DNS down three days later, during an unrelated node drain, with nobody connecting the two events.

Objectives

Prerequisites