CoreDNS failure · Break/Fix · Kubernetes

Reported symptoms

At 09:05 the checkout service p99 goes from 140 milliseconds to 5.2 seconds. Not a curve — a step. It holds for eleven minutes, falls back, and steps up again at 09:23.

By 09:40 there are four incidents open, filed by four teams who have not spoken to each other:

Checkout. p99 at 5.2 seconds in bursts. Error rate normal. CPU and memory flat. No deploy since Tuesday.
Payments. Intermittent java.net.UnknownHostException for ledger.payments.svc.cluster.local, a Service that resolves instantly every time anyone tests it by hand.
Data platform. The nightly reconciliation job timed out against an external API for the second time this week.
Edge. The ingress controller is returning 504 for three backends that share nothing except the cluster.

Nothing is down. Every Deployment is at full replicas, no node reports pressure, and no application Pod has restarted. The CoreDNS Pods have zero restarts.

They also have an age measured in hours, because CoreDNS was scaled from two replicas to eight at 16:20 yesterday, after a similar scatter of complaints. The complaints stopped for the rest of the afternoon. That is why nobody looked further.

The questions in the room:

Is this one problem or four?
If scaling CoreDNS fixed it yesterday, why is it back?
Why is the number always five seconds?

Evidence provided

Read-only / Safeeight Ready, no restarts, six of them created yesterday afternoon

$ kubectl get pods -n kube-system -l k8s-app=kube-dns -o wide

NAME                       READY   STATUS    RESTARTS   AGE     NODE
coredns-6b9f4c8d4-2xk7p    1/1     Running   0          17h     node-04
coredns-6b9f4c8d4-4wmzt    1/1     Running   0          17h     node-11
coredns-6b9f4c8d4-8jd6r    1/1     Running   0          17h     node-19
coredns-6b9f4c8d4-hq2vn    1/1     Running   0          43d     node-02
coredns-6b9f4c8d4-k5tbc    1/1     Running   0          17h     node-27
coredns-6b9f4c8d4-p7rlx    1/1     Running   0          17h     node-33
coredns-6b9f4c8d4-t9nqw    1/1     Running   0          43d     node-08
coredns-6b9f4c8d4-vz4hd    1/1     Running   0          17h     node-41

Illustrative output

Read-only / Safeall eight at roughly a core, all at once — this is not one hot replica

$ kubectl top pods -n kube-system -l k8s-app=kube-dns --no-headers

coredns-6b9f4c8d4-2xk7p   961m   58Mi
coredns-6b9f4c8d4-4wmzt   974m   61Mi
coredns-6b9f4c8d4-8jd6r   952m   57Mi
coredns-6b9f4c8d4-hq2vn   1003m  74Mi
coredns-6b9f4c8d4-k5tbc   968m   59Mi
coredns-6b9f4c8d4-p7rlx   949m   56Mi
coredns-6b9f4c8d4-t9nqw   991m   72Mi
coredns-6b9f4c8d4-vz4hd   957m   58Mi

Illustrative output

Read-only / Safeafter kubectl port-forward -n kube-system deploy/coredns 9153:9153— three NXDOMAIN for every NOERROR, and it is not a spike

$ curl -s http://127.0.0.1:9153/metrics | grep '^coredns_dns_responses_total' | grep 'cluster.local'

coredns_dns_responses_total{rcode="NOERROR",server="dns://:53",zone="cluster.local"} 1.94e+08
coredns_dns_responses_total{rcode="NXDOMAIN",server="dns://:53",zone="cluster.local"} 5.81e+08
coredns_dns_responses_total{rcode="SERVFAIL",server="dns://:53",zone="cluster.local"} 214

Illustrative output

Aggregated across the eight replicas, the monitoring stack puts the current rates at roughly 36,000 NXDOMAIN per second and 12,000 NOERROR per second, for about 48,000 queries per second in total. Six weeks ago the same panel read about 12,000 per second. The cluster has 3,010 Pods today against 2,570 six weeks ago.

Read-only / Safethe default the kubelet writes, unchanged

$ kubectl exec -n prod deploy/checkout -- cat /etc/resolv.conf

nameserver 10.96.0.10
search prod.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

Illustrative output

Read-only / Safesomebody is asking this question, thousands of times a second

$ kubectl exec -n prod deploy/checkout -- nslookup billing.prod.svc.cluster.local.svc.cluster.local

Server:		10.96.0.10
Address:	10.96.0.10:53

** server can't find billing.prod.svc.cluster.local.svc.cluster.local: NXDOMAIN

Illustrative output

Read-only / Safesix weeks ago, reviewed and approved, and correct as far as it went

$ git log --oneline -1 --since=2026-06-20 -- platform/clients/service-addresses.yaml

a41c9e2 platform: address services by fully qualified name everywhere

Illustrative output

Work the evidence before reading on

Every component here is healthy. CoreDNS is Ready, the API server is fine, the network is fine, and the Services all resolve.

Take the ratio of NXDOMAIN to NOERROR and hold it next to the search path in the resolv.conf. How many queries does one name lookup cost?
The query rate quadrupled while the Pod count rose by under a fifth. What else changed in that window?
The recurring latency figure is 5.2 seconds. What in this system has a five-second constant in it?

Before continuing: if the answer were “CoreDNS is undersized”, what should have happened after yesterday afternoon that did not?

Root cause

1. ndots:5 makes a fully qualified name the expensive one

The kubelet writes options ndots:5 into every Pod resolv.conf. The rule is simple and its consequence is not: a name with fewer than five dots is tried against the search path first, and only tried as written once the search path is exhausted.

billing.prod.svc.cluster.local has four dots. So the resolver asks, in order:

billing.prod.svc.cluster.local.prod.svc.cluster.local — NXDOMAIN
billing.prod.svc.cluster.local.svc.cluster.local — NXDOMAIN
billing.prod.svc.cluster.local.cluster.local — NXDOMAIN
billing.prod.svc.cluster.local — the answer

Four queries. The short name billing, which has no dots, matches the first search entry and needs one. The fully qualified name — the one everyone agrees is the correct, unambiguous way to address a Service — is four times more expensive than the short name it replaced.

Commit a41c9e2, six weeks ago, replaced the short names with fully qualified ones across the fleet. It was a good change made for a good reason, it passed review, and it quadrupled the cluster DNS load without touching a single line of application logic.

2. The multiplier is eight, not four

The C library resolves a hostname by asking for A and AAAA records concurrently. Every step in the list above is therefore two UDP exchanges, not one: eight packets out, eight conntrack entries, eight replies, for one name.

The arithmetic closes. 12,000 NOERROR per second, divided by two for the A/AAAA pair, is about 6,000 real name lookups per second across 3,010 Pods — two lookups per Pod per second, which is an unremarkable number for this fleet. Those 6,000 lookups are costing 48,000 queries. Before the change they cost about 12,000. Nothing about the applications got busier.

3. The negative answers are the ones the cache helps least

The standard Corefile caches successful answers for thirty seconds and denials for five. That is a deliberate and sensible default — a negative answer is much more likely to become wrong — but it means the three NXDOMAIN queries in every lookup expire six times faster than the one useful answer. The amplification is concentrated exactly where the cache is weakest.

All four queries are answered locally by the kubernetes plugin, which is authoritative for cluster.local and does not forward. So the cost is not upstream latency. It is CPU, sockets and conntrack entries on the resolver path, which is why the failure looks like packet loss rather than like slow DNS.

4. Five seconds is the resolver, not the application

When the resolver is saturated enough to drop a UDP reply, the client does not get an error. It waits for the timeout, which defaults to five seconds, and then retries. That is the entire explanation of a p99 that steps to 5.2 seconds instead of drifting upward, and of a Java service that reports UnknownHostException for a name that resolves perfectly when you test it by hand a second later.

Under the hood

The reason this is invisible on a dashboard is that no layer reports it as an error until the very last one.

CoreDNS is doing exactly what it is asked. Every NXDOMAIN it returns is a correct answer to a question the client genuinely asked. Its own error counter does not move, its readiness probe passes, and its logs are quiet — the errors plugin logs errors, and there are none.

The application sees a name resolution that took five seconds. The metric it exports is request latency, not DNS latency, so the five seconds lands in the same bucket as a slow database. Four teams therefore see four different stories about their own dependencies.

The two numbers that make it visible are both derived, and neither is on a default dashboard:

Queries per Pod per second. Total coredns_dns_requests_total rate divided by the cluster Pod count. It is stable for a given fleet and a given set of client conventions, so a step change in it means a convention changed. This is the number that would have caught commit a41c9e2 six weeks ago.
NXDOMAIN share of cluster.local responses. Sustained above about half it means the search path is being walked for names that do not need it, and there is no other common cause.

There is a third way to see it, useful when the metrics are not yet wired: on a single Pod, resolve the name with a trailing dot. billing.prod.svc.cluster.local. is absolute, bypasses the search path entirely, and answers in one query. If adding a dot changes the behaviour, ndots is your answer.

Restarting CoreDNS is the instinct and it is the one move that always makes this worse

The reflex when DNS is unhappy is kubectl rollout restart deployment/coredns -n kube-system. Consider what it does here.

Each replica holds its own cache. A rolling restart empties all of them within a couple of minutes, at a moment when the resolver is already at its CPU ceiling. Every cached answer in the cluster has to be recomputed at once, on fewer replicas than usual, while the client-side retries from the queries that were dropped during the restart arrive on top.

The second reflex is to scale again — eight replicas to sixteen. That will work. It worked yesterday. It buys roughly as much time as the ratio of the numbers, and it makes the next recurrence harder to diagnose because the symptom will have moved to a different hour of the day.

Neither is forbidden. Scaling as a deliberate hold, with an owner and an end time by which the real change ships, is a reasonable call at 09:40 with four incidents open. Scaling instead of finding the multiplier is how a cluster arrives at sixteen CoreDNS replicas and no idea why it needs them.

Resolution

State the finding before touching anything: the query rate per Pod, the NXDOMAIN share, and the arithmetic that connects them to commit a41c9e2. Four incidents close into one on that sentence.
Do not restart CoreDNS. Say so out loud, because somebody is already typing it.
Pick one Deployment to prove the fix on. Checkout is the loudest, which makes it the best evidence.
Add dnsConfig with ndots: 2 to that Deployment pod template and let it roll. A four-dot name now goes out as written on the first attempt.
Watch the response counters split by rcode for that namespace. The NXDOMAIN share is the number that proves it worked; the latency graph is a lagging confirmation.
Roll the same change to the rest of the affected workloads in batches, watching the aggregate query rate come down in steps as you go.
If the change cannot be reviewed and rolled today, hold deliberately: scale CoreDNS to buy the day, record the owner and the date the ndots change ships, and say plainly that the scale-out is not the fix.
Leave the eight replicas in place until at least one full morning peak has passed with the new query rate. Scaling down is a separate, later, deliberate change.

Verification

The NXDOMAIN share of cluster.local responses has fallen from roughly three per NOERROR to a small residue. This is the primary check and it is the one that can fail.
Total query rate has fallen by close to three quarters for the workloads you changed, with no change in their application request rate. If request rate moved too, you have measured a traffic dip rather than a fix.
CoreDNS CPU is off its ceiling and stays off through the next morning peak. Yesterday failed this test; it is the one that distinguishes a fix from headroom.
A Pod in a changed workload shows options ndots:2 in its resolv.conf, and the Service name still resolves from inside it.
The 5.2 second latency band has disappeared from the checkout histogram rather than shrunk. A smaller band means the resolver is still dropping replies, only less often.
The Java service has stopped logging UnknownHostException. Confirm from its logs, not from the absence of new tickets.
The three other incidents are closed against this root cause, in writing. An incident that quietly stops being mentioned will be reopened as a new one next month.

Prevention

Alert on queries per Pod, not on CoreDNS being up. A resolver that is Running, Ready and answering correctly can still be the reason four services are slow. The rate divided by the Pod count is stable for a given set of client conventions, so a step in it means a convention changed.
Alert on the NXDOMAIN share of responses in the cluster zone. A sustained majority of negative answers has essentially one cause.
Set ndots deliberately in the workload template rather than inheriting the kubelet default, and put the arithmetic next to it:

spec:
  template:
    spec:
      dnsConfig:
        options:
          # Cluster names have at most 4 dots. ndots:2 sends any name with
          # 2 or more dots out as written first, so a fully qualified
          # Service name costs one query instead of four.
          - name: ndots
            value: "2"

Treat a fleet-wide change in how services address each other as a change to DNS load, and measure the query rate either side of it. Commit a41c9e2 was correct in intent and had a four-times cost that nobody costed.
Record CoreDNS scale-outs as incidents. A lever that removes a symptom without changing the cause hides the cause for exactly as long as the headroom lasts, and the next occurrence starts with a cluster that is already four times over-provisioned.
Consider NodeLocal DNSCache for the durable fix rather than the incident fix. A per-node cache shortens the path, keeps answers on the node, and reduces the conntrack pressure that turns saturation into dropped replies. It is a DaemonSet that changes every Pod resolver path, so it belongs in a maintenance window and not in a 09:40 incident.

What if the clients had used a trailing dot instead?

billing.prod.svc.cluster.local. — with the final dot — is an absolute name. The resolver does not consult the search path for it at all, whatever ndots says. One query, no NXDOMAIN, no ndots change, no Pod restart if the name comes from a ConfigMap.

It is the cheapest possible fix and it is fragile in a specific way: the trailing dot is invisible in most user interfaces, survives badly through templating and copy-paste, and is silently dropped by some client libraries that normalise hostnames. A convention that depends on a character nobody can see will decay.

Lowering ndots has a cost of its own worth naming. With ndots:2, a name with two or more dots is sent as written before the search path is tried, so a name that was meant to be completed by the search path now makes a failing query first — and if that query leaves the cluster, an internal Service name has been sent to an external resolver. In an estate where the upstream resolver answers unknown names with a wildcard rather than NXDOMAIN, that failing query can come back with an address, and the client will use it.

The safe combination is both: set ndots deliberately for the cluster, and know which of your names are absolute and which rely on the search path. The dangerous combination is neither, which is where this cluster was.

Why this matters

Every component in this incident was healthy, and that is the point. CoreDNS was Ready and answering every query correctly. The applications were correct. The change that caused it was correct. No probe failed, no log line named the problem, and the only error anyone could see was a hostname lookup that worked when you tried it.

DNS is the dependency almost every workload uses on almost every request, so its cost multiplies by the whole fleet and its degradation arrives as somebody else’s symptom. That is why a resolver needs metrics about the shape of its traffic and not only about its own health, and why “is CoreDNS up” is not a useful question.

The wider habit is the one to keep. When a resource is saturated, the first question is not how much more of it to buy — it is what changed about the demand. Here the demand quadrupled six weeks before anybody noticed, because the change that quadrupled it looked like tidying up.

Reported symptoms

Evidence

Root cause

Remediation

Verification

Prevention

Reported symptoms

Evidence provided

Work the evidence before reading on

Root cause

1. ndots:5 makes a fully qualified name the expensive one

2. The multiplier is eight, not four

3. The negative answers are the ones the cache helps least

4. Five seconds is the resolver, not the application

Resolution

Verification

Prevention