DNS latency and nodelocaldns — caching at the node level

DNS latency is a common production issue. This lesson walks the sources, the measurement, nodelocaldns as the mitigation, and the discipline.

The latency sources

flowchart LR
    A[Sources of latency] --> B[Search path overhead]
    A --> C[Upstream latency]
    A --> D[CoreDNS load]
    A --> E[Network round trips]
    B --> B1[5+ queries per short name]
    C --> C1[5-50ms per query]
    D --> D1["CPU/memory pressure"]
    E --> E1[Pod to CoreDNS network]

The latency sources:

• Search path overhead. Short names trigger 5+ queries.
• Upstream latency. Each query is a round trip to upstream DNS (5-50ms).
• CoreDNS load. Many queries stress CoreDNS CPU; latency increases.
• Network round trips. Pod to CoreDNS network; sub-millisecond in a VPC, higher across regions.

Measuring DNS latency

# Inside a Pod
kubectl exec -it myapp-pod -- nslookup kubernetes.default

Server:    10.96.0.10
Address:   10.96.0.10#53

Name:   kubernetes.default.svc.cluster.local
Address: 10.96.0.1

Measure the time:

time kubectl exec myapp-pod -- nslookup billing

real    0m0.025s

25ms for one short-name query. With many queries, the total is significant.

CoreDNS metrics

coredns_dns_request_count_total
coredns_dns_request_duration_seconds_sum
coredns_dns_request_duration_seconds_count

The CoreDNS metrics (exposed on :9153):

• coredns_dns_request_count_total — total requests by type.
• coredns_dns_request_duration_seconds_sum — total duration.
• coredns_dns_request_duration_seconds_count — total count.

Average latency:

sum(rate(coredns_dns_request_duration_seconds_sum[5m])) /
sum(rate(coredns_dns_request_duration_seconds_count[5m]))

Alert when average latency exceeds 50ms.

nodelocaldns

flowchart LR
    A[Pod] -->|169.254.20.10| B[nodelocaldns on node]
    B -->|cache hit| C[<1ms response]
    B -->|cache miss| D[CoreDNS]
    D -->|response| B
    B -->|cached| A

nodelocaldns architecture:

• A DaemonSet runs on every node.
• Pods are configured to query the node-local cache (169.254.20.10) instead of CoreDNS directly.
• Cache hit: <1ms response.
• Cache miss: query CoreDNS, cache the response.

The installation:

kubectl apply -f https://k8s.io/examples/admin/dns/nodelocaldns.yaml

The cache configuration

# nodelocaldns ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
  name: nodelocaldns
data:
  Corefile: |
    cluster.local:53 {
        errors
        cache {
            success 9984 30
            denial 9984 5
        }
        forward . __PILLAR__UPSTREAM__ {
            prefer_health
        }
    }

The cache TTL:

• success 9984 30. Cache successful responses for 9984 seconds (one cache TTL); per-record 30 seconds (the inner TTL).
• denial 9984 5. Cache denials for 9984 seconds; per-record 5 seconds.

The outer TTL is the cache lifetime; the inner TTL is the upstream’s TTL. The cache serves the inner TTL.

The configuration

# Pod spec with nodelocaldns
dnsConfig:
  nameservers:
    - 169.254.20.10
  searches:
    - prod-app.svc.cluster.local
    - svc.cluster.local
    - cluster.local
  options:
    - name: ndots
      value: "2"

The Pod is configured to use 169.254.20.10 (the nodelocaldns IP) as the primary nameserver. The search path and ndots are tuned as before.

Quiz

The operational discipline

DNS latency mitigation in production rests on five non-negotiable elements:

• Measure DNS latency. Use CoreDNS metrics or dig timings.
• Install nodelocaldns. Reduces CoreDNS load and improves latency.
• Tune ndots and search path. ndots:2 is a good balance.
• Use FQDNs in application code. Eliminates search path overhead.
• Profile before and after. Verify the improvements.

DNS latency is production performance. The discipline is to measure, optimise, and verify.