CoreDNS scale and tuning — the resolver at scale

The default CoreDNS deployment is two replicas with 100m of CPU, and it does not grow when the cluster does. A cluster that has quietly gone from 200 Pods to 3000 throttles its resolver long before anyone thinks to look at it, and the symptom arrives as connection timeouts scattered across unrelated services rather than as anything named DNS. The levers are the replica count, the cache TTL, and what the forward plugin does with the queries it cannot answer locally; each has a cost, and the resolver’s own metrics show which one is short.

The CoreDNS scale

The CoreDNS scale is governed by:

• Replicas. The number of CoreDNS Pods. The default is 2.
• CPU/memory. The CoreDNS Pods’ resources. The default is 100m CPU / 256Mi memory.
• Cache. The CoreDNS cache size. The default is 30s TTL.
• Forward. The forward plugin’s behaviour. The default is to forward to /etc/resolv.conf.

flowchart TD
    A[Query rate] --> B{Replicas sufficient?}
    B -->|No| C[Scale CoreDNS]
    B -->|Yes| D{CPU sufficient?}
    D -->|No| E[Increase CPU]
    D -->|Yes| F{Cache sufficient?}
    F -->|No| G[Tune cache]
    F -->|Yes| H[Forward plugin]

The scale is the diagnostic.

The replicas

The CoreDNS replicas are the most common scaling lever. The canonical rule: 1 CoreDNS replica per 1000 Pods.

# Check the current replicas
kubectl get deployment coredns -n kube-system

# Scale the replicas
kubectl scale deployment/coredns -n kube-system --replicas=4

# Autoscale with HPA
kubectl autoscale deployment/coredns -n kube-system --min=2 --max=10 --cpu-percent=80

The replicas are the cluster’s DNS capacity.

The cache

The CoreDNS cache reduces the upstream query rate. The default cache is 30s TTL. The cache is per-Pod, so the cluster-wide cache hit rate is (1 - 1/replicas) * 100%.

.:53 {
    cache 30
}

To tune the cache:

.:53 {
    cache 300
    # or
    cache {
        success 1000
        denial 1000
        prefetch 10
    }
}

The cache is the cluster’s DNS efficiency.

The forward plugin

The forward plugin forwards queries to upstream DNS. The default is to forward to /etc/resolv.conf (the node’s upstream DNS). The forward can be tuned to use specific upstream servers.

.:53 {
    forward . 1.1.1.1 8.8.8.8 {
        prefer_udp
        max_fails 5
        expire 30s
    }
}

The forward is the cluster’s external DNS path.

The diagnostic

The canonical diagnostic:

# 1. Check the CoreDNS replicas
kubectl get deployment coredns -n kube-system

# 2. Check the CoreDNS CPU/memory
kubectl top pods -n kube-system -l k8s-app=kube-dns

# 3. Check the CoreDNS metrics
COREDNS_POD=$(kubectl get pods -n kube-system -l k8s-app=kube-dns \
  -o jsonpath='{.items[0].metadata.name}')
kubectl port-forward -n kube-system "$COREDNS_POD" 9153:9153
# Browse to http://localhost:9153/metrics

# 4. Check the CoreDNS logs
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=200

# 5. Check the CoreDNS cache hit rate
# (from the metrics: coredns_cache_hits_total / coredns_cache_lookups_total)

The diagnostic is the replicas, the CPU/memory, the metrics, and the logs.

Common failures

• Replicas insufficient. The CoreDNS Pods are CPU-throttled; the query rate is high.
• Cache insufficient. The cache hit rate is low; the upstream DNS is overwhelmed.
• Forward plugin failing. The upstream DNS is unreachable; the forward fails.

Production discipline

CoreDNS at scale is the cluster’s DNS hypothesis. The discipline is to scale the replicas, tune the cache, monitor the metrics. The DNS is the cluster’s name resolution; the remediation is the tuning.

• Scale the replicas. The canonical rule is 1 per 1000 Pods.
• Tune the cache. The cache hit rate is the cluster’s DNS efficiency.
• Tune the forward. The forward is the cluster’s external DNS path.

Quiz