CoreDNS autoscaling and tuning — performance for the cluster

CoreDNS autoscaling is managed by the cluster-proportional-autoscaler. The autoscaler scales the CoreDNS Deployment based on the number of nodes and cores. The tuning of the cache, the upstream concurrency, and the memory is critical for performance at scale. This lesson walks the autoscaling, the tuning, and the operational discipline.

The autoscaler

The cluster-proportional-autoscaler scales the CoreDNS Deployment based on the cluster’s size:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: coredns-autoscaler
  namespace: kube-system
spec:
  template:
    spec:
      containers:
        - name: autoscaler
          image: registry.k8s.io/cpa/cluster-proportional-autoscaler:1.8.6
          args:
            - --namespace=kube-system
            - --configmap=coredns-autoscaler
            - --target=Deployment/coredns
            - --default-params={"linear":{"coresPerReplica":256,"nodesPerReplica":16,"preventSinglePointFailure":true}}

The configuration:

• coresPerReplica: 256: one replica per 256 cores.
• nodesPerReplica: 16: one replica per 16 nodes.
• preventSinglePointFailure: true: ensure at least 2 replicas.

The cluster operator can tune the autoscaler based on the cluster’s size.

The metrics

The autoscaler is driven by the number of nodes and cores. The cluster operator can monitor the CoreDNS metrics to verify the autoscaler’s decisions:

# The IP of a CoreDNS Pod (kubectl get pods -n kube-system -l k8s-app=kube-dns -o wide):
COREDNS_IP=192.0.2.31

curl "http://$COREDNS_IP:9153/metrics"

coredns_dns_requests_total{server="dns://:53",zone="."} 10000
coredns_dns_responses_total{server="dns://:53",zone=".",rcode="NOERROR"} 9500
coredns_cache_hits_total{server="dns://:53",zone="."} 5000
coredns_cache_misses_total{server="dns://:53",zone="."} 1000

The metrics show the request rate, the response rate, the cache hit ratio, and the upstream latency. The cluster operator must monitor these metrics.

The cache tuning

The cache plugin’s configuration affects the performance:

cache 30 {
    success 9984 30
    denial 9984 5
}

The options:

• success 9984 30: cache 9984 successful responses with a TTL of 30 seconds.
• denial 9984 5: cache 9984 denial responses with a TTL of 5 seconds.

The cluster operator can tune the cache based on the workload:

• Larger cache: more memory, more cache hits.
• Smaller cache: less memory, fewer cache hits.
• Longer TTL: less upstream load, more stale records.
• Shorter TTL: more upstream load, fresher records.

The upstream concurrency

The forward plugin’s max_concurrent option controls the maximum number of concurrent upstream queries:

forward . /etc/resolv.conf {
    max_concurrent 1000
}

The default is 1000. The cluster operator can tune the concurrency based on the upstream’s capacity:

• Higher concurrency: more upstream load, lower latency.
• Lower concurrency: less upstream load, higher latency.

The memory tuning

The CoreDNS Pods have a memory limit:

resources:
  limits:
    memory: 170Mi
  requests:
    cpu: 100m
    memory: 70Mi

The default memory limit is 170Mi. The cluster operator can tune the memory based on the cluster’s size:

• More memory: more cache, fewer evictions.
• Less memory: less cache, more evictions.

The failure modes

The CoreDNS scaling’s failure modes:

• Autoscaler not scaling: the autoscaler is not scaling. The fix is to verify the autoscaler.
• Cache too small: the cache evictions are high. The fix is to increase the memory.
• Upstream saturated: the upstream is overwhelmed. The fix is to reduce the concurrency.
• Sync loop slow: the CoreDNS is slow to reload. The fix is to investigate the API server.
• Memory pressure: the CoreDNS Pods are evicted. The fix is to increase the memory.

The operational discipline

The CoreDNS scaling’s operational discipline:

• Document the autoscaler. The autoscaler is the cluster’s scaling configuration.
• Monitor the metrics. The metrics are the leading indicator.
• Tune the cache. The cache is the performance optimization.
• Tune the upstream concurrency. The concurrency is the upstream load.
• Tune the memory. The memory is the cache size.
• Test the scaling in staging. The scaling must work for the workload.
• Plan the scaling’s evolution. The scaling is the cluster’s DNS performance.

Quiz

Production discipline

• The autoscaler is the cluster’s scaling configuration. The cluster operator must tune the autoscaler.
• Document the autoscaler. The autoscaler is the cluster’s scaling configuration.
• Monitor the metrics. The metrics are the leading indicator.
• Tune the cache. The cache is the performance optimization.
• Tune the upstream concurrency. The concurrency is the upstream load.
• Tune the memory. The memory is the cache size.
• Test the scaling in staging. The scaling must work for the workload.
• Plan the scaling’s evolution. The scaling is the cluster’s DNS performance.
• Document the scaling’s design. The scaling is the cluster’s DNS performance; the documentation is the reference.
• Train the operations team on the scaling diagnostics. The diagnostics are the team’s tools.