The Kubernetes plugin — how CoreDNS serves cluster records

The CoreDNS Kubernetes plugin serves DNS records for Services and Pods. The plugin watches the API for changes; the records are updated automatically. This lesson walks the plugin, the records, and the operational discipline.

What the Kubernetes plugin does

The Kubernetes plugin serves DNS records for Services and Pods based on the cluster.local zone. The plugin watches the API for Services, Endpoints, EndpointSlices, and Pods.

sequenceDiagram
    autonumber
    participant API as Kubernetes API
    participant KP as Kubernetes plugin
    participant DNS as CoreDNS
    participant C as Client
    API->>KP: Service billing added
    KP->>KP: build DNS records
    KP->>DNS: serve A record 10.96.0.10
    C->>DNS: billing.prod-app.svc.cluster.local
    DNS-->>C: 10.96.0.10

The plugin is the cluster’s DNS backend. The records are generated from the API’s state.

The records

The plugin serves the following records for Services:

• A: <svc>.<ns>.svc.cluster.local -> ClusterIP
• AAAA: <svc>.<ns>.svc.cluster.local -> IPv6 ClusterIP (dual-stack)
• SRV: _<port>._<proto>.<svc>.<ns>.svc.cluster.local -> port and target
• PTR: for reverse DNS

The plugin serves the following records for Pods (when enabled):

• A: <pod-ip>.<ns>.pod.cluster.local -> Pod name
• AAAA: IPv6 Pod IP

# Substitute a Pod that has a DNS client installed:
POD=dnsutils-6b8f4c9d5f-7t2qm

# A record
kubectl exec "$POD" -- nslookup billing.prod-app.svc.cluster.local

# SRV record
kubectl exec "$POD" -- dig SRV _http._tcp.billing.prod-app.svc.cluster.local

# PTR record
kubectl exec "$POD" -- dig -x 10.244.1.5

The watch loop

The plugin uses an informer to watch the API:

// Pseudocode
serviceInformer := factory.Core().V1().Services().Informer()
endpointSliceInformer := factory.Discovery().V1().EndpointSlices().Informer()
podInformer := factory.Core().V1().Pods().Informer()

serviceInformer.AddEventHandler(...)
endpointSliceInformer.AddEventHandler(...)
podInformer.AddEventHandler(...)

The plugin maintains an in-memory cache of the DNS records. The cache is updated on every event; the plugin serves the records from the cache.

The plugin configuration

The plugin’s configuration in the Corefile:

kubernetes cluster.local in-addr.arpa ip6.arpa {
    pods insecure
    fallthrough in-addr.arpa ip6.arpa
    ttl 30
    upstream
    noendpoints
}

The options:

• pods insecure: serve Pod records via the <pod-ip>.<namespace>.pod.cluster.local format.
• fallthrough in-addr.arpa ip6.arpa: forward PTR queries to the upstream if not matched.
• ttl 30: the TTL for the records.
• upstream: use the upstream resolvers for external queries.
• noendpoints: do not serve Endpoint records (for performance).

The endpoint records

The plugin serves Endpoint records based on the EndpointSlice. The Endpoint records are used by the DNS client to discover the Pod IPs:

# Substitute a Pod that has a DNS client installed:
POD=dnsutils-6b8f4c9d5f-7t2qm

kubectl exec "$POD" -- dig SRV _http._tcp.billing.prod-app.svc.cluster.local

_http._tcp.billing.prod-app.svc.cluster.local. 30 IN SRV 10 50 80 billing.prod-app.svc.cluster.local.

The SRV record includes the port and the target. The target is the Service’s DNS name; the A record for the target is the ClusterIP.

The plugin’s metrics

The plugin exposes metrics via the Prometheus plugin:

# Substitute the CoreDNS Pod's IP (`kubectl get pods -n kube-system -l k8s-app=kube-dns -o wide`):
COREDNS_IP=192.0.2.53

curl "http://$COREDNS_IP:9153/metrics"

coredns_dns_requests_total{server="dns://:53",zone="cluster.local"} 5000
coredns_dns_responses_total{server="dns://:53",zone="cluster.local",rcode="NOERROR"} 4800
coredns_dns_responses_total{server="dns://:53",zone="cluster.local",rcode="NXDOMAIN"} 100

The metrics show the request count, the response count, and the response code. The cluster operator monitors these metrics.

The failure modes

The Kubernetes plugin’s failure modes:

• API server unreachable: the plugin cannot watch the API. The fix is to restore the API connectivity.
• RBAC misconfigured: the plugin does not have permission to watch Services. The fix is to update the ServiceAccount.
• Watch loop broken: the plugin is not receiving events. The fix is to restart the Pods.
• Stale records: the records are not updated. The fix is to restart the Pods.
• Plugin crash: the plugin’s process is not running. The fix is to restart the Pods.

The operational discipline

The Kubernetes plugin’s operational discipline:

• Document the plugin’s configuration. The plugin is the cluster’s DNS backend.
• Monitor the plugin’s metrics. The metrics are the leading indicator.
• Test the plugin in staging. The plugin must work for the workload.
• Plan the plugin’s evolution. The plugin is the cluster’s DNS backend.
• Document the troubleshooting. The troubleshooting is the cluster’s operational reference.
• Audit the plugin’s RBAC. The ServiceAccount must have the right permissions.

Quiz

Production discipline

• The Kubernetes plugin is the cluster’s DNS backend. The cluster operator must monitor the plugin’s metrics.
• Document the plugin’s configuration. The plugin is the cluster’s DNS backend.
• Monitor the plugin’s metrics. The metrics are the leading indicator.
• Test the plugin in staging. The plugin must work for the workload.
• Plan the plugin’s evolution. The plugin is the cluster’s DNS backend.
• Document the troubleshooting. The troubleshooting is the cluster’s operational reference.
• Audit the plugin’s RBAC. The ServiceAccount must have the right permissions.
• Train the operations team on the plugin’s diagnostics. The diagnostics are the team’s tools.
• Document the plugin’s design. The plugin is the cluster’s DNS backend; the documentation is the reference.