NXDOMAIN, no endpoints, and stub-domain — the DNS resolution failures

An application that cannot reach billing.prod.svc.cluster.local reports the same symptom whichever way DNS failed, and the three ways fail at different points in the resolution path. NXDOMAIN means the name never resolved; an answer carrying a ClusterIP that nothing responds on means the Service’s EndpointSlice is empty; a stub-domain timeout means cluster-local names are healthy and the upstream server is not. Reading which of the three you have from a single lookup is what decides whether you fix a manifest, a label selector, or CoreDNS itself.

The DNS resolution path

The DNS resolution path is:

flowchart LR
    A[Pod] --> B[resolv.conf]
    B --> C[kube-dns Service]
    C --> D[CoreDNS Pod]
    D --> E{Cluster-local?}
    E -->|Yes| F[API server]
    E -->|No| G[Upstream DNS]

The path is: Pod → kube-dns Service → CoreDNS Pod → API server (for cluster-local) or upstream DNS (for external).

NXDOMAIN

A NXDOMAIN response is the DNS resolver’s signal that the name does not exist. The diagnostic is:

• The name is wrong. The application has a typo in the Service name.
• The namespace is wrong. The application is querying billing.prod.svc.cluster.local but the Service is in the default namespace.
• The Service does not exist. The Service has been deleted.
• The CoreDNS is failing. The CoreDNS Pod is failing to answer the query.

# Substitute your own value before running:
SOURCE_POD=web-5f9c7d8b6c-2xk9p

# Test the DNS
kubectl exec -it "$SOURCE_POD" -- nslookup billing.prod.svc.cluster.local

# Check the Service
kubectl get service billing -n prod

# Check the CoreDNS Pods
kubectl get pods -n kube-system -l k8s-app=kube-dns -o wide

No endpoints

A “no endpoints” response is when the CoreDNS resolves the Service name to a ClusterIP, but the ClusterIP is unreachable because the EndpointSlice is empty (see Part CXXI-02).

# Substitute your own value before running:
SOURCE_POD=web-5f9c7d8b6c-2xk9p

# Test the DNS (returns the ClusterIP)
kubectl exec -it "$SOURCE_POD" -- nslookup billing.prod.svc.cluster.local

# Check the EndpointSlice
kubectl get endpointslices -n prod -l kubernetes.io/service-name=billing

# Check the Pods
kubectl get pods -n prod -l app=billing -o wide

The diagnostic is the EndpointSlice and the Pods.

Stub-domain

A stub-domain is a custom DNS configuration for a specific domain. The CoreDNS forwards queries for the stub-domain to an external DNS server. The stub-domain is configured in the CoreDNS ConfigMap.

.:53 {
    kubernetes cluster.local in-addr.arpa ip6.arpa {
      pods insecure
      fallthrough in-addr.arpa ip6.arpa
    }
    forward . /etc/resolv.conf
    cache 30
    loop
    reload
    loadbalance
}
example.com:53 {
    forward . 1.2.3.4
    cache 30
}

The stub-domain failure is the upstream DNS being unreachable.

# Substitute your own value before running:
SOURCE_POD=web-5f9c7d8b6c-2xk9p

# Test the stub-domain
kubectl exec -it "$SOURCE_POD" -- nslookup db.example.com

# Check the upstream DNS
kubectl exec -it "$SOURCE_POD" -- nslookup db.example.com 1.2.3.4

The diagnostic command sequence

# Substitute your own value before running:
SOURCE_POD=web-5f9c7d8b6c-2xk9p

# 1. Check the Pod's resolv.conf
kubectl exec -it "$SOURCE_POD" -- cat /etc/resolv.conf

# 2. Test the DNS
kubectl exec -it "$SOURCE_POD" -- nslookup billing.prod.svc.cluster.local

# 3. Test the upstream DNS
kubectl exec -it "$SOURCE_POD" -- nslookup google.com

# 4. Check the CoreDNS Pods
kubectl get pods -n kube-system -l k8s-app=kube-dns -o wide

# 5. Check the CoreDNS logs
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=200

# 6. Check the Service
kubectl get service billing -n prod

# 7. Check the EndpointSlice
kubectl get endpointslices -n prod -l kubernetes.io/service-name=billing

The command sequence is the canonical flow extended with the DNS resolution path.

Production discipline

DNS resolution failures are the cluster’s DNS hypothesis. The discipline is to walk the DNS resolution path, identify the failure mode, apply the remediation. The DNS is the cluster’s name resolution; the remediation is the DNS resolution recovery.

• Test cluster-local first. The cluster-local DNS is the CoreDNS’s primary function.
• Test external DNS. The external DNS is the upstream forwarding.
• Check the ConfigMap. The ConfigMap is the CoreDNS’s configuration.

Quiz