KubernetesCXXI · Service TroubleshootingService troubleshooting
Headless and ExternalName — the DNS-only Services
What you'll learn
- Apply the 11-step methodology to headless and ExternalName Services
- Distinguish a headless Service from a regular ClusterIP
- Diagnose ExternalName DNS failures
- Identify the production failure modes of headless and ExternalName Services
Prerequisites
Verified against Kubernetes 1.34.x · kubeadm 1.34.x · kubectl 1.34.x · etcd 3.6.x · CoreDNS 1.11.x · containerd 1.7.x / 2.x · 2026-08-16
A headless Service has no ClusterIP and an ExternalName Service has no endpoints at all, so neither of them produces a kube-proxy rule anywhere in the cluster. Everything they do happens in DNS: the answer is a list of Pod IPs in one case and a CNAME in the other. That makes the failure surface small and unusual — an empty answer, a stale answer, or a CNAME that resolves nowhere — and this lesson walks each of them.
The headless Service
A headless Service is one with clusterIP: None. The
EndpointSlice controller publishes the Pod IPs directly to
the Service’s DNS. The application’s DNS query returns the
Pod IPs, not a ClusterIP.
flowchart LR
A[Application] --> B[DNS query]
B --> C[Pod IPs]
C --> D[Direct routing]
The kube-proxy is not in the path. The application connects directly to the Pod IPs.
A real headless Service:
apiVersion: v1
kind: Service
metadata:
name: billing
namespace: prod
spec:
clusterIP: None
selector:
app: billing
ports:
- port: 80
targetPort: 8080
The diagnostic is the DNS:
# Substitute your own values before running:
SOURCE_POD=checkout-6b9f4c7d55-2xk9p
kubectl exec -it "$SOURCE_POD" -- nslookup billing.prod.svc.cluster.local
The response returns the Pod IPs:
Server: 10.96.0.10
Address 1: 10.96.0.10 kube-dns.svc.cluster.local
Name: billing.prod.svc.cluster.local
Address 1: 10.244.5.23
Address 2: 10.244.5.24
Address 3: 10.244.7.10
Address 4: 10.244.7.11
The headless Service’s DNS returns the Pod IPs directly.
The ExternalName Service
An ExternalName Service is one with kind: ExternalName. The
Service’s DNS is a CNAME to an external DNS name. The
application’s DNS query returns the external name.
flowchart LR
A[Application] --> B[DNS query]
B --> C[CNAME: external-db.example.com]
C --> D[External resolution]
The kube-proxy is not in the path. The application connects to the external service.
A real ExternalName Service:
apiVersion: v1
kind: Service
metadata:
name: external-db
namespace: prod
spec:
type: ExternalName
externalName: db.example.com
The diagnostic is the DNS:
# Substitute your own values before running:
SOURCE_POD=checkout-6b9f4c7d55-2xk9p
kubectl exec -it "$SOURCE_POD" -- nslookup external-db.prod.svc.cluster.local
The response returns the CNAME:
Server: 10.96.0.10
Address 1: 10.96.0.10 kube-dns.svc.cluster.local
Name: external-db.prod.svc.cluster.local
external-db.prod.svc.cluster.local canonicalName = db.example.com.
The ExternalName Service’s DNS returns the CNAME.
Headless failures
The most common headless failures:
- No Pods Ready. The EndpointSlice is empty because no Pods match the selector and are Ready.
- Selector mismatch. The Service’s selector does not match the Pods’ labels.
- DNS cache. The application’s DNS cache is stale.
# Substitute your own values before running:
SOURCE_POD=checkout-6b9f4c7d55-2xk9p
# Diagnose the headless Service
kubectl get endpointslices -n prod -l kubernetes.io/service-name=billing
# Test the DNS
kubectl exec -it "$SOURCE_POD" -- nslookup billing.prod.svc.cluster.local
# Clear the DNS cache (in the application)
kubectl exec -it "$SOURCE_POD" -- nscd -i hosts
ExternalName failures
The most common ExternalName failures:
- External DNS not reachable. The CoreDNS Pod cannot reach the external DNS server.
- External service is down. The external service is down.
- DNS cache. The application cached the CNAME’s old target and keeps dialing the previous host until the TTL expires.
# Substitute your own values before running:
SOURCE_POD=checkout-6b9f4c7d55-2xk9p
# Test the DNS
kubectl exec -it "$SOURCE_POD" -- nslookup external-db.prod.svc.cluster.local
# Test the external DNS
kubectl exec -it "$SOURCE_POD" -- nslookup db.example.com 8.8.8.8
# Test the external service
kubectl exec -it "$SOURCE_POD" -- curl -v http://db.example.com
Production discipline
A headless or ExternalName Service failure is the cluster’s DNS hypothesis. The discipline is to walk the DNS-only path, identify the failure mode, apply the remediation. The flow is shorter; the discipline is the same.
- Walk the DNS-only path. Headless and ExternalName Services bypass the kube-proxy.
- Test the Pod readiness for headless. The EndpointSlice is the source of truth for the Pod IPs.
- Test the external DNS for ExternalName. The CNAME is the source of truth.
Quiz
Knowledge check · 4 questions
Q1. What is the difference between a headless Service and a regular ClusterIP?
Q2. An ExternalName Service's DNS returns a CNAME to an external DNS name.
Q3. An operator runs `kubectl exec -it <source-pod> -- nslookup billing.prod.svc.cluster.local`. The response is NXDOMAIN. The Service is headless with selector `app=billing`. The Pods are Running with `app=billing`. What is the diagnostic and remediation?
The Service is `billing` in namespace `prod`. The Service is headless (clusterIP: None). The Pods are `billing-7d8f-abcde` and `billing-7d8f-def01` in namespace `prod`. The Pods are Running with `app=billing`. The DNS query returns NXDOMAIN.
Q4. Name the diagnostic command for a headless Service and an ExternalName Service.
Passing score: 75%. Answers are checked in this browser.