External DNS and split-horizon — the cluster to enterprise DNS

Nothing inside the cluster notices when external DNS records go stale. The Service is healthy, the load balancer holds an address, and the only symptom is that clients outside the cluster still resolve last week’s IP, because the external-dns controller’s provider credentials expired or its API calls are being rate-limited. Split-horizon makes that harder to catch: internal and external clients are answered differently by design, so resolving the name from inside a Pod proves nothing about what the outside world sees.

The external-dns controller

The external-dns controller is a Deployment that watches the cluster’s Services and Ingresses, and publishes the DNS records to the enterprise DNS provider (e.g., Route53, Cloud DNS, Azure DNS).

flowchart LR
    A[Service/Ingress] --> B[external-dns controller]
    B --> C[DNS provider API]
    C --> D[Enterprise DNS]
    D --> E[External clients]

The external-dns controller is the cluster’s DNS publication.

The DNS annotations

The external-dns controller uses the Service’s annotations to determine the DNS records to publish:

apiVersion: v1
kind: Service
metadata:
  name: billing
  namespace: prod
  annotations:
    external-dns.alpha.kubernetes.io/hostname: billing.example.com
    external-dns.alpha.kubernetes.io/ttl: "300"
spec:
  type: LoadBalancer
  ...

The annotations are the cluster’s DNS configuration.

The split-horizon DNS

The split-horizon DNS is a DNS configuration that returns different answers based on the client’s network. The cluster-internal clients get the ClusterIP; the external clients get the cloud LB’s IP.

flowchart TD
    A[Client query: billing.example.com] --> B{Client network?}
    B -->|Internal| C[Cluster-IP: 10.96.45.123]
    B -->|External| D[LoadBalancer IP: 1.2.3.4]

The split-horizon is the cluster’s DNS view.

The diagnostic

The canonical diagnostic:

# Substitute your own value before running: a pod in the cluster to run the
# lookup from.
SOURCE_POD=checkout-api-5f9c7d8b6c-2xk9p

# 1. Check the external-dns controller
kubectl get pods -n kube-system -l app=external-dns

# 2. Check the external-dns logs
kubectl logs -n kube-system -l app=external-dns --tail=200

# 3. Check the Service's annotations
kubectl get service billing -n prod -o yaml | grep -A5 annotations

# 4. Test the DNS from inside the cluster
kubectl exec -it "$SOURCE_POD" -- nslookup billing.prod.svc.cluster.local

# 5. Test the DNS from outside the cluster
nslookup billing.example.com

The diagnostic is the external-dns controller, the annotations, and the tests.

Common failures

• external-dns controller failing. The controller is not publishing the records. The remediation is to restart the controller.
• DNS provider authentication failing. The controller’s credentials are not valid. The remediation is to update the credentials.
• DNS provider rate limiting. The controller is exceeding the provider’s API rate limit. The remediation is to reduce the publish rate.
• Split-horizon misconfiguration. The internal and external clients are getting the wrong DNS records.

# Restart the external-dns controller
kubectl rollout restart deployment/external-dns -n kube-system

# Check the controller's logs for DNS provider errors
kubectl logs -n kube-system -l app=external-dns --tail=200 | grep -i "error"

The remediation

The remediation depends on the cause:

# Option 1: Restart the external-dns controller
kubectl rollout restart deployment/external-dns -n kube-system

# Option 2: Update the credentials
kubectl edit secret external-dns-credentials -n kube-system

# Option 3: Reduce the publish rate
kubectl edit deployment/external-dns -n kube-system
# Change --interval to a higher value

# Option 4: Fix the split-horizon
# (DNS provider-specific configuration)

The remediation is the DNS integration recovery.

Production discipline

External DNS is the cluster’s hypothesis. The discipline is to walk the canonical flow extended with the DNS integration, identify the failure mode, apply the remediation. The DNS is the cluster’s name resolution; the integration is the bridge.

• Check the external-dns controller. The controller is the cluster’s DNS publication.
• Check the credentials. The credentials are the controller’s authentication.
• Check the DNS provider. The provider is the cluster’s DNS destination.

Quiz