Search path and ndots — the DNS client configuration

The Pod’s resolv.conf is the source of truth for the DNS client. The ndots and search path are the diagnostic. The remediation is the DNS client configuration. The discipline is the same canonical flow extended with the DNS client.

The Pod’s resolv.conf

The Pod’s resolv.conf is the DNS client configuration. The file is generated by the kubelet based on the Pod’s DNS policy and the namespace’s DNS configuration.

# Substitute your own value before running:
SRC_POD=checkout-7d4f8c9b5-x2kqn

kubectl exec -it "$SRC_POD" -- cat /etc/resolv.conf

A real Pod’s resolv.conf:

nameserver 10.96.0.10
search prod.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

The components are:

• nameserver: the IP address of the kube-dns Service (10.96.0.10).
• search: the search path for short names.
• options ndots:5: the threshold for direct queries.

The search path

The search path is the list of domains that the DNS resolver appends to a short name. A query for billing becomes:

• billing.prod.svc.cluster.local.
• billing.svc.cluster.local.
• billing.cluster.local.
• billing. (the absolute name)

The resolver attempts each in order until one resolves.

flowchart TD
    A[Query: billing] --> B[billing.prod.svc.cluster.local.]
    B --> C{Found?}
    C -->|No| D[billing.svc.cluster.local.]
    D --> E{Found?}
    E -->|No| F[billing.cluster.local.]
    F --> G{Found?}
    G -->|No| H[billing.]
    H --> I{Found?}
    I -->|No| J[NXDOMAIN]

The search path is the cause of 5 queries for a single name.

The ndots option

The ndots option is the threshold for direct queries. If the name has fewer dots than ndots, the resolver tries the search path first. If the name has at least ndots dots, the resolver tries the absolute name first.

With ndots:5, the name billing.prod.svc.cluster.local has 4 dots, which is less than 5. The resolver tries the search path first: billing.prod.svc.cluster.local.prod.svc.cluster.local., which is invalid. Then billing.prod.svc.cluster.local.svc.cluster.local., etc. The resolver makes 5 queries to find the name.

flowchart TD
    A[Query: billing.prod.svc.cluster.local] --> B{ndots:5?}
    B -->|Yes| C[search path first]
    C --> D[billing.prod.svc.cluster.local.prod.svc.cluster.local.]
    D --> E{NXDOMAIN?}
    E -->|Yes| F[billing.prod.svc.cluster.local.svc.cluster.local.]
    F --> G{NXDOMAIN?}
    G -->|Yes| H[billing.prod.svc.cluster.local.cluster.local.]
    H --> I{NXDOMAIN?}
    I -->|Yes| J[billing.prod.svc.cluster.local.]
    J --> K{Found?}
    K -->|Yes| L[Return]

The ndots option is the cause of 5 queries for a single name.

The diagnostic

The canonical diagnostic:

# Substitute your own value before running:
SRC_POD=checkout-7d4f8c9b5-x2kqn

# 1. Check the Pod's resolv.conf
kubectl exec -it "$SRC_POD" -- cat /etc/resolv.conf

# 2. Test the DNS query
kubectl exec -it "$SRC_POD" -- nslookup billing

# 3. Test the absolute name
kubectl exec -it "$SRC_POD" -- nslookup billing.prod.svc.cluster.local

# 4. Test the search path
kubectl exec -it "$SRC_POD" -- nslookup billing.prod.svc.cluster.local.prod.svc.cluster.local

The diagnostic is the resolv.conf and the query response.

The ndots:5 anti-pattern

The ndots:5 default is an anti-pattern. The default causes 5 queries for every cluster-local name. The remediation is to either:

1. Use the absolute name. The application uses billing.prod.svc.cluster.local instead of billing.
2. Set ndots:2. The Pod’s dnsConfig sets ndots: 2, which is enough for cluster-local names.

apiVersion: v1
kind: Pod
metadata:
  name: billing
spec:
  dnsConfig:
    options:
    - name: ndots
      value: "2"
  containers:
  - name: billing
    image: registry.example.com/billing:1.2.3

The remediation is the DNS client configuration.

Common failures

• Search path is wrong. The Pod’s namespace is wrong. kubectl exec -it <source-pod> -- cat /etc/resolv.conf shows the search path.
• ndots is too high. The Pod makes 5 queries for every name. The CoreDNS is overwhelmed.
• Search path is too long. The search path has too many entries; the resolver makes too many queries.

Production discipline

The Pod’s resolv.conf is the cluster’s DNS client. The discipline is to walk the canonical flow extended with the DNS client, identify the failure mode, apply the remediation. The DNS is the cluster’s name resolution; the remediation is the DNS client configuration.

• Check the Pod’s resolv.conf. The resolv.conf is the DNS client configuration.
• Diagnose the search path. The search path is the cause of multiple queries.
• Diagnose the ndots. The ndots is the cause of 5 queries per name.

Quiz