ndots and search paths — the DNS resolution chain

ndots and the search path are the source of DNS latency in many Kubernetes deployments. This lesson walks the resolution chain, the latency implications, and the operational discipline.

The resolution chain

flowchart LR
    A["Query: billing"] --> B{ndots check}
    B -->|"fewer than 5 dots"| C[Search path]
    C --> C1[billing.prod-app.svc.cluster.local]
    C1 -->|NXDOMAIN| C2[billing.svc.cluster.local]
    C2 -->|NXDOMAIN| C3[billing.cluster.local]
    C3 -->|NXDOMAIN| D["Name as given, then upstream DNS"]
    B -->|"5 or more dots, or trailing dot"| D

The resolution chain:

1. The query is billing — no dots, which is fewer than ndots:5, so the resolver treats it as unqualified.
2. The resolver tries the search path in order:
   • billing.prod-app.svc.cluster.local
   • billing.svc.cluster.local
   • billing.cluster.local
3. If all return NXDOMAIN, the name is finally tried as given and goes to the upstream DNS.
4. billing.example.com has two dots, still fewer than five, so it walks the same search path before the external lookup that was actually wanted.
5. billing.example.com. ends in a dot. It is absolute, so the search path is skipped entirely — one query, regardless of ndots.

The rule is one-directional: a name with fewer dots than ndots is tried against the search domains first; a name with ndots dots or more, or a trailing dot, is tried as an absolute name first.

ndots configuration

apiVersion: v1
kind: Pod
metadata:
  name: myapp
spec:
  dnsConfig:
    options:
      - name: ndots
        value: "2"
  containers:
    - name: app
      image: myapp

The ndots option is the dot count at which a name stops being treated as unqualified:

• Default: 5. A name with fewer than five dots is tried against the search domains first.
• Lower value: fewer names go through the search path, so external names such as api.example.com resolve in one query — at the cost of breaking any name that relied on the search path to be completed.
• Higher value: more names go through the search path, so even long external names pay the full walk before the lookup that was wanted.

For most workloads, ndots:2 is a good balance: a single-label Service name such as billing still expands through the search path, while anything with two or more dots is resolved directly.

The search path

apiVersion: v1
kind: Pod
metadata:
  name: myapp
spec:
  dnsConfig:
    searches:
      - prod-app.svc.cluster.local
      - svc.cluster.local
      - cluster.local
  containers:
    - name: app
      image: myapp

The search path can be customised. The default is:

• <namespace>.svc.cluster.local
• svc.cluster.local
• cluster.local

A custom search path can shorten the chain (e.g., add prod-app.local for a specific namespace).

The latency impact

flowchart LR
    A["Pod: myapp"] -->|billing| B[4 DNS queries]
    B -->|5-10ms each| C[20-40ms latency]
    A -->|"billing.prod-app.svc.cluster.local."| D[1 DNS query]
    D -->|5-10ms| E[5-10ms latency]

The latency:

• Short name (billing). 4 queries × 5-10ms = 20-40ms total latency.
• FQDN with a trailing dot (billing.prod-app.svc.cluster.local.). 1 query × 5-10ms = 5-10ms total latency.

The difference: roughly 4x latency reduction. Without the trailing dot the FQDN has four dots, still fewer than five, so it walks the search path too — the saving comes from making the name absolute, not from its length.

Mitigation: nodelocaldns

flowchart LR
    A[Pod] -->|billing| B[nodelocaldns cache]
    B -->|cache hit| C[Immediate response]
    B -->|cache miss| D[CoreDNS]
    D -->|response| B
    B -->|cached| A

nodelocaldns (covered in the previous lesson) caches DNS responses at the node level. For cached queries, the latency is <1ms.

For applications that make many DNS queries (thousands per second), nodelocaldns is essential.

Mitigation: explicit FQDNs

# Use FQDN instead of short name
import psycopg2
conn = psycopg2.connect(
    host="db.prod-data.svc.cluster.local.",  # absolute FQDN
    port=5432,
    database="mydb"
)

Using fully qualified names in application code:

• Eliminates the search path overhead.
• Reduces the DNS query count by roughly 4x.
• Reduces DNS latency by roughly the same factor.

Write the trailing dot — db.prod-data.svc.cluster.local. — so the name is absolute regardless of the Pod’s ndots. The trade-off: the name must be correct; an incorrect namespace in an absolute name fails outright rather than falling back through the search path.

Quiz

The operational discipline

ndots and search path tuning in production rests on five non-negotiable elements:

• Use FQDNs in application code. Eliminates the search path overhead.
• Install nodelocaldns. Caches at the node level.
• Tune ndots. ndots:2 is a good balance.
• Customise the search path. Add namespace-specific entries.
• Profile DNS latency. Measure before and after changes.

DNS latency is invisible until it isn’t. The discipline is to measure and tune.