Stub domains and upstream resolvers — extending CoreDNS for custom domains

Stub domains and upstream resolvers let CoreDNS forward queries for specific zones to specific upstream resolvers. The use cases are split-horizon DNS, external services, and integration with corporate DNS. This lesson walks the configuration, the use cases, and the operational discipline.

What stub domains are

A stub domain is a DNS zone that is served by a specific upstream resolver. The CoreDNS plugin configuration includes a server block for the stub domain:

.:53 {
    errors
    health
    kubernetes cluster.local in-addr.arpa ip6.arpa {
        pods insecure
        fallthrough in-addr.arpa ip6.arpa
    }
    forward . /etc/resolv.conf
    cache 30
    loop
    reload
}

example.com:53 {
    errors
    cache 30
    forward . 10.0.0.10 10.0.0.11
}

internal.example.com:53 {
    errors
    cache 30
    forward . 10.0.0.20
}

The server block example.com:53 defines the stub domain for example.com. The forward plugin sends queries to the upstream resolvers 10.0.0.10 and 10.0.0.11.

sequenceDiagram
    autonumber
    participant C as Pod
    participant DNS as CoreDNS
    participant U as Upstream
    C->>DNS: billing.example.com
    DNS->>DNS: match server block example.com
    DNS->>U: 10.0.0.10
    U-->>DNS: 192.0.2.10
    DNS-->>C: 192.0.2.10

The server block matches the zone; the forward plugin sends the query to the upstream.

The use cases

The stub domains’ use cases:

• Split-horizon DNS: the same name resolves to different IPs depending on the source. The in-cluster DNS returns the internal IP; the external DNS returns the public IP.
• External services: the cluster needs to resolve names from a corporate DNS (e.g., Active Directory).
• Performance: the upstream resolvers are optimized for specific zones (e.g., a CDN).
• Compliance: the cluster must use specific resolvers for compliance.

The cluster operator must verify the use cases before configuring the stub domains.

The forwarding chain

The forwarding chain is determined by the server block order. The first matching server block handles the query; the next blocks are skipped.

.:53 {
    forward . /etc/resolv.conf
}

example.com:53 {
    forward . 10.0.0.10
}

internal.example.com:53 {
    forward . 10.0.0.20
}

A query for billing.example.com matches the example.com block; the query is forwarded to 10.0.0.10. A query for db.internal.example.com matches the internal.example.com block; the query is forwarded to 10.0.0.20.

The upstream resolvers

The upstream resolvers are configured in the forwarding plugin:

forward . 10.0.0.10 10.0.0.11 {
    max_concurrent 1000
    prefer_udp
}

The options:

• 10.0.0.10 10.0.0.11: the upstream resolvers.
• max_concurrent 1000: the maximum number of concurrent upstream queries.
• prefer_udp: prefer UDP over TCP.

The failure modes

The stub domains’ failure modes:

• Upstream resolver down: the upstream is unavailable. The fix is to verify the upstream.
• Network unreachable: the CoreDNS Pods cannot reach the upstream. The fix is to verify the network.
• Corefile misconfigured: the server block has a syntax error. The fix is to verify the Corefile.
• Cache stale: the cache returns stale records. The fix is to reduce the cache TTL.
• NXDOMAIN flooding: the upstream returns NXDOMAIN for many queries. The fix is to set the cache TTL for denial responses.

The operational discipline

The stub domains’ operational discipline:

• Document the stub domains. The Corefile is the cluster’s DNS configuration.
• Audit the upstream resolvers. The upstream resolvers are the cluster’s external DNS.
• Test the stub domains in staging. The stub domains must work for the workload.
• Monitor the upstream resolvers. The upstream latencies are the leading indicator.
• Plan the stub domains’ evolution. The upstream resolvers can change.
• Document the troubleshooting. The troubleshooting is the cluster’s operational reference.

Quiz

Production discipline

• Document the stub domains. The Corefile is the cluster’s DNS configuration.
• Audit the upstream resolvers. The upstream resolvers are the cluster’s external DNS.
• Test the stub domains in staging. The stub domains must work for the workload.
• Monitor the upstream resolvers. The upstream latencies are the leading indicator.
• Plan the stub domains’ evolution. The upstream resolvers can change.
• Document the troubleshooting. The troubleshooting is the cluster’s operational reference.
• Use a CI check for the Corefile. The CI check can catch the syntax error at every change.
• Train the operations team on the CoreDNS diagnostics. The diagnostics are the team’s tools.
• Document the stub domains’ design. The stub domains are the cluster’s DNS configuration; the documentation is the reference.