Custom DNS for service discovery · Docker & Containers

The embedded resolver handles service-to-service resolution within a Docker network, on one host, with no configuration. Past that boundary — several hosts, containers that must resolve VMs, split-horizon answers, an external registry — you need a real DNS server, and the usual choice is CoreDNS.

Adding it is not hard. Deploying it correctly runs into three problems that are specific to doing this inside Docker, and all three are more interesting than the Corefile.

First: check that you actually need it

Reach for a DNS server when you have one of these:

Multi-host. A service on host A is not in host B’s registry. The embedded resolver is per-daemon and has no cross-host view.
Containers must resolve non-containers. VMs, bare-metal databases, appliances. Your internal DNS knows them; the embedded resolver forwards to it and this already works — which is why this reason is usually not one.
Split-horizon. The same name must answer differently inside and outside.
An external registry — Consul, etcd, a cloud service registry — is the source of truth for what exists.

Not reasons to add it:

“Container names should have a domain suffix.” Network aliases do that.
“I want round-robin across replicas.” You already have that, and it is not health-aware in CoreDNS either unless you add health checking.
“I want DNS-based failover.” DNS is a poor failover mechanism whoever serves it. Put a proxy in front.

Every one of those is a resolver you now have to keep running, and it sits on the critical path of every service in the stack.

Problem one: the resolver needs an address, not a name

You are configuring what resolves names. So you cannot refer to the resolver by name. Give it a fixed address on a user-defined network with an explicit subnet, and point everything at that.

services:
  coredns:
    image: coredns/coredns:1.11.3
    command: ["-conf", "/etc/coredns/Corefile"]
    volumes:
      - ./Corefile:/etc/coredns/Corefile:ro
    networks:
      dnsnet:
        ipv4_address: 172.28.0.53
    restart: unless-stopped

  app:
    image: myapp:1.0.0
    dns:
      - 172.28.0.53
    networks:
      - dnsnet
    depends_on:
      - coredns

networks:
  dnsnet:
    ipam:
      config:
        - subnet: 172.28.0.0/24

Three details in that file are the ones people get wrong:

command:, not args:. There is no args attribute in the Compose specification. A service with args: is not passing arguments to anything; Compose rejects the key, or in older tooling ignores it, and the container runs its default command.
The static address requires an explicit subnet:. ipv4_address on a network with no IPAM configuration is an error, because Docker has no guarantee the address is in the range it allocated.
dns: sets the forwarding target of the embedded resolver. It does not replace 127.0.0.11 in the container’s resolv.conf, and container names on dnsnet keep resolving through the embedded resolver as before. This is usually what you want: the embedded registry stays authoritative for container names, and CoreDNS handles everything it does not know.

Problem two: port 53 is already taken

The instinct is to publish CoreDNS on the host so that host processes and other machines can use it. On a modern Debian or Ubuntu host that collides.

Read-only / Safewhat already owns 53— systemd-resolved's stub listener. It binds 127.0.0.53:53, not 0.0.0.0:53.

$ sudo ss -ulnp 'sport = :53'

UNCONN 0  0  127.0.0.53%lo:53  0.0.0.0:*  users:(("systemd-resolve",pid=812,fd=13))

Illustrative output

The stub binds one loopback address, so -p 53:53 on 0.0.0.0 often succeeds — and then you have two resolvers on the host answering on different addresses, with the host using one and the containers using the other. Which is worse than a clean conflict, because nothing fails.

Do not publish port 53 unless the host needs it

If only containers use CoreDNS, do not publish it at all. Containers reach 172.28.0.53 over the Docker network; publishing adds nothing except exposure. An open recursive resolver on a public address is an amplification reflector, and it will be found — DNS amplification scanning is continuous and indiscriminate.

If host processes genuinely need it, bind to a specific address rather than all of them:

ports: ["127.0.0.1:5353:53/udp", "127.0.0.1:5353:53/tcp"]

and point the host’s resolver at 127.0.0.1:5353 explicitly. Note that both protocols are needed. Publishing only 53/udp works for small answers and fails for anything over 512 bytes, where the server sets the truncated bit and the client is expected to retry over TCP. The symptom is that most names resolve and a few — usually the ones with many records — time out.

Problem three: the forwarding loop

Here is the Corefile that looks obviously right and is not:

. {
    forward . /etc/resolv.conf
    cache 30
    log
    errors
}

forward . /etc/resolv.conf tells CoreDNS to use the nameservers in its own resolv.conf. Inside a container on a user-defined network, that file contains nameserver 127.0.0.11. The embedded resolver forwards what it does not know to the host’s upstream — but if the daemon has been configured to point at this CoreDNS instance, the query comes straight back.

CoreDNS asks the embedded resolver, which asks CoreDNS. Every external lookup in the stack now consumes resources at both ends until the client times out.

The loop plugin exists for exactly this: it detects the condition at startup and refuses to start rather than serving a broken resolver.

Configuration changea Corefile that does not loop— Name the upstream resolvers explicitly. Never forward to your own resolv.conf inside a container.

# ./Corefile
. {
  # Explicit upstreams. Not /etc/resolv.conf.
  forward . 192.0.2.53 198.51.100.53 {
      policy sequential
      health_check 5s
  }
  cache 30
  loop
  log
  errors
  reload
}

# Split-horizon: answer internal names ourselves
internal.example.com {
  file /etc/coredns/internal.example.com.zone
  log
  errors
}

loop is cheap insurance and belongs in every Corefile. reload picks up Corefile changes without a restart. health_check on the forward plugin is what stops a dead upstream from absorbing a third of your queries — the weakness the embedded resolver has and cannot fix.

Two resolvers in the path, and which one answers what

With dns: [172.28.0.53] on a container attached to a user-defined network, a lookup traverses both resolvers, in a fixed order:

The application calls getaddrinfo, which reads /etc/hosts first.
Failing that, it queries 127.0.0.11 — still the only nameserver in the container’s resolv.conf.
The embedded resolver checks the daemon registry: is this a container name or alias on one of this container’s networks? If so, it answers, and CoreDNS never sees the query.
Otherwise it forwards to 172.28.0.53.
CoreDNS applies its own plugin chain and forwards what it cannot answer to the real upstreams.

This layering is a feature. Container names stay authoritative in the registry, where they are always current, and CoreDNS never has to know that containers exist. It also means a CoreDNS zone entry for a name that is also a container name is unreachable — step 3 answers first, every time.

That is the one to remember when a zone file appears not to be working. Confirm which resolver answered before editing the zone again: dig @127.0.0.11 name and dig @172.28.0.53 name will disagree, and the first one is what your applications get.

Forwarding to a service registry

Consul serves DNS on port 8600 for the .consul zone. Forward only that zone, not everything:

consul. {
    forward . 192.0.2.10:8600
    cache 5
    errors
}

. {
    forward . 192.0.2.53 198.51.100.53
    cache 30
    loop
    errors
}

Scoping the stanza to consul. matters. A single . block forwarding everything to Consul makes the service registry a hard dependency for resolving github.com, and its outage becomes a total DNS outage.

Keep the cache TTL low for the registry zone — service registration changes in seconds, and a 30-second cache is 30 seconds of traffic to an instance that has deregistered.

What happens when the CoreDNS container is down?

Everything pointed at it stops resolving external names. Container-to- container resolution keeps working, because the embedded resolver answers those from the registry without forwarding.

That split is the confusing part at 03:00: the stack looks internally healthy, services find each other, and every outbound call fails with a name resolution error. It reads as an internet problem rather than a DNS one.

Three things reduce the blast radius, in order of value:

restart: unless-stopped on the CoreDNS service. Most of these incidents are a container that exited and stayed exited.
A second upstream in the containers’ dns: list — a real resolver, not a second CoreDNS on the same host. It is only useful for the unreachable case, since an NXDOMAIN from the first ends the search, but unreachable is exactly this case.
Do not put CoreDNS on the same host it resolves for, if you can avoid it. A resolver that shares a failure domain with its clients is not redundancy.

Verify the failure mode deliberately: stop the container and confirm what still works. Doing that in a maintenance window is much cheaper than discovering the shape of it during an incident.

What happens if you point the Docker daemon at CoreDNS instead of individual containers?

Setting "dns": ["172.28.0.53"] in daemon.json applies to every container, which is tidy — and creates a startup ordering problem, because CoreDNS is itself a container.

While CoreDNS is starting, every other container’s forwarding target is unreachable. Anything that resolves a name during its own start-up fails. With restart policies the stack eventually converges, but a cold boot produces a burst of crash-looping services and a set of logs that all blame DNS without any of them being the cause.

It also creates the loop described above if CoreDNS’s own resolv.conf is used as its forward target, since the daemon has now pointed the embedded resolver at CoreDNS.

Prefer per-service dns: for the services that need it, and leave the daemon default pointing at a resolver that exists before Docker does.

Sanity check

dig @127.0.0.11 <peer-service> from inside a container answers from the registry, and dig @<coredns-addr> example.com answers from CoreDNS. You checked both, separately.
CoreDNS is not publishing port 53 on 0.0.0.0.
The Corefile contains loop, and CoreDNS started, which means it is not forwarding to itself.
You have stopped the CoreDNS container once, on purpose, and know exactly what broke.