Break/Fix 18: Containers lose external DNS after a host network change

Reported symptoms

A netplan change lands on the Docker hosts during a routine network migration. Nothing appears to break — the hosts resolve names, the running containers keep serving.

The next deploy fails. The recreated container cannot reach the payment provider:

Post "https://api.example.com/v1/charge": dial tcp: lookup
api.example.com on 127.0.0.11:53: server misbehaving

The error names Docker’s embedded resolver, so the first hour goes into Docker’s DNS. Meanwhile the same container resolves postgres and redis on its Compose network without any trouble, which makes the embedded resolver look intermittently broken rather than misconfigured — the worst possible signal.

Diagnosis

Establish which half of resolution works. That split is the whole diagnosis.

docker exec app getent hosts postgres
docker exec app getent hosts api.example.com

Internal succeeds, external fails. The embedded resolver is answering for names it owns and failing to forward the rest, so look at what it is forwarding to.

docker exec app cat /etc/resolv.conf
readlink -f /etc/resolv.conf
cat /run/systemd/resolve/stub-resolv.conf

# inside the container
nameserver 127.0.0.11
options ndots:0

# on the host
/run/systemd/resolve/stub-resolv.conf
nameserver 127.0.0.53
options edns0 trust-ad

There it is. The host’s resolv.conf names a loopback address. That address is the systemd-resolved stub, reachable only in the host’s network namespace. Inside the container, 127.0.0.53 is the container’s own loopback, where nothing is listening.

Confirm what the daemon did about it:

journalctl -u docker --since -2h --no-pager | grep -i 'nameserver\|dns'

level=warning msg="Local (127.0.0.53) DNS resolver found in resolv.conf
and containers can't use it. Using default external servers: [nameserver
8.8.8.8 nameserver 8.8.4.4]"

The daemon told you, at container-create time, hours before the outage, in a log nobody reads. On a host with an egress firewall, those public fallbacks are unreachable, so the forwarder has no working upstream at all.

Prove the path rather than the resolver:

docker run --rm --dns 192.0.2.53 alpine getent hosts api.example.com

Success here means routing, firewall and the container network are all fine, and the only defect is the resolver list.

Resolution path

1. Pin real upstream resolvers at the daemon.
2. Edit /etc/docker/daemon.json:
3. ```json
4. {
5. "dns": ["192.0.2.53", "198.51.100.53"],
6. "dns-search": ["example.com"]
7. }
8. ```
9. dockerd --validate first if your version supports it, then systemctl reload docker (or restart — see the caution below).
10. Or fix the host file instead.
11. ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf points the host at the non-stub file, which lists the real upstream servers rather than the loopback stub. Both approaches work; the daemon.json one does not depend on the host resolver stack staying the way you left it.
12. Recreate the containers.
13. docker compose up -d --force-recreate — a running container will not pick up the new resolver list.
14. Handle the per-network case if it applies.
15. A container can override the daemon default with --dns or the Compose dns: key. Check for one before concluding the daemon setting did not take.

Verification

1. External resolution works from a fresh container.
2. docker run --rm alpine getent hosts api.example.com
3. The resolver list in a recreated container is the intended one.
4. docker exec app cat /etc/resolv.conf — on a user-defined network this still reads 127.0.0.11, which is correct; verify the upstreams with docker run --rm --network none alpine cat /etc/resolv.conf or by inspecting the daemon config.
5. Internal service names still resolve. docker exec app getent hosts postgres
6. The daemon no longer warns. journalctl -u docker --since -10m | grep -ci "default external servers" returns 0.

Prevention

• Set dns in daemon.json on every Docker host. Inheriting the host resolver stack means your container DNS configuration changes whenever someone touches netplan.
• Add an external-resolution probe that runs inside a container. The host resolving names proves nothing: the host has a working stub listener and the container does not.
• Alert on the daemon’s own warning. Using default external servers in the dockerd log is a one-line, unambiguous predictor of this outage, emitted before anything breaks.
• Put “recreate containers” in the change record for any resolver change. Configuration that is copied at create time does not propagate to running workloads, and the delay is what turns a five-minute fix into an incident.