Resolver behaviour and timeouts — glibc, musl, and slow lookups

The same application, the same Compose file, the same host. Built on python:3.12-slim it is fine. Rebuilt on python:3.12-alpine — the “smaller, faster” choice somebody made in a Dockerfile cleanup — it starts intermittently failing to resolve an internal name, and only under load.

Nothing about Docker changed. The C library did, and the C library is what performs the lookup.

Two resolvers, two philosophies

Debian, Ubuntu and the -slim images use glibc. Alpine uses musl. Their resolvers make opposite choices:

Every row of that table has produced a production incident somewhere. Three of them matter enough to work through.

Parallel queries make split-horizon DNS nondeterministic

glibc asks nameserver 1, waits, and only asks nameserver 2 if the first does not answer. musl asks all of them at once and accepts whichever response arrives first.

Now consider the very common corporate arrangement: nameserver 1 is the internal resolver that knows about corp.example.com, and nameserver 2 is a public resolver added “for redundancy”. A lookup for billing.corp.example.com:

• Under glibc, the internal resolver answers, and the public one is never asked. Correct, every time.
• Under musl, both are asked. The public resolver returns NXDOMAIN quickly because the zone is not public. If that arrives first, the application is told the name does not exist.

It works most of the time — whenever the internal resolver happens to be faster — and fails under load, when it is not. That is the shape of the bug: intermittent, load-correlated, and completely invisible in the application code.

The 512-byte cliff

A DNS response that does not fit in a UDP datagram is returned truncated, with a flag telling the client to retry over TCP. musl did not support DNS over TCP until version 1.2.4, so on older Alpine images a truncated response is simply a failed lookup.

You meet this when a name has many A records — a scaled Compose service, a large round-robin pool, a cloud load balancer that returns a dozen addresses. The name resolves everywhere except in the Alpine container, and it started failing on the day somebody scaled the service up.

$ docker exec web sh -c 'ls /lib/ld-musl-* 2>/dev/null || ldd --version 2>&1 | head -1'

/lib/ld-musl-x86_64.so.1

A path under /lib/ld-musl- means musl; a ldd (GNU libc) 2.36 banner means glibc. Run it against every image in the stack once — mixed-libc stacks are common and nobody documents them.

Alpine 3.18 and later ship musl 1.2.4 or newer, so this is now mostly a legacy-image problem. It is still worth knowing, because “mostly fixed” images live a long time in a registry.

Timeouts: what a failing lookup actually costs

resolv.conf(5) documents the two knobs and their defaults: timeout defaults to 5 seconds, attempts defaults to 2. glibc walks the nameserver list attempts times, waiting up to timeout each time.

So the worst case for one name, with two nameservers and defaults, is 5 × 2 × 2 = 20 seconds. Multiply by the search-list expansion from the previous lesson and a single connect() to a name that does not exist can block a thread for well over a minute.

CTR=web

# Wall time for the path the application actually takes
docker exec "$CTR" sh -c 'time getent hosts api' 2>&1

# Where the time goes, per query
docker exec "$CTR" dig +stats +noall +answer +comments api

Tighten the defaults for anything that must fail fast:

services:
web:
  image: myapp:1.4.2
  dns_opt:
    - timeout:2
    - attempts:2

When a DNS outage is not reported as a DNS outage

Keeping DNS out of the hot path

The structural fix is to resolve less often, not to resolve faster:

• Pool connections. A connection pool that keeps sockets open resolves once per new connection, not once per request. This is the single largest reduction available.
• Cache deliberately, with a bounded TTL. A resolving cache in the container — or a per-host caching resolver the containers point at — turns a resolver outage into a slow degradation instead of an immediate one.
• Keep the search list short. Every extra suffix multiplies the cost of every failure.
• Do not put DNS in the readiness probe’s critical path unless you genuinely want the container marked unready when DNS is slow. Sometimes you do. Decide, rather than inheriting it.

Sanity check