Networking failures — packets not reaching the container

The container is running. The application inside is running. But traffic from outside is not reaching it. This is the most common production networking failure.

The trouble is that the failure message is almost always the same regardless of which hop is broken. “Connection refused” and “connection timed out” are produced by the host firewall, by a missing DNAT rule, by the container’s namespace having no route, by the application binding to the wrong address, and by an upstream proxy — five layers, two messages. Reading the message tells you nothing. Only a command that isolates a hop does.

The path, hop by hop

flowchart LR
  A[Client] --> B[Host firewall<br/>filter INPUT]
  B --> C["docker-proxy<br/>or nat DOCKER DNAT"]
  C --> D["filter DOCKER-USER<br/>then DOCKER-FORWARD"]
  D --> E["docker0 / br-xxxx<br/>bridge"]
  E --> F["veth pair into<br/>the netns"]
  F --> G["eth0 inside<br/>the container"]
  G --> H["the process's<br/>listening socket"]

Eight things, and any one of them alone will produce “it does not work”. Walking them left to right is the obvious approach and the slow one.

Start in the middle

The first command should split the path, not step along it. The natural split point is the host reaching the container’s own IP directly: that skips the firewall, the DNAT and the proxy, while still exercising the bridge, the veth, the namespace and the socket.

CONTAINER=web
NET=bridge
CPORT=80
HPORT=8080

IP=$(docker inspect "$CONTAINER" \
--format "{{(index .NetworkSettings.Networks \"$NET\").IPAddress}}")
echo "container ip = $IP"

curl -sS -m 3 -o /dev/null -w 'direct:    %{http_code} %{errormsg}\n' \
"http://$IP:$CPORT/" || true
curl -sS -m 3 -o /dev/null -w 'published: %{http_code} %{errormsg}\n' \
"http://127.0.0.1:$HPORT/" || true

Two commands, and you are in one branch instead of eight.

Branch A — both fail: is anything listening?

Before blaming Docker, ask whether there is a socket to connect to. Run the check inside the container’s network namespace, because that is where the socket lives.

CONTAINER=web

# If the image has iproute2
docker exec "$CONTAINER" ss -tlnp

# If it does not, borrow the host's tooling via the container's netns
PID=$(docker inspect "$CONTAINER" --format '{{.State.Pid}}')
sudo nsenter -t "$PID" -n ss -tlnp

# Or attach a throwaway container to the same namespace
docker run --rm --network "container:$CONTAINER" nicolaka/netshoot ss -tlnp

Illustrative output, and the whole diagnosis:

State   Recv-Q  Send-Q  Local Address:Port   Peer Address:Port
LISTEN  0       511         127.0.0.1:80          0.0.0.0:*

127.0.0.1:80, not 0.0.0.0:80. The application is bound to the loopback interface of its own network namespace. Nothing outside that namespace can reach it — not the host, not another container, not the DNAT rule. Every layer of Docker networking is working perfectly and delivering packets to an interface with nothing on it.

If ss shows the socket bound to 0.0.0.0 and the direct curl still fails, the fault is between the bridge and the namespace:

CONTAINER=web
PID=$(docker inspect "$CONTAINER" --format '{{.State.Pid}}')

# Interfaces inside the namespace: expect lo and eth0, both UP
sudo nsenter -t "$PID" -n ip -br addr

# The default route out of the namespace should be the bridge gateway
sudo nsenter -t "$PID" -n ip route

# The host end of the veth pair, and which bridge it is enslaved to
ip -br link | grep -E 'docker0|br-|veth'

Branch B — direct works, published fails

The fault is in publishing. Check the mapping, then the DNAT rule, then the firewall, in that order — cheapest first.

CONTAINER=web
HPORT=8080

# 1. Did the daemon record a mapping at all?
docker port "$CONTAINER"

# 2. Is the DNAT rule programmed? (nat table, DOCKER chain)
sudo iptables -t nat -S DOCKER | grep -- "--dport $HPORT" || echo 'NO DNAT RULE'

# 3. Are the forward chains letting it through?
sudo iptables -S DOCKER-USER
sudo iptables -S DOCKER-FORWARD 2>/dev/null | head -20

# 4. Is something bound on the host side?
sudo ss -tlnp "sport = :$HPORT"

Step 1 returning nothing means the container was started without -p at all, which is a surprisingly common outcome of editing a Compose file and forgetting that docker compose up does not re-create a container whose definition it thinks is unchanged. docker compose up --force-recreate settles it.

Container-to-container: a different tree

Everything above is about ingress. Containers failing to reach each other has its own short tree, and the first branch is which network they are on.

FROM_C=api
TO_NAME=db

# Which networks is each container on? They must share one.
docker inspect "$FROM_C" --format '{{range $k,$v := .NetworkSettings.Networks}}{{$k}} {{end}}'
docker inspect "$TO_NAME" --format '{{range $k,$v := .NetworkSettings.Networks}}{{$k}} {{end}}'

# Does the name resolve at all?
docker exec "$FROM_C" getent hosts "$TO_NAME"

# What resolver is the container using?
docker exec "$FROM_C" cat /etc/resolv.conf

# If it resolves, is the port open?
docker exec "$FROM_C" sh -c "nc -z -w3 $TO_NAME 5432 && echo open || echo closed"

The resolver line is the discriminator that surprises people. Docker’s documentation is explicit: containers on a custom network “use Docker’s embedded DNS server”, whose address is 127.0.0.11, and on a user-defined bridge “containers can resolve each other by name or alias”. Containers on the default bridge network receive a copy of the host’s /etc/resolv.conf instead and “can only access each other by IP addresses”.

So cat /etc/resolv.conf inside the container is a one-line network diagnosis:

nameserver 127.0.0.11

means you are on a user-defined network and name resolution between containers should work. Anything else — the host’s real nameservers — means you are on the default bridge and getent hosts db will fail no matter how correct the rest of your configuration is. The fix is to create a user-defined network and attach both containers to it, which is what Compose does for you automatically and is the main reason docker run examples and Compose stacks behave differently.

When you need to see the packets

If the tree has not settled it, capture. tcpdump inside the container’s namespace tells you whether packets are arriving at all, which separates “never delivered” from “delivered and refused”.

CONTAINER=web
PID=$(docker inspect "$CONTAINER" --format '{{.State.Pid}}')

# Inside the container's network namespace, using the host's tcpdump
sudo nsenter -t "$PID" -n tcpdump -n -i eth0 -c 20 'tcp port 80'

# On the host, on the bridge
sudo tcpdump -n -i docker0 -c 20 'tcp port 80'

# Conntrack: is the NAT translation being created?
sudo conntrack -L 2>/dev/null | grep -E 'dport=8080|dport=80' | head

Reading it: SYN arriving at eth0 with no SYN-ACK back means the namespace received it and nothing was listening — go back to the bind address. SYN on docker0 but nothing at eth0 means the veth or the forward chains dropped it. Nothing on docker0 at all means the packet never got past DNAT or the host firewall.

Verification

#!/usr/bin/env bash
set -euo pipefail
CONTAINER=web
HPORT=8080
HOST_IP=192.0.2.10

# 1. The daemon has a mapping
docker port "$CONTAINER" | grep -q "$HPORT" || { echo 'FAIL: no port mapping' >&2; exit 1; }

# 2. The DNAT rule exists
sudo iptables -t nat -S DOCKER | grep -q -- "--dport $HPORT" \
|| { echo 'FAIL: no DNAT rule' >&2; exit 1; }

# 3. It answers on loopback
code=$(curl -sS -m 3 -o /dev/null -w '%{http_code}' "http://127.0.0.1:$HPORT/")
[ "$code" = '200' ] || { echo "FAIL: loopback returned $code" >&2; exit 1; }

# 4. It answers on the routable address, which is what clients use
code=$(curl -sS -m 3 -o /dev/null -w '%{http_code}' "http://$HOST_IP:$HPORT/")
[ "$code" = '200' ] || { echo "FAIL: external address returned $code" >&2; exit 1; }

echo OK

Step 4 is the one that catches a publication bound to 127.0.0.1 on the host — which is correct and deliberate for a database, and a silent outage for a service that is supposed to be reachable.

Knowledge check