medium riskservice affecting~30 min

Runbook: Container unreachable — work down the network path in order

1 · Prerequisites

Confirm every item is in place before any state change.

Shell access on the Docker host, with sudo for iptables and tcpdump
You can state the failure as a direction - inbound from outside, outbound to the internet, or container to container - because the tree branches on that
The container name, the port it should serve, and the client address are all recorded
A change window, or agreement that reconnecting a container will briefly drop its connections
jq is installed, or you are prepared to read raw JSON from docker inspect

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· CONTAINER=web
· docker inspect -f '{{range $n, $c := .NetworkSettings.Networks}}{{$n}} ip={{$c.IPAddress}} aliases={{$c.Aliases}}{{end}}' "$CONTAINER"
· docker inspect -f '{{json .NetworkSettings.Ports}}' "$CONTAINER"
· docker port "$CONTAINER"
· docker network ls
· docker exec "$CONTAINER" cat /etc/resolv.conf
· ss -ltnp | grep -E ':80|:443|:8080'

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1State the direction of the failure before running anything - inbound, outbound, or container to container. Each branch below rules out a different set of causes, and running them out of order wastes the incident.
2Run docker inspect -f '{{json .NetworkSettings.Networks}}' "$CONTAINER" and read which networks it is on; expect a user-defined network name. The literal name bridge means the default bridge, which has no service discovery, and that alone explains most cannot-resolve reports.
3For a container-to-container failure, run docker exec "$CONTAINER" getent hosts other-service; expect the peer container IP. No output on the default bridge confirms the missing service discovery; no output on a user-defined network means the peer is not on that network or is not running.
4Run docker exec "$CONTAINER" cat /etc/resolv.conf; expect nameserver 127.0.0.11 on a user-defined network. Any other nameserver means the embedded resolver was overridden by --dns and container names will not resolve.
5Confirm the application is listening and on which address with docker exec "$CONTAINER" ss -ltnp or docker top "$CONTAINER"; expect 0.0.0.0 or a wildcard bind. A bind to 127.0.0.1 is unreachable from outside the container no matter which ports you publish.
6Run docker port "$CONTAINER"; expect a mapping such as 8080/tcp then 0.0.0.0:8080. A HostIp of 127.0.0.1 means the port is reachable only from the host itself, and an empty result means nothing was published at all.
7Test from the host first with curl against 127.0.0.1 and the published port; expect a response. Success here and failure from a remote client moves the problem to the host firewall, and failure here keeps it inside the container.
8Inspect the firewall with sudo iptables -S DOCKER-USER and sudo iptables -L -n -v; expect no DROP or REJECT rule matching the client address or the published port. Docker inserts its own rules, so any custom rule must live in DOCKER-USER to be seen.
9If small requests succeed and large transfers or TLS handshakes hang, compare MTUs with ip link show docker0 and docker exec "$CONTAINER" ip link show eth0 against the host uplink; expect the container MTU to be no larger than the smallest link on the path.
10Apply the single fix the tree identified, change nothing else, and re-run the exact command that failed at the start.

4 · Verification

Confirm the procedure actually fixed the problem.

✓docker exec "$CONTAINER" getent hosts other-service returns the peer IP address on the same user-defined network
✓curl -sS -o /dev/null -w %{http_code} against the published port from a remote client returns the expected status code, not a connection timeout
✓docker port "$CONTAINER" shows the intended container port mapped to the intended host address, and 0.0.0.0 rather than 127.0.0.1 where remote access is required
✓ss -ltnp on the host shows a listener on the published host port
✓A transfer larger than one MTU - for example curl against a file of several megabytes - completes rather than stalling part way
✓sudo iptables -S DOCKER-USER contains no rule that matches the client address for the published port

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶If a container was attached to an extra network with docker network connect, detach it with docker network disconnect NETWORK CONTAINER and confirm docker inspect no longer lists that network
↶If a DOCKER-USER rule was inserted, remove exactly that rule with sudo iptables -D DOCKER-USER followed by the same rule specification; iptables changes do not survive a reboot unless they were persisted, so check whether they were
↶If the network MTU was changed, the network must be recreated to change it back, which means every attached container is disconnected and reconnected. Schedule it rather than doing it mid-incident
↶If a container was recreated to move it onto a different network, redeploy the previous definition from source control and confirm the original network membership
↶Recreating a container to change its network is not reversible in place - the original container is gone. Capture docker inspect output before you recreate anything

6 · Escalation

When the runbook isn't enough, contact:

· The container is reachable from the host but not from another host, and the host firewall is clean: escalate to the network team with the tcpdump capture from both ends
· Packets arrive on the host interface but never reach the container: escalate to the platform team, because the fault is in the daemon or the bridge rather than in the application
· DNS resolves inside one container and not another on the same network: escalate to the platform team with both resolv.conf files and the docker network inspect output
· A path MTU problem crosses a VPN or an overlay you do not control: escalate to whoever owns that tunnel, with the size at which transfers begin to fail

“The container is unreachable” describes at least six unrelated faults. This runbook walks them in the order that eliminates the most possibilities per command, and each step states what its output rules in and what it rules out. Work down. Do not skip to the firewall because the firewall is the exciting answer — it is near the bottom of the list because it is rarely the cause.

Step 0: Name the direction

Read-only / Safescope the failure— Establishes which of three problems you have. No side effects.

CONTAINER=web

# Inbound: can a client outside the host reach the published port?
# Outbound: can the container reach the internet or another host?
# East-west: can this container reach another container?

docker ps --filter name="$CONTAINER" --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'

The three directions share almost no causes. An inbound failure is about publishing, binding and the host firewall. An outbound failure is about DNS, the default route and egress filtering. An east-west failure is about network membership and service discovery. Deciding which one you have costs nothing and removes two thirds of the tree.

Step 1: Which network is it on?

Read-only / Safenetwork membership— Reads container and network configuration. No side effects.

docker inspect -f '{{json .NetworkSettings.Networks}}' "$CONTAINER" | jq .

# Everything attached to a given network, with its addresses
docker network inspect app-net -f '{{json .Containers}}' | jq .
docker network inspect app-net -f '{{.Driver}} {{json .Options}} {{json .IPAM.Config}}'

What this rules in or out. A network literally named bridge is the default bridge. A network with any other name that you created is a user-defined bridge. The difference is not cosmetic:

	Default bridge	User-defined bridge
Name-based resolution between containers	No	Yes
Aliases (`--network-alias`)	No	Yes
Containers reachable by	IP address only	Name, alias, or IP
Isolation from unrelated containers	None — every container shares it	Yes, per network
Attach or detach on a running container	No	Yes, with `docker network connect`

Step 2: The embedded resolver

On a user-defined network, Docker runs an embedded DNS server and points the container at it. Its address is 127.0.0.11 — inside the container’s own namespace, so it is not something you can query from the host.

Read-only / SafeDNS inside the container— Reads resolver configuration and performs lookups. No state change.

docker exec "$CONTAINER" cat /etc/resolv.conf

# Resolve a peer service by name
docker exec "$CONTAINER" getent hosts db
docker exec "$CONTAINER" getent hosts api

# External resolution, to separate 'DNS is broken' from 'this name is unknown'
docker exec "$CONTAINER" getent hosts example.com

# What names the daemon has registered for a container
docker inspect -f '{{range .NetworkSettings.Networks}}{{.DNSNames}} {{.Aliases}}{{end}}' "$CONTAINER"

What this rules in or out.

nameserver 127.0.0.11 present, peer name resolves — DNS is fine. Move to step 3.
nameserver 127.0.0.11 present, peer name does not resolve but example.com does — the peer is not on this network, is not running, or you are using the wrong name. Compare against docker network inspect.
nameserver is something else entirely — a --dns flag or a daemon-wide dns setting replaced the embedded resolver. Container names will not resolve, because only the embedded resolver knows them. External names still will, which is exactly why this gets misdiagnosed.
Nothing resolves, including example.com — the embedded resolver cannot reach its upstream. Check the host’s own /etc/resolv.conf, which is where it forwards to.

getent hosts is deliberate here. nslookup and dig are absent from most application images, and ping resolves but also tests ICMP, which many networks drop — so a failed ping tells you two things at once and neither clearly.

Step 3: Is anything actually listening, and where?

Read-only / Safelistening sockets— Reads socket state inside the container namespace. No state change.

# Inside the container
docker exec "$CONTAINER" ss -ltnp

# If the image has no ss, read the kernel table directly
docker exec "$CONTAINER" cat /proc/net/tcp

# From the host, for an image with no shell at all
docker top "$CONTAINER"

Read the Local Address column and nothing else for a moment:

Local Address	Reachable from	Verdict
`0.0.0.0:8080`	Anywhere the network allows	Correct for a published service
`[::]:8080`	Anywhere, IPv6 wildcard, usually dual-stack	Normally fine
`127.0.0.1:8080`	Only from inside this container	This is the fault
nothing on 8080	Nowhere	The app failed to start, or uses a different port

An app bound to 127.0.0.1 inside the container cannot be published

Publishing a port sets up a mapping from a host address to the container’s address on its network interface. If the application is listening only on the container’s loopback, there is nothing on that interface to map to. docker run -p 8080:8080 succeeds, docker port shows a mapping, the firewall is clean, and every connection is still refused.

This is the most consistently misdiagnosed item in this runbook, because every layer you inspect looks correct. The fix is in the application’s configuration — bind to 0.0.0.0, or to all interfaces — not in Docker.

Note the asymmetry: binding to 0.0.0.0 inside a container is not the exposure risk it is on a host. The container’s network namespace is already the isolation boundary, and what the outside world can reach is decided by which ports you publish and to which host address.

Step 4: The published port and the host address

Read-only / Safeport mapping— Reads the mapping and tests it locally. No state change.

docker port "$CONTAINER"
docker inspect -f '{{json .NetworkSettings.Ports}}' "$CONTAINER" | jq .

# From the host itself
curl -sS -o /dev/null -w 'host loopback %{http_code}\n' http://127.0.0.1:8080/

# From the host, over its routable address
HOSTIP=192.0.2.10
curl -sS -o /dev/null -w 'host address %{http_code}\n' http://"$HOSTIP":8080/

# What the host has open
ss -ltnp | grep 8080

What this rules in or out. docker port printing nothing means no port was published — the container is reachable from other containers on its network and from nowhere else. A HostIp of 127.0.0.1 means the publish was scoped to the host loopback deliberately, which is correct for a service behind a local reverse proxy and wrong if you expected remote access.

Succeeding over the host loopback but failing over the host’s routable address, from the same host, points at the bind address of the mapping. Succeeding from the host both ways but failing from a remote client moves you to step 5.

Step 5: The firewall, and the chain that matters

Read-only / Safefirewall rules— Lists rules and counters. Does not modify the ruleset.

sudo iptables -S DOCKER-USER
sudo iptables -S FORWARD
sudo iptables -L -n -v --line-numbers | head -60

# Are packets even arriving? Watch while a client retries.
sudo tcpdump -ni any port 8080 -c 20

Docker installs its own chains and processes them ahead of anything appended to FORWARD. Its documentation states that DOCKER-USER is “a placeholder for user-defined rules that will be processed before rules in the DOCKER-FORWARD and DOCKER chains”, and that “packets that get accepted or rejected by rules in these custom chains will not be seen by user-defined rules appended to the FORWARD chain.”

The consequence is blunt: a rule you added to FORWARD may never be consulted. Filtering for containers goes in DOCKER-USER.

Published ports bypass ufw and firewalld

Docker’s own documentation is direct about this: “When you publish a container’s ports using Docker, traffic to and from that container gets diverted before it goes through the ufw firewall settings”, because “Docker routes container traffic in the nat table, which means that packets are diverted before it reaches the INPUT and OUTPUT chains that ufw uses.”

So ufw status can read deny 8080 while the container answers on 8080 from the whole internet. If you are diagnosing the opposite problem — a port that should be closed and is not — this is why.

Two safe fixes: publish to a specific host address (-p 127.0.0.1:8080:8080) so the port is never on a routable interface, or add the restriction to DOCKER-USER. Do not turn the firewall off to test.

If you must restrict access, Docker documents the shape of the rule:

Configuration changerestrict by source— Inserts a filtering rule. Changes reachability immediately; not persistent across reboot unless saved.

# Allow only one subnet to reach containers via the external interface.
# ext_if is the host's external interface name, for example eth0.
sudo iptables -I DOCKER-USER -i ext_if ! -s 192.0.2.0/24 -j DROP

# Verify, then confirm the intended client still works
sudo iptables -S DOCKER-USER

Note the rule is inserted with -I, not appended, because DOCKER-USER ends by returning to Docker’s own chains. Note also that this is a documented, supported control — it is a narrowing, not the removal of one. Never fix a reachability problem by flushing the ruleset or stopping the firewall.

Step 6: MTU

The signature is unmistakable once you know it: small requests succeed instantly, large responses or TLS handshakes hang and then time out. Nothing is dropped in a way any log records.

Read-only / SafeMTU comparison— Reads interface MTUs and probes path MTU. No state change.

# Host uplink, docker bridge, container interface
ip link show
ip link show docker0
docker exec "$CONTAINER" ip link show eth0

# What the network was configured with
docker network inspect app-net -f '{{json .Options}}'

# Probe: send a large packet that must not be fragmented.
# Reduce the size until it succeeds; that is your path MTU minus overhead.
docker exec "$CONTAINER" ping -c 2 -M do -s 1472 example.com

What this rules in or out. A container MTU larger than the smallest link on the path — common behind a VPN, a tunnel, or a cloud overlay — produces exactly this stall. A ping -M do that succeeds at 1472 and fails at 1473 puts the path MTU at 1500. If it fails well below that, set the network’s MTU to match using the com.docker.network.driver.mtu driver option when the network is created.

What if it worked yesterday and nothing changed?

Something changed; the question is what. Three candidates account for most of these.

A container was recreated onto a different network. A docker compose up after an edit recreates containers, and a service moved between networks resolves peers it used to see and loses ones it did not.

The peer’s IP moved. Container addresses are assigned on start. Anything that cached an IP rather than resolving the name — a config file, a connection pool, an /etc/hosts entry someone added as a workaround — now points at the wrong container, or at nothing.

The host firewall was reloaded. Reloading firewalld or restarting ufw can rebuild chains and drop the Docker rules, which reappear only when the daemon next reconciles them. Restarting the Docker daemon reinstates them, at the cost of restarting containers with a restart policy.

The tree, condensed

Symptom	First command	If it fails
Container cannot resolve another container	`docker inspect -f '{{json .NetworkSettings.Networks}}'`	On `bridge`, there is no service discovery — move both to a user-defined network
Name resolves nowhere, including external	`docker exec CONTAINER cat /etc/resolv.conf`	Not `127.0.0.11`, so the embedded resolver was overridden
Connection refused from the host	`docker exec CONTAINER ss -ltnp`	Bound to `127.0.0.1` inside the container
Connection refused from remote only	`docker port CONTAINER`	`HostIp` is `127.0.0.1`, so it is host-local by design
Connection times out from remote	`sudo iptables -S DOCKER-USER`	A DROP rule matches the client
Packets never arrive at all	`sudo tcpdump -ni any port 8080`	Upstream routing or a network ACL, not Docker
Small requests fine, large ones hang	`ip link show` on both sides	MTU mismatch on the path
Port open despite a firewall deny rule	`docker port CONTAINER`	Publishing bypasses ufw; restrict in `DOCKER-USER` or publish to a specific address