Skip to main content
RunBook Academy

← All break/fix scenarios in Docker & Containers

advancedNetworking~20 min

Break/Fix 11: Default-DROP nftables policy blocks a published port

Reported symptoms

  • ●docker ps shows the container is running and healthy.
  • ●From inside the host, curl localhost:8080 returns the application.
  • ●From outside the host, the connection is refused or times out.
  • ●sudo ufw status shows the host firewall "off" or "allow"; the port is still closed from outside.

Evidence

  • · ss -tlnp shows the docker-proxy listening on 0.0.0.0:8080.
  • · iptables -t nat -L DOCKER -n -v shows the DNAT rule for port 8080.
  • · nft list ruleset shows the FORWARD chain (or the bridge input chain) dropping the packet before the bridge ever sees it.
  • · tcpdump -i any -n port 8080 shows packets arriving at the host but no SYN-ACK leaving it.
Diagnosis and resolutionclick to reveal

Root cause

The host's nftables (or iptables) default-DROP policy applies to the FORWARD chain and / or the bridge-input path. Docker's port publishing uses DNAT and the kernel's bridge-netfilter path to deliver packets to the container; the host INPUT chain is bypassed entirely, so an INPUT-chain allow rule does not help. The DOCKER-USER chain (a hook point in the FORWARD path that the operator owns) is where a published-port allow rule belongs.

Remediation

Add the allow rule to DOCKER-USER, not to INPUT. Docker creates DOCKER-USER automatically; rules placed there apply to packets on their way to a published-port DNAT. For this scenario: ``` sudo nft add rule ip filter DOCKER-USER iifname "eth0" tcp dport 8080 accept ``` Then verify by probing from off the host, not from inside it. A second host on the same network is the right test. From the host itself, packets bypass the firewall by design.

Verification

curl from a second host on the same network succeeds and the application responds. The nft counter on the DOCKER-USER rule increments on each successful connection. The INPUT chain counter does not increment for traffic to the published port (the rule is on DOCKER-USER, not INPUT).

Prevention

Document the firewall as part of the host baseline. The DOCKER-USER chain is the contract for "what traffic reaches my published ports"; INPUT-chain rules do not apply to them. Validate by probing from outside the host on every deploy that changes publish or routing.

Reported symptoms

  • docker ps shows the container is running and healthy.
  • From inside the host, curl localhost:8080 returns the application.
  • From outside the host, the connection is refused or times out.
  • sudo ufw status shows the host firewall is off or has no rule for 8080.

Evidence provided

$ ss -tlnp | grep 8080
LISTEN 0 4096 0.0.0.0:8080 ... users:(("docker-proxy",...))

$ sudo iptables -t nat -L DOCKER -n -v
Chain DOCKER (1 references)
 pkts bytes target     prot opt in  out  source   destination
   12   720 DNAT       tcp  --  0.0.0.0/0  0.0.0.0/0  tcp dpt:8080 to:172.17.0.2:8080

$ sudo nft list ruleset | grep -A 1 'chain input'
chain input {
    type filter hook input priority 0; policy drop;
}

The docker-proxy is bound, the DNAT rule is in place, but the default-DROP policy on the FORWARD path (or the bridge’s no-track path) is killing the packet before it reaches the container.

Resolution path

  1. Confirm the packet reaches the host.
  2. tcpdump -i any -n port 8080 while reproducing. If no SYN ever shows up at the wire, the issue is upstream of the host (routing, ISP, peer).
  3. Confirm the host accepts the packet on the FORWARD path.
  4. If the packet arrives but no SYN-ACK leaves, the host is dropping it. nft list ruleset and look at FORWARD; iptables -L FORWARD -n -v is the iptables equivalent.
  5. Add the allow rule to DOCKER-USER, not INPUT.
  6. DOCKER-USER is the operator-owned chain in the FORWARD path that Docker creates. Rules there apply to traffic on its way to a published port.
  7. sudo nft add rule ip filter DOCKER-USER iifname "eth0" tcp dport 8080 accept
  8. Verify by probing from outside the host.
  9. A second host on the same network is the right test. curl http://docker-host:8080. From the host itself, the firewall is bypassed and the test is meaningless.
  10. For ufw specifically: recognise that ufw INPUT rules do not apply here.
  11. ufw manages the INPUT chain by default. Published ports never see INPUT. Either reconfigure ufw to manage DOCKER-USER too (DEFAULT_FORWARD_POLICY in /etc/default/ufw, the manual DOCKER-USER hook in /etc/ufw/after.rules), or use nftables directly.

Verification

  1. curl from a second host succeeds.
  2. The application responds with the expected payload.
  3. The DOCKER-USER counter increments.
  4. nft list ruleset after the test connection shows the rule has matched.
  5. The INPUT counter does not increment for traffic to the published port.
  6. This is the expected behaviour: published-port traffic bypasses INPUT.