Cross-host networking and service discovery — the hard part

Scheduling containers across hosts is the easy half. The hard half is that a container on node A now has to reach a container on node C, by a name, over a network that neither of them can see, through whatever firewalls and MTUs sit between the two hosts.

This is where single-host intuition breaks most sharply, and where the failures are least like anything in the Docker course so far.

The problem statement

On one host, two containers on the same bridge share a Linux bridge device. The kernel switches frames between two veth pairs. There is no encapsulation, no routing, and the MTU is the bridge’s MTU.

Across hosts, none of that is available. Container addresses are private to each host’s networking and mean nothing on the physical network. Three approaches exist, and every orchestrator uses one of them:

Overlay is the default because it asks nothing of the physical network. That convenience is paid for in the next section.

Encapsulation costs MTU

A VXLAN header adds 50 bytes to every packet: 14 bytes of outer Ethernet, 20 of outer IPv4, 8 of UDP, and 8 of VXLAN. On a standard 1500-byte underlay, that leaves 1450 bytes for the inner frame, which is why Docker sets the overlay network MTU to 1450 rather than 1500.

$ docker network inspect bridge --format '{{json .Options}}'

{"com.docker.network.bridge.default_bridge":"true","com.docker.network.bridge.enable_icc":"true","com.docker.network.bridge.enable_ip_masquerade":"true","com.docker.network.bridge.host_binding_ipv4":"0.0.0.0","com.docker.network.bridge.name":"docker0","com.docker.network.driver.mtu":"1500"}

If you also encrypt the overlay — --opt encrypted, which wraps the VXLAN traffic in IPsec ESP — the overhead grows again, and the MTU has to come down further. Encryption also costs measurable CPU on every node at high packet rates, which is worth measuring rather than assuming.

The ports a cluster needs

Cross-host container networking has a control plane and a data plane, and they use different ports. Firewall rules written for one and not the other produce a cluster that forms and then does not pass traffic.

For Docker Swarm:

ss -lunp | grep 4789
nc -vzu 192.0.2.21 4789

Kubernetes has an equivalent set that depends on the CNI plugin: the API server on 6443/TCP, etcd on 2379-2380/TCP between control-plane nodes, the kubelet on 10250/TCP, and whatever the CNI plugin uses — VXLAN on 8472/UDP for Flannel and some Calico configurations, Geneve on 6081/UDP for others. Read your plugin’s documentation; there is no universal list.

Service discovery adds indirection

On a single host, db resolves to one container’s IP. Across a cluster it resolves to something more abstract, and there are two models.

Virtual IP (VIP). The default in Swarm and the ClusterIP model in Kubernetes. The service name resolves to a single stable virtual address, and the kernel — via IPVS, iptables, or eBPF depending on the platform — load balances connections to the real backends behind it. Clients see one address that never changes.

DNS round-robin. The service name resolves to the list of backend addresses, and the client picks. Swarm calls this --endpoint-mode dnsrr; Kubernetes calls it a headless service.

$ docker exec myapp.1.abc123 getent hosts db

10.0.1.7        db

A single answer for a service with four replicas is the tell that you are looking at a VIP.

Debugging encapsulated traffic

Two things change about packet capture once traffic is encapsulated. On the host, you see UDP 4789, not your application’s port — so a filter for tcp port 5432 finds nothing even though the database is being queried constantly. And host firewall rules that match on the application port never fire for cross-host traffic, because at that point in the stack the packet is a UDP datagram.

# On the host: the encapsulated view
tcpdump -ni any -c 20 udp port 4789

# Inside the container's namespace: the decapsulated view
nsenter --target "$(docker inspect --format '{{.State.Pid}}' myapp.1.abc123)" \
--net -- tcpdump -ni any -c 20 tcp port 5432

The second command is the one that answers most questions, because it shows the traffic as the application sees it. tcpdump running in the container’s network namespace with the host’s binaries needs neither a shell nor tcpdump inside the image.

Knowledge check