Network namespaces by hand - building container networking with ip

A container runtime does not have a networking subsystem. It calls the same kernel interfaces ip calls, in the same order, and everything it produces is inspectable with the tools you already use. This lesson builds that network by hand.

The point is not that you will provision containers this way. It is that four commands stand between “the container has no network” and a working one, and each of them has exactly one characteristic failure. Once you have made all four fail on purpose, container networking stops being opaque.

The whole exercise needs root and touches only virtual devices. Every object created here disappears when you delete the namespace at the end.

A namespace on its own

sudo ip netns add blue
ip netns list
sudo ip netns exec blue ip -o link show

$ sudo ip netns exec blue ip -o link show

1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN mode DEFAULT group default qlen 1000\    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

One interface, and it is down. That is a complete network namespace: its own interfaces, routing table, ARP table, conntrack table, socket table and firewall rule set, sharing nothing with the host.

The isolation is total in both directions:

# The host's listening sockets
ss -ltn | wc -l

# The namespace's listening sockets - there are none
sudo ip netns exec blue ss -ltn

Bring loopback up before anything else. This is the first characteristic failure:

sudo ip -n blue link set lo up

Skip it and the namespace has no 127.0.0.1. Anything inside that connects to loopback - a health check, a sidecar, a database client configured for localhost - fails with “Network is unreachable” while the external interface works perfectly. It reads as an application bug and it is a missing one-line command.

A veth pair: the virtual cable

A veth device is always created as a pair. Whatever enters one end leaves the other. Think of it as a patch cable with a plug at each end, and remember that you must configure both.

# Create the cable. Both ends land on the host initially.
sudo ip link add veth-blue type veth peer name veth-blue-host

# Push one end into the namespace
sudo ip link set veth-blue netns blue

# Address and enable the namespace end
sudo ip -n blue addr add 192.0.2.2/24 dev veth-blue
sudo ip -n blue link set veth-blue up

# Address and enable the host end
sudo ip addr add 192.0.2.1/24 dev veth-blue-host
sudo ip link set veth-blue-host up

ip -n blue is shorthand for ip netns exec blue ip, and it is worth using because it makes the namespace explicit on every line.

Now the two ends can reach each other:

sudo ip netns exec blue ping -c 2 192.0.2.1

The second characteristic failure lives here: a veth end that is up in the namespace but down on the host. The link shows LOWER_UP missing and every ping fails silently. A veth pair needs both ends up, always - there is no autonegotiation to save you.

Getting out: a default route and forwarding

The namespace can reach the host. It cannot reach anything beyond it.

sudo ip -n blue route add default via 192.0.2.1

# The host must be willing to forward between interfaces
sudo sysctl -w net.ipv4.ip_forward=1

# And the packets need a source address the outside world can answer
sudo nft -f - <<'EOF'
table ip nat {
  chain postrouting {
    type nat hook postrouting priority srcnat; policy accept;
    ip saddr 192.0.2.0/24 oifname != "lo" masquerade
  }
}
EOF

$ sudo sysctl net.ipv4.ip_forward

net.ipv4.ip_forward = 1

Two failures, and they present identically at first glance.

ip_forward=0. The packet reaches the host, the host refuses to route it between interfaces, and it is dropped without a reply. tcpdump on the host end of the veth shows the outbound packet and nothing on the uplink.

No masquerade rule. The packet is forwarded and leaves with source 192.0.2.2, which the upstream network has never heard of. It goes out, the reply has nowhere to go, and the symptom is a timeout. tcpdump on the uplink shows the request leaving and no reply.

The distinguishing test is one capture on the uplink: did the packet leave the machine at all?

Several namespaces on a bridge

Two containers need to talk to each other, which means a switch. That is a Linux bridge, and the pattern is the same one every container runtime uses.

# The switch
sudo ip link add name br-lab type bridge
sudo ip addr add 198.51.100.1/24 dev br-lab
sudo ip link set br-lab up

# One namespace, one cable, one port on the switch
for NS in red green; do
  sudo ip netns add "$NS"
  sudo ip link add "veth-$NS" type veth peer name "br-$NS"
  sudo ip link set "veth-$NS" netns "$NS"
  sudo ip link set "br-$NS" master br-lab
  sudo ip link set "br-$NS" up
  sudo ip -n "$NS" link set lo up
  sudo ip -n "$NS" link set "veth-$NS" up
done

sudo ip -n red   addr add 198.51.100.10/24 dev veth-red
sudo ip -n green addr add 198.51.100.11/24 dev veth-green
sudo ip -n red   route add default via 198.51.100.1
sudo ip -n green route add default via 198.51.100.1

sudo ip netns exec red ping -c 2 198.51.100.11

On a host running containers, this is exactly what you find:

$ ip -o link show type veth

7: vethb35ea9a@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-0bd6d89efeaa state UP mode DEFAULT group default
8: vethc9bcc3d@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-0bd6d89efeaa state UP mode DEFAULT group default
9: veth775bb45@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-0bd6d89efeaa state UP mode DEFAULT group default

Read that output as three sentences. @if2 is the peer index - interface 2 inside the namespace, which is why it is always a low number and never matches anything on the host. master br-0bd6d89efeaa is the bridge it is plugged into. And each of those host-side ends has a partner in a namespace you cannot see in ip netns list.

Attaching to a container namespace

This is the part that turns the exercise into a tool. A running container has a network namespace with no name. Give it one:

# Find the container's main process. Any runtime, via its own tooling,
# or from the cgroup - see the host-side attribution lesson.
PID=24601

# Register the running namespace under a name
sudo ip netns attach webns "$PID"

ip netns list
sudo ip netns exec webns ip -o addr show
sudo ip netns exec webns ss -ltnp
sudo ip netns exec webns nft list ruleset

ip netns attach bind-mounts /proc/$PID/ns/net into /run/netns/webns. Nothing about the container changes; you have added a name for something that already existed. Every ip netns and every namespace-aware tool now works against it, including tcpdump:

sudo ip netns exec webns tcpdump -ni any -c 20 'tcp port 8080'

That capture is inside the container’s network stack, taken with the host’s tcpdump, requiring nothing to be installed in the container image. It is the answer to “the image is distroless and I need to see the traffic”.

The equivalent without registering a name:

sudo nsenter -t 24601 -n ss -ltnp
sudo nsenter -t 24601 -n ip route

nsenter -n enters only the network namespace, so the command that runs is the host’s binary with the host’s libraries, looking at the container’s network. That combination is usually what you want.

Cleaning up

sudo ip netns delete blue
sudo ip netns delete red
sudo ip netns delete green
sudo ip link delete br-lab

Deleting a namespace destroys the interfaces inside it, and a veth whose peer is destroyed disappears too - so the host-side ends go with them. ip netns delete webns removes only the name you attached; the container keeps running and keeps its namespace.

Knowledge check