The Pod network namespace and veth pair — what the kubelet actually creates

A Pod’s network is the result of three Linux kernel primitives: the network namespace, the veth pair, and the route. The CNI plugin configures them on every ADD. This lesson walks the primitives, the mechanism by which they connect the Pod to the host, and the operational discipline of inspecting them.

The Linux network namespace

A Linux network namespace is a private copy of the network stack: interfaces, routes, iptables, sockets. A process in one namespace cannot see the interfaces of another. The CNI plugin creates a namespace for the Pod; the Pod’s containers share the namespace.

# List network namespaces on the host
ip netns list

cni-abc123 (id: 0)
cni-def456 (id: 1)
cni-ghi789 (id: 2)

The cni- prefix is the convention for namespace filenames. The CNI plugin creates the namespace by calling unshare(CLONE_NEWNET) or ip netns add.

The Pod’s containers share the namespace. A multi- container Pod has all of its containers in one namespace; the namespace is created by the sandbox container (the paused container) and inherited by every subsequent container.

The veth pair

A veth pair is a pair of virtual Ethernet interfaces that act as a cable: whatever enters one end exits the other. The CNI plugin creates the pair, places one end inside the namespace (named eth0 by convention), and leaves the other end on the host.

# List the host-side veth interfaces
ip link show | grep -E "cali|lxc|veth"

cali1234: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 ...
cali5678: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 ...

The host-side name depends on the CNI: Calico uses cali*, Cilium uses lxc*, Flannel uses flannel.1 or cni0. The naming is the first signal the operator uses to identify the CNI.

flowchart LR
    A[Pod netns: eth0] -->|veth pair| B[Host: cali1234]
    B --> C[Linux bridge cni0 or routing]
    C --> D[Host network]

The veth pair is the cable that connects the Pod’s network namespace to the host’s network. Without the veth, the namespace is isolated and the Pod has no network.

The route

The host needs to know how to reach the Pod’s IP. The CNI plugin adds a route to the host’s routing table:

ip route show

10.244.1.0/24 dev cni0 proto kernel scope link src 10.244.1.1
10.244.2.5 dev cali5678 scope link  # the Pod's IP

The Pod’s IP is a /32 (host) route on the host-side veth. The route is local: it does not leave the host; the kernel knows the IP is reachable via the veth.

For Pods on other nodes, the host has a route that points to the node’s IP:

10.244.2.0/24 via 10.0.0.5 dev eth0  # Pods on node-2

The route is added by the CNI plugin (or by the BGP daemon, if BGP is in use). The cluster’s routing topology is the merge of every node’s routes.

The sandbox

The kubelet creates the Pod’s namespace via a sandbox container. The sandbox is a paused container that holds the namespace; subsequent containers (the Pod’s actual workloads) join the namespace via the container runtime.

# List the sandbox containers
crictl ps -a | grep "sandbox"

abc123 sandbox billing-7d4  Running
def456 sandbox billing-7d5  Running

The sandbox is the holder of the namespace. When the sandbox is destroyed, the namespace is destroyed; the veth is removed by the CNI plugin’s DEL.

How the kubelet invokes the CNI

The kubelet, on sandbox creation, calls the CRI (RuntimeService.RunPodSandbox). The runtime creates the sandbox and the namespace. The kubelet then calls libcni with the namespace path:

CNI_NETNS=/var/run/netns/cni-abc123
CNI_IFNAME=eth0
CNI_CONTAINERID=abc123

libcni invokes the plugin’s binary; the plugin creates the veth pair, attaches one end to the namespace, assigns the IP, configures the routes, and returns the result.

Inspecting the Pod’s network

The operator can inspect the Pod’s network:

# Substitute your own value before running:
POD=web-7c8d9f4b5-qr2mn

# The Pod's network namespace
kubectl exec "$POD" -- ip addr show

# The host-side veth
ip link show | grep -E "cali|lxc"

# The host routes
POD_IP=$(kubectl get pod "$POD" -o jsonpath='{.status.podIP}')
ip route show | grep "$POD_IP"

The Pod’s network namespace is mounted at /var/run/netns/cni-<id> on the host. The operator can enter the namespace with nsenter:

nsenter --net=/var/run/netns/cni-abc123 ip addr show

The output shows the Pod’s interfaces and routes. The fix is to compare the output against the expected configuration.

The failure modes

The Pod network’s failure modes:

• Missing veth: the CNI plugin failed to create the veth pair. The Pod has no network. The fix is to check the plugin’s logs.
• Missing route: the host has no route to the Pod’s IP. The fix is to verify the CNI plugin’s setup.
• MTU mismatch: the veth has a different MTU than the host. The fix is to match the MTU.
• Namespace leak: the namespace is not cleaned up after the Pod is removed. The fix is to check the DEL path.

The operational discipline

The Pod network’s operational discipline:

• Audit the veth pairs on every node. A growing number of veth pairs without a corresponding Pod count is a leak.
• Audit the host routes. Routes to non-existent Pods are a leak.
• Use nsenter to inspect the Pod’s network. The operator must be able to enter the namespace.
• Test the network in staging. The Pod network is critical infrastructure; the test must cover the full lifecycle.
• Document the network topology. The operator must understand the veth, the route, and the bridge.

Quiz

Production discipline

• The Pod network is built from Linux kernel primitives. The CNI configures them on every ADD.
• The veth pair is the cable. One end in the namespace, one end on the host.
• The host route is the path. The host routes traffic to the Pod via the host-side veth.
• The sandbox is the namespace holder. Add and DEL are tied to the sandbox’s lifecycle.
• Audit the veth pairs and routes. A growing number without a corresponding Pod count is a leak.
• Use nsenter to inspect the Pod’s network. The operator must be able to enter the namespace.
• Test the network in staging. The Pod network is critical infrastructure.
• Document the network topology. The operator must understand the veth, the route, and the bridge.