Pod-to-Pod communication — same node, different node

Pod-to-Pod communication is the foundation of the Kubernetes networking model. The traffic takes one of two paths: same-node (via the host bridge) or cross-node (via the overlay or the routed network). This lesson walks both paths, the failure modes of each, and the operational discipline of inspecting the traffic.

Same-node: the bridge path

On the same node, every Pod’s veth is attached to a Linux bridge on the host. The bridge is a Layer 2 segment: every veth on the bridge can reach every other veth without going through the host network.

flowchart LR
    A[Pod A: eth0] -->|veth| B[cali1234]
    C[Pod B: eth0] -->|veth| D[cali5678]
    B --> E[Bridge cni0]
    D --> E
    E -->|Layer 2| F[Host network]

The bridge is configured by the CNI plugin. Calico uses a bridge named cni0 (or no bridge at all, in the routed mode); Cilium uses no bridge and routes between veths directly; Flannel uses a bridge named cni0.

The same-node path:

1. Pod A sends a packet to Pod B’s IP.
2. The kernel looks up the route: Pod B’s IP is via the bridge.
3. The kernel forwards the packet to the veth that matches Pod B’s IP.
4. Pod B receives the packet.

The path is short, the latency is low, and the debugging is simple: the operator can tcpdump on either veth to see the traffic.

Cross-node: the overlay path

When Pods on different nodes communicate, the traffic must leave the source node and reach the destination node. The CNI plugin offers two paths:

The overlay path

An overlay encapsulates the Pod’s packet in a host- network packet. The overlay is invisible to the host network: the host sees the encapsulation header, not the Pod’s IP.

sequenceDiagram
    autonumber
    participant PA as Pod A
    participant VA as Host veth
    participant OVA as Overlay VXLAN
    participant OVB as Overlay VXLAN
    participant VB as Host veth
    participant PB as Pod B
    PA->>VA: eth0: 10.244.1.5 -> 10.244.2.6
    VA->>OVA: encap to node-2 IP, dst UDP 4789
    OVA->>OVB: tunnel node-1 -> node-2
    OVB->>VB: decap to 10.244.2.6
    VB->>PB: 10.244.1.5 -> 10.244.2.6

Common overlay encapsulations:

• VXLAN: Layer 2 over UDP port 4789. The de-facto standard for overlay CNIs.
• IPIP: Layer 3 over IP protocol 4. Used by Calico in IPIP mode.
• GENEVE: Layer 2 over UDP port 6081. Used by some CNIs.

The overlay path is simple: the cluster operator does not need to advertise Pod CIDRs to the underlying network. The trade-off is encapsulation overhead (typically 50-100 bytes per packet) and the inability to inspect Pod traffic with host-network tools.

The routed path

A routed path uses BGP to advertise the Pod CIDRs to the underlying network. The host network knows how to route to every Pod’s IP. The Pod’s packet is sent without encapsulation.

sequenceDiagram
    autonumber
    participant PA as Pod A
    participant HA as Node-1 host
    participant HB as Node-2 host
    participant PB as Pod B
    PA->>HA: 10.244.1.5 -> 10.244.2.6
    HA->>HB: route via BGP: 10.244.2.0/24 via node-2
    HB->>PB: 10.244.1.5 -> 10.244.2.6

The routed path is efficient: no encapsulation, no MTU issues. The trade-off is that the underlying network must support the Pod CIDRs; the cluster operator must coordinate with the network team.

The CNI’s role

The CNI plugin is responsible for setting up the bridge, the routes, and the overlay. The cluster operator audits the CNI’s configuration:

# Calico's BGP peers
calicoctl node status

# Calico's IPIP mode
calicoctl get ippool -o yaml

# Flannel's backend
kubectl logs -n kube-system -l app=flannel | grep backend

The CNI’s failure modes are covered in Part XXXVI.

Inspecting Pod-to-Pod traffic

The operator can inspect the traffic:

# Substitute your own Pod name before running:
POD=checkout-7d4f8c9b5-x2kqn

# Capture on the Pod's interface
kubectl exec "$POD" -- tcpdump -i eth0 -n

# Capture on the host's veth
tcpdump -i cali1234 -n

# Capture on the overlay
tcpdump -i flannel.1 -n

The captures show the actual packets. The operator can correlate the Pod’s IP with the host’s veth and the overlay’s tunnel.

The failure modes

The Pod-to-Pod communication’s failure modes:

• No route to the Pod’s IP: the host does not have a route for the Pod’s IP. The fix is to check the CNI plugin’s routes.
• Bridge split: the bridge is missing one of the Pods. The fix is to verify the veth pair.
• Overlay misconfiguration: the VXLAN tunnel is not established. The fix is to check the CNI plugin’s setup.
• BGP misconfiguration: the BGP session is not established. The fix is to check the BGP peers.
• MTU mismatch: the overlay can’t carry the encapsulated packet. The fix is to lower the MTU.

The operational discipline

The Pod-to-Pod communication’s operational discipline:

• Document the cluster’s network topology. The operator must understand the bridge, the route, and the overlay.
• Audit the CNI’s configuration. The CNI’s configuration is the cluster’s network topology.
• Capture the traffic with tcpdump. The operator must be able to capture and analyse the packets.
• Test the network in staging. The Pod network is critical infrastructure; the test must cover every path.
• Monitor the network’s metrics. Latency, packet loss, and BGP session state are the leading indicators.

Quiz

Production discipline

• Pod-to-Pod traffic takes two paths. Same-node via the bridge; cross-node via the overlay or BGP.
• The overlay adds encapsulation overhead. The cluster operator must size the MTU accordingly.
• The routed path is efficient but requires the underlying network to support it. The cluster operator must coordinate with the network team.
• Document the cluster’s network topology. The operator must understand the bridge, the route, and the overlay.
• Capture the traffic with tcpdump. The operator must be able to capture and analyse the packets.
• Test the network in staging. The Pod network is critical infrastructure; the test must cover every path.
• Monitor the network’s metrics. Latency, packet loss, and BGP session state are the leading indicators.
• Plan the network’s evolution. The CNI’s path is set at cluster design time; changing it is disruptive.