Pod-to-Pod across nodes — the cross-node routing problem

Pod-to-Pod communication across nodes is the cluster’s core networking capability. The cluster’s network must route traffic between per-node Pod CIDRs; the CNI plugin implements the routing. This lesson walks the cross-node routing, the overlay and BGP solutions, the MTU implications, and the operational patterns.

The cross-node routing problem

The cluster’s network must route traffic between Pods on different nodes. The Pod’s IP is allocated from the node’s CIDR; the cluster’s network must route between the node’s CIDRs.

flowchart LR
    A[Pod 1<br/>10.244.1.5<br/>node-1] -->|route 10.244.2.0/24| B[Cluster network]
    B -->|route 10.244.1.0/24| C[Pod 2<br/>10.244.2.6<br/>node-2]

The cluster’s network has a route for each node’s CIDR. The Pod’s traffic is routed according to the destination IP.

The routing is implemented by the CNI plugin. The CNI plugin’s implementation is either:

• Overlay: the CNI plugin creates an overlay network that encapsulates the Pod’s traffic. The cluster’s network routes the overlay packets.
• BGP: the CNI plugin uses BGP to advertise the Pod’s CIDRs to the network. The network routes the Pod’s traffic natively.

The two solutions have different trade-offs.

The overlay solution

The overlay solution:

flowchart LR
    A[Pod 1<br/>10.244.1.5<br/>node-1] -->|encapsulated packet| B[Overlay network]
    B -->|route to node-2| C[Pod 2<br/>10.244.2.6<br/>node-2]

The overlay network is a virtual network on top of the cluster’s physical network. The Pod’s traffic is encapsulated in the overlay packets; the cluster’s network routes the overlay packets.

The overlay’s encapsulation:

• VXLAN: the overlay uses VXLAN. The Pod’s traffic is encapsulated in VXLAN packets.
• IPIP: the overlay uses IPIP. The Pod’s traffic is encapsulated in IPIP packets.
• GENEVE: the overlay uses GENEVE. The Pod’s traffic is encapsulated in GENEVE packets.

The overlay’s encapsulation adds overhead. The encapsulated packet is larger than the original packet; the MTU is reduced.

The BGP solution

The BGP solution:

flowchart LR
    A[Pod 1<br/>10.244.1.5<br/>node-1] -->|route 10.244.2.0/24| B[Cluster network]
    B -->|route 10.244.1.0/24| C[Pod 2<br/>10.244.2.6<br/>node-2]

The BGP solution advertises the Pod’s CIDRs to the network. The network routes the Pod’s traffic natively without encapsulation.

The BGP’s advertisement:

• eBGP: the BGP peers are external (different AS). The cluster’s network is a different AS.
• iBGP: the BGP peers are internal (same AS). The cluster’s network is the same AS.

The BGP’s advertisement is the cluster’s native routing. The network’s router knows the Pod’s CIDRs; the network routes the Pod’s traffic.

The trade-offs

The overlay solution’s trade-offs:

• Simplicity: the overlay is simpler to deploy. The cluster’s network does not need to know the Pod’s CIDRs.
• Portability: the overlay works on any network. The cluster’s network does not need to support BGP.
• MTU overhead: the encapsulation adds overhead. The MTU is reduced; the cluster may need to tune the MTU.

The BGP solution’s trade-offs:

• Performance: the BGP solution is faster. The network routes the Pod’s traffic natively.
• Scalability: the BGP solution is more scalable. The network’s routers can handle the Pod’s traffic.
• Network requirements: the BGP solution requires the network to support BGP. The cluster’s network must be configured.

The production rule is to choose the solution that matches the cluster’s network. An overlay is appropriate for a cluster on a network that does not support BGP; a BGP solution is appropriate for a cluster on a network that supports BGP.

The MTU implications

The MTU implications are the operational concern. The overlay’s encapsulation adds overhead; the MTU is reduced.

Standard MTU: 1500 bytes
VXLAN overhead: 50 bytes
Effective MTU: 1450 bytes

The Pod’s application must use the MTU of the overlay network. The application may need to set the MTU manually.

The MTU’s failure modes:

• MTU too large: the packet is fragmented; the performance is degraded.
• MTU too small: the packet is sent in multiple fragments; the performance is degraded.
• PMTUD failure: the packet is dropped; the application retries.

The production rule is to set the MTU of the Pod’s application to the effective MTU of the overlay.

The CNI plugin’s routing

The CNI plugin’s routing:

ip route

default via 10.0.5.1 dev eth0
10.244.1.0/24 dev cali1234 scope link  src 10.244.1.1
10.244.2.0/24 via 10.0.5.22 dev eth0

The node’s routes show the Pod’s CIDRs. The CNI plugin adds the routes on node startup.

The CNI plugin’s routing is the cluster’s networking implementation. The CNI plugin’s configuration is the cluster’s networking configuration.

The cluster’s network design

The cluster’s network design:

flowchart LR
    A[Node 1] -->|10.244.1.0/24| B[Cluster network]
    C[Node 2] -->|10.244.2.0/24| B
    D[Node 3] -->|10.244.3.0/24| B
    E[Node 4] -->|10.244.4.0/24| B
    B -->|BGP or overlay| F[Pod-to-Pod routing]

The cluster’s network is a routed network. The Pod’s CIDRs are advertised via BGP or encapsulated in the overlay.

The cluster’s network design is the operator’s responsibility. The production rule is to design the network to support the cluster’s traffic.

The CNI plugin’s troubleshooting

The CNI plugin’s troubleshooting:

# Substitute your own value before running:
POD=web-5f9c7d8b6c-2xk9p

# Check the Pod's network interfaces
kubectl exec "$POD" -- ip addr show

# Check the Pod's routes
kubectl exec "$POD" -- ip route

# Check the node's routes
ip route

# Check the CNI's logs
journalctl -u kubelet | grep -i cni

The CNI plugin’s troubleshooting is the operator’s diagnostic. The fix is to investigate the CNI plugin’s logs and the network’s routes.

Quiz

Production discipline

• The cross-node routing is the cluster’s core networking capability. The CNI plugin implements the routing.
• Choose the BGP solution for production. The performance is better; the scalability is better.
• Consider the MTU implications. The overlay’s encapsulation reduces the MTU; the application must use the effective MTU.
• Audit the routing at every release. The CNI plugin’s configuration should be version-controlled; the audit catches the failures.
• Monitor the routing metrics. The CNI plugin’s metrics expose the cluster’s networking health; the operator should alert on the threshold.
• Test the routing in non-production. A staging cluster that mirrors production is the right place to test the routing.
• Document the routing design. The CNI plugin is the cluster’s networking implementation; the documentation is the cluster’s networking reference.