Pod-to-Pod troubleshooting — diagnosing the Pod network

Pod-to-Pod troubleshooting verifies the CNI plugin’s configuration. The diagnostic tools are kubectl exec, nsenter, ip route, tcpdump. This lesson walks the diagnostic flow, the failure modes, and the operational discipline.

The diagnostic flow

The diagnostic flow for Pod-to-Pod traffic:

flowchart TD
    A[Pod-to-Pod failed] --> B{Pods have IPs?}
    B -->|No| C[Check CNI]
    B -->|Yes| D{veth pair present?}
    D -->|No| E[Check CNI logs]
    D -->|Yes| F{routes correct?}
    F -->|No| G[Check IPAM]
    F -->|Yes| H{packet capture}
    H -->|No packets| I[Check host network]
    H -->|packets| J[Check MTU]

The diagnostic flow is the operator’s guide. The cluster operator must follow the flow.

Step 1: Verify the Pod IPs

The first step is to verify the Pods have IPs:

kubectl get pods -o wide

NAME     READY   STATUS    IP            NODE
billing  1/1     Running   10.244.1.5    node-1
database 1/1     Running   10.244.2.5    node-2

The Pods must have IPs. If the Pods do not have IPs, the CNI is not configured.

Step 2: Verify the veth pair

The second step is to verify the veth pair:

# Pod name from the `kubectl get pods -o wide` output above:
POD=billing

kubectl exec "$POD" -- ip link show

eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500

The Pod’s interface is eth0. The host-side veth is cali* (Calico), lxc* (Cilium), or flannel.1 (Flannel):

ip link show | grep -E "cali|lxc"

The veth pair must be on both sides. If the veth is missing, the CNI is failing.

Step 3: Verify the routes

The third step is to verify the routes:

# Pod name from the `kubectl get pods -o wide` output above:
POD=billing

kubectl exec "$POD" -- ip route

default via 169.254.1.1 dev eth0

The Pod’s default route must be present. The host must have a route to the Pod’s IP:

# Pod IP from the `kubectl get pods -o wide` output above:
POD_IP=10.244.1.5

ip route show | grep "$POD_IP"

The route must be on the host’s veth. If the route is missing, the IPAM is failing.

Step 4: Capture the traffic

The fourth step is to capture the traffic:

# Pod name from the `kubectl get pods -o wide` output above:
POD=billing

kubectl exec "$POD" -- tcpdump -i eth0 -n

The capture shows the packets. The operator can analyze the capture with wireshark or tcpdump.

Step 5: Verify the CNI plugin

The fifth step is to verify the CNI plugin:

kubectl logs -n kube-system -l k8s-app=calico-node

The CNI plugin’s logs show the configuration. The operator must verify the CNI is running.

The failure modes

The Pod-to-Pod traffic’s failure modes:

• Pod has no IP: the CNI is not configured. The fix is to check the CNI’s logs.
• veth pair missing: the CNI is failing. The fix is to delete the Pod so the sandbox is rebuilt and the CNI ADD runs again.
• Route missing: the IPAM is failing. The fix is to check the IPAM.
• MTU mismatch: the encapsulation overhead is too high. The fix is to lower the MTU.
• CNI plugin crashed: the CNI is not running. The fix is to restart the CNI.

The operational discipline

The Pod-to-Pod troubleshooting’s operational discipline:

• Document the diagnostic flow. The flow is the cluster’s operational reference.
• Use the diagnostic tools. The tools are the operator’s primary tools.
• Follow the checklist. The checklist is the operator’s guide.
• Document the troubleshooting. The troubleshooting is the cluster’s operational reference.
• Train the operations team on the diagnostics. The diagnostics are the team’s tools.
• Set up alerts on the CNI metrics. The alerts are the leading indicator of outages.

Quiz

Production discipline

• The diagnostic flow is systematic. The cluster operator must follow the flow.
• Document the diagnostic flow. The flow is the cluster’s operational reference.
• Use the diagnostic tools. The tools are the operator’s primary tools.
• Follow the checklist. The checklist is the operator’s guide.
• Document the troubleshooting. The troubleshooting is the cluster’s operational reference.
• Train the operations team on the diagnostics. The diagnostics are the team’s tools.
• Set up alerts on the CNI metrics. The alerts are the leading indicator of outages.
• Document the troubleshooting in the runbook. The runbook is the cluster’s operational reference.