Packet capture and network performance — diagnosing the network

Packet capture is the operator’s last resort for diagnosing the network. The capture is performed with tcpdump on the Pod’s interface or the host’s veth. This lesson walks the packet capture, the analysis, and the operational discipline.

The packet capture

The packet capture is performed with tcpdump:

# Substitute your own Pod name:
POD=web-5f9c7d8b6c-2xk9p

kubectl exec "$POD" -- tcpdump -i eth0 -w /tmp/capture.pcap

The capture is written to a file. The file can be downloaded and analyzed with wireshark.

# The same Pod you captured on above:
POD=web-5f9c7d8b6c-2xk9p

kubectl cp "$POD":/tmp/capture.pcap /tmp/capture.pcap
wireshark /tmp/capture.pcap

The capture shows the packets on the Pod’s interface.

The host-side capture

The host-side capture is performed on the host’s veth:

# Find the veth
ip link show | grep -E "cali|lxc"

# Capture on the veth
tcpdump -i cali1234 -w /tmp/capture.pcap

The host-side capture is more reliable than the Pod-side capture. The Pod’s image may not include tcpdump.

The capture analysis

The capture is analyzed with wireshark or tcpdump:

tcpdump -r /tmp/capture.pcap -nn

12:00:00.000 IP 10.244.1.5.8080 > 10.244.2.5.5432: Flags [S], seq 1234, win 64240, length 0
12:00:00.001 IP 10.244.2.5.5432 > 10.244.1.5.8080: Flags [S.], seq 5678, ack 1235, win 64240, length 0
12:00:00.002 IP 10.244.1.5.8080 > 10.244.2.5.5432: Flags [.], ack 5679, win 64240, length 0

The capture shows the TCP handshake. The operator can analyze the timing, the flags, and the payload.

The network performance metrics

The network performance metrics include:

• Latency: the round-trip time between the source and the destination.
• Packet loss: the percentage of packets lost.
• Bandwidth: the throughput between the source and the destination.
• Jitter: the variation in latency.

The metrics are collected via:

# Substitute your own values before running:
DEST=192.0.2.25
PORT=8080

# ICMP latency
mtr "$DEST"

# TCP latency
tcping "$DEST" "$PORT"

# Bandwidth
iperf3 -c "$DEST"

The metrics are the operator’s primary tools for network performance.

The network performance troubleshooting

The network performance troubleshooting:

flowchart TD
    A[Network slow] --> B{Latency high?}
    B -->|Yes| C[Capture packets]
    B -->|No| D{Packet loss?}
    D -->|Yes| E[Check routes]
    D -->|No| F{Bandwidth low?}
    F -->|Yes| G[Check MTU]
    F -->|No| H[Check application]

The troubleshooting is the operator’s guide. The cluster operator must follow the flow.

The failure modes

The packet capture and network performance’s failure modes:

• tcpdump not installed: the Pod’s image does not include tcpdump. The fix is to use a debug sidecar.
• Permission denied: the Pod’s security context denies the packet capture. The fix is to grant the permission.
• MTU mismatch: the packets are fragmented. The fix is to match the MTU.
• NetworkPolicy blocks: the NetworkPolicy blocks the traffic. The fix is to add the allow rules.

The operational discipline

The packet capture’s operational discipline:

• Document the diagnostic flow. The flow is the cluster’s operational reference.
• Use the diagnostic tools. The tools are the operator’s primary tools.
• Follow the checklist. The checklist is the operator’s guide.
• Document the troubleshooting. The troubleshooting is the cluster’s operational reference.
• Train the operations team on the diagnostics. The diagnostics are the team’s tools.
• Set up alerts on the network metrics. The alerts are the leading indicator of outages.
• Use the host-side capture when possible. The host-side capture is more reliable than the Pod-side capture.

Quiz

Production discipline

• The packet capture is the operator’s last resort. The cluster operator must use the systematic methodology first.
• Document the diagnostic flow. The flow is the cluster’s operational reference.
• Use the diagnostic tools. The tools are the operator’s primary tools.
• Follow the checklist. The checklist is the operator’s guide.
• Document the troubleshooting. The troubleshooting is the cluster’s operational reference.
• Train the operations team on the diagnostics. The diagnostics are the team’s tools.
• Set up alerts on the network metrics. The alerts are the leading indicator of outages.
• Use the host-side capture when possible. The host-side capture is more reliable than the Pod-side capture.
• Document the troubleshooting in the runbook. The runbook is the cluster’s operational reference.