tcpdump in Kubernetes — the flags that matter

tcpdump on a Linux VM is straightforward. tcpdump on a Kubernetes node is the same tool with three additional wrinkles: BPF expressions that match Pod CIDRs and Service IPs, overlay decoders that surface the inner packet, and file rotation that does not fill the host filesystem. This lesson walks the flags that matter.

The minimum tcpdump command for Kubernetes

tcpdump -ni eth0 -s 0 -w /tmp/cap.pcap -W 5 -C 100 \
        -e -tttt 'net 10.244.0.0/16'

The cluster’s Pod CIDR is configured at kubeadm init time or by the CNI. The standard defaults are 10.244.0.0/16 (Flannel, Calico default), 10.0.0.0/8 (some EKS), or 192.168.0.0/16. The cluster operator should know the CIDR; the kubectl cluster-info dump output includes it.

kubectl cluster-info dump | grep -i cidr

BPF expressions for Kubernetes

BPF (Berkeley Packet Filter) is the in-kernel filter language tcpdump compiles your expression into. The expressions that matter for Kubernetes:

# A specific Pod IP
host 10.244.1.5

# A specific Service IP (ClusterIP)
host 10.96.45.200

# The Pod CIDR (all Pods)
net 10.244.0.0/16

# The Service CIDR (typically 10.96.0.0/12)
net 10.96.0.0/12

# A specific port
port 8080

# Combinations
'tcp and port 8080 and net 10.244.0.0/16'
'(src host 10.244.1.5 or src host 10.244.1.6) and dst port 443'
'tcp[tcpflags] & (tcp-syn) != 0 and not src host 10.244.1.5'

The last expression — tcp[tcpflags] — is a tcpdump extension that examines TCP flags directly. It catches SYNs, which is what you want when the symptom is “the connection is not establishing.”

# Show only TCP SYNs (connection attempts)
tcpdump -ni eth0 'tcp[tcpflags] & tcp-syn != 0 and tcp[tcpflags] & tcp-ack == 0'

Overlay decoding

A cluster with a VXLAN overlay hides Pod IPs from the wire. tcpdump decodes VXLAN if it sees UDP/4789 and the packet is large enough:

tcpdump -ni eth0 -e -vv 'udp port 4789'

13:42:01.123456 node-1 > node-2: VXLAN, flags [I] (0x08), vni 1
  IP 10.244.1.5.42184 > 10.244.2.6.8080: Flags [S], seq 100, win 64240

The first line is the outer header (UDP/4789, VXLAN VNI). The second is the inner IP packet. The Pod IPs are visible.

For IPIP (another common overlay), the tunnel device is typically tunl0:

tcpdump -ni tunl0 host 10.244.1.5

For Geneve (used by some Calico configurations), UDP/6081:

tcpdump -ni eth0 -e -vv 'udp port 6081'

Snap length and ring buffers

tcpdump writes every packet to disk by default. On a busy node, a 1-minute capture can be hundreds of MB. The production-grade capture uses:

• -s 0 — full snap length (no truncation).
• -W 5 -C 100 — 5 files of 100 MB each, ring-buffered.
• -c 1000 — stop after N packets (good for “I want to see the next 1000 SYNs”).

The ring-buffer pattern is the production default: a bounded pcap that survives a crash, rotates without filling the disk, and can be deleted in one command after the investigation.

# Capture the next 60 seconds, rotating every 100 MB
timeout 60 tcpdump -ni eth0 -s 0 -w /tmp/cap.pcap \
         -W 10 -C 100 'tcp and port 8080'

The -W flag is the file count; the -C flag is the file size in MB. With 10 files of 100 MB, the worst case is 1 GB on the host. Plan accordingly.

Reading the pcap after capture

Reading the pcap on the same host re-runs the kernel’s packet path and pollutes the capture with the reader’s packets. The production pattern:

1. Capture on the node.
2. Copy the pcap to an analyst workstation with scp or kubectl cp (from a debug Pod).
3. Read on the workstation with tcpdump -r cap.pcap or tshark -r cap.pcap.

# Read the pcap on an analyst workstation
tcpdump -nr /tmp/cap.pcap 'tcp and port 8080'

# Or with tshark for protocol-aware decoding
tshark -r /tmp/cap.pcap -Y 'tcp.port == 8080' \
       -T fields -e ip.src -e ip.dst -e tcp.srcport -e tcp.dstport

The -r flag reads from a file. Do not re-capture on the host.

Time-stamping and clock skew

A cluster’s nodes run NTP, but a capture that spans nodes needs the timestamps to be comparable. Use -j adapter or -tttt for human-readable time, and confirm the host clock is sane:

# Check the host clock
date -u; ntpq -p

# Capture with nanosecond precision
tcpdump -ni eth0 --time-stamp-type=adapter -tttt

A 100 ms clock skew between two nodes turns a “packet arrived at node-2 50 ms after node-1 sent it” observation into garbage. The capture must be on NTP-synced hosts.

Quiz

Production discipline

• Always use -n to disable DNS resolution. A capture that triggers reverse lookups pollutes the host resolver and the capture itself.
• Always use -s 0 for full snap length. The legacy default truncates packets and hides application data.
• Always use a ring buffer (-W N -C M). A capture that fills the host disk is a self-inflicted outage.
• Always use a BPF filter. Capturing every packet on eth0 writes the entire node’s traffic to disk. Filter on the Pod CIDR, the Service CIDR, or a specific port.
• Copy the pcap off the host before reading. Reading on the host with -i eth0 while another process is capturing pollutes both captures.
• Confirm the overlay before drawing conclusions. Calico VXLAN uses UDP/4789, IPIP uses tunl0, Cilium VXLAN uses UDP/8472, BGP-routed clusters have no encapsulation.