Packet capture in Kubernetes — what you actually need and why

Packet capture in Kubernetes is not tcpdump on a host. The Pod, the host, and the control plane sit in different network namespaces; overlay encapsulation hides Pod-to-Pod traffic from the host interface; and the multi-tenant blast radius of capture on a shared node is large. This lesson sets the frame: where you can capture, what each option actually shows you, and what to never do in production.

Why packet capture comes up in production

A SRE on call does not reach for tcpdump for fun. They reach for it when:

• An application behaves correctly against a sidecar and fails against a Service.
• A connection that works from inside the Pod fails from outside, or vice versa.
• A workload that runs fine on a Linux VM breaks the moment it is moved into a Pod.
• mTLS, Istio, or Linkerd rewrites traffic, and the rewritten byte stream is what the application is debugging.
• kubectl logs is empty because the failure is at L4 — the TCP handshake fails, packets are dropped, or a firewall rejects the SYN before any application log can fire.

The first diagnostic move is almost always kubectl describe, then kubectl logs, then metrics. Packet capture is the move after the application-layer investigation has narrowed the failure to a network-layer problem. Operators who reach for tcpdump first spend hours chasing packets when the answer was in a startup probe.

Where the capture can attach

Kubernetes has four natural capture points:

flowchart LR
    A[Pod A<br/>10.244.1.5] --> B[veth pair on host]
    B --> C[cni0 bridge]
    C --> D[node-1 host interface]
    D --> E[underlay network]
    E --> F[node-2 host interface]
    F --> G[cni0 bridge on node-2]
    G --> H[veth pair on node-2]
    H --> I[Pod B<br/>10.244.2.6]
    subgraph "capture points"
      P1["1. inside Pod A<br/>eth0"]
      P2["2. on node-1<br/>cni0 / host-side veth"]
      P3["3. on node-1<br/>eth0 (overlay or underlay)"]
      P4["4. inside a sidecar<br/>sharing Pod A's net ns"]
    end

Each capture point shows a different view of the same packet:

A packet that begins as src=10.244.1.5 dst=10.244.2.6 inside Pod A travels as src=node-1-IP dst=node-2-IP (VXLAN) on the wire, then re-emerges as src=10.244.1.5 dst=10.244.2.6 inside Pod B. A capture on node-1 eth0 shows the encapsulated form. A capture on node-2 cni0 shows the decapsulated form.

Overlay encapsulation makes capture deceptive

Most production CNIs use an overlay (VXLAN, IPIP, Geneve) to span nodes without changing the underlay. The Pod IP is hidden from the wire between nodes. A capture on node-1 eth0 shows:

# VXLAN-encapsulated
outer IP: node-1 -> node-2
UDP dst port 4789
  VXLAN VNI 1
    inner IP: 10.244.1.5 -> 10.244.2.6

Operators who do not realize the cluster uses an overlay look at the outer addresses, conclude the wrong Pod is the source, and chase a phantom routing problem. The fix is to filter on the inner IP with tcpdump’s vxlan decoder or to use a CNI-aware tool that does the decap for you.

Multi-tenant blast radius

Capturing on a node’s cni0 or eth0 interface sees every Pod on the node, not just the Pod the operator is debugging. In a multi-tenant cluster, this is a privacy incident waiting to happen: HTTP plaintext, credentials in URLs, PII, JWTs — all visible in cleartext to anyone with shell on the node.

The production rules:

• Capture on a dedicated debug node (a node labeled debug=true with only debug workloads).
• Capture inside a sidecar sharing the Pod’s network namespace, not on the host.
• Capture inside a debug Pod in the same namespace as the suspect workload, attached to a debug emptyDir volume.
• Never capture on a multi-tenant node without an explicit change ticket, RBAC scoping, and a written retention plan for the pcap.

Legal and policy constraints

Production clusters run under contracts that govern captured data:

• PCI-DSS: card data is in scope; pcap files containing PAN are themselves in-scope artifacts. Retention must be declared; encryption at rest is required.
• HIPAA: PHI is in scope; the same constraints apply, with breach-notification exposure.
• GDPR: personal data captured in a pcap on EU residents is processing under GDPR. The lawful basis, retention, and access controls must be declared.
• SOC 2: the change ticket, the access log, and the deletion of the pcap are evidence.

Capture is not a free action in regulated clusters. Every capture carries the same paperwork as a code change.

Quiz

Production discipline

• Pick the capture point from the symptom. Inside the Pod first, the node second, the wire third. Operators who default to tcpdump -i eth0 see the overlay and chase the wrong layer.
• Capture on a debug node, not a multi-tenant node. A pcap on a multi-tenant node is a privacy incident. The default is a dedicated node pool with only debug workloads, a retention policy on the pcap, and a deletion timestamp.
• Account for overlay encapsulation. VXLAN hides Pod IPs from the wire; tcpdump without VXLAN decoding shows node IPs. Decode the inner header before drawing conclusions.
• Treat pcap as regulated data. Card data, PHI, PII, and JWTs appear in pcap files. The retention, access, and deletion rules are part of the change ticket.
• Capture is a last resort. kubectl describe, kubectl logs, and metrics solve most failures in seconds. Packet capture is the move for failures where the application logs are empty because the packet never reached the application.