Capturing inside a container — Pod-level network namespaces

Capturing inside a Pod is the closest layer to the symptom. The Pod sees the byte stream the application reads and writes — the same eth0 interface, the same routing table, the same source IP. This is the production default for “the application says the connection is wrong.” This lesson covers the kubectl and container mechanics.

Why capture inside the Pod

The Pod’s eth0 is the application’s view of the network. It shows:

• The Pod’s IP (after the CNI assigned it).
• The Service IP, if the application is talking to a Service.
• The TCP retransmissions, RSTs, and connection state the application is dealing with.
• The DNS lookups the application made.

It does not show:

• The CNI’s bridge (cni0) or veth pair traffic.
• The overlay encapsulation (if the destination is on another node).
• Other Pods on the same node.

For “the application is failing to talk to the Service,” the inside-the-Pod capture is exactly what is needed.

kubectl debug and ephemeral containers

kubectl debug (Kubernetes 1.20+) attaches an ephemeral container to a running Pod for debugging. The ephemeral container shares the Pod’s namespace (network, PID, IPC) by default. This is the production path:

# Substitute your own values before running:
NS=production                       # the Pod's namespace
POD=checkout-api-7d9f6c8b45-r4nq2   # the Pod to attach to
CONTAINER=checkout                  # the container to share namespaces with

kubectl debug -n "$NS" -it "$POD" --image=nicolaka/netshoot \
             --target="$CONTAINER"

Defaulting debug container name to debugger-xxxxx.
Pod "<pod>" debug container name: debugger-xxxxx
If you don't see a command prompt, try pressing enter.

The nicolaka/netshoot image is a debug image with tcpdump, curl, dig, iftop, and other network tools pre-installed. --target is the original container the debugger shares namespaces with.

# Inside the debug container
tcpdump -ni eth0 -w /tmp/cap.pcap

The capture writes to the debug container’s filesystem by default. To extract the pcap:

# Same namespace and Pod as the debug session above:
NS=production
POD=checkout-api-7d9f6c8b45-r4nq2

kubectl cp "$NS/$POD:/tmp/cap.pcap" /tmp/cap.pcap \
            -c debugger-xxxxx

Capturing with kubectl debug on a node-level problem

kubectl debug can also create a debug Pod on a specific node with --node and access the node’s namespaces:

# Substitute your own value before running:
NODE=worker-03.example.com

kubectl debug "node/$NODE" -it --image=nicolaka/netshoot

This creates a debug Pod on the node with hostNetwork: true. The Pod sees the node’s interfaces, the node’s routes, the CNI’s bridge, and (if the debug container has the right capabilities) the host’s other namespaces.

# Inside the node-level debug Pod
tcpdump -ni cni0 -w /tmp/cap.pcap 'net 10.244.0.0/16'

This is a halfway point: capture is on the node, but the capture process runs as a Pod (with its own RBAC, its own audit trail, its own pcap file in its own filesystem). The blast radius is still the entire node, but the operator does not need shell on the host.

Sharing the network namespace with a long-running sidecar

For long-running debugging (e.g., “I need to capture for an hour to catch the intermittent failure”), ephemeral containers are wrong — they exist only for the duration of the kubectl debug session. The pattern is a debug sidecar:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
spec:
  template:
    spec:
      shareProcessNamespace: true
      containers:
      - name: app
        image: app:v1
      - name: debug
        image: nicolaka/netshoot
        securityContext:
          capabilities:
            add: ["NET_ADMIN", "NET_RAW"]
        volumeMounts:
        - name: pcap
          mountPath: /pcap
      volumes:
      - name: pcap
        emptyDir: {}

The debug container shares the Pod’s network namespace by default. It runs tcpdump -ni eth0 -w /pcap/cap.pcap -W 10 -C 100 continuously, rotating pcap files. The operator copies the pcap with kubectl cp when an incident is in progress.

Distinguishing Pod-level from node-level capture

The capture point narrows the problem. Pod-level capture localizes to the Pod; node-level capture localizes to the node; cluster-wide capture requires the underlay.

Quiz

Production discipline

• Default to kubectl debug + ephemeral container + netshoot. It is the lowest-friction path, it scopes the capture to the Pod, and it requires no host access.
• Use kubectl cp to extract the pcap — never cat inside the debug session for large files.
• For long-running capture, use an incident-response workflow with a temporary debug Pod, not a permanent sidecar in production manifests.
• Do not install tcpdump on production containers. It changes the image, pollutes the audit trail, and adds attack surface.
• Capture at the layer closest to the symptom. Inside the Pod first; on the node second; on the wire third.