Capture performance, retention, and the cost of seeing every byte

Capture is not free. A tcpdump -i eth0 on a busy node costs CPU, memory, and disk; the cost scales with the traffic rate. Production capture bounds the cost with BPF filters, ring buffers, and time limits; declares the retention up front; and chooses between wire capture, flow logs, and metrics based on the scale and budget of the cluster.

The cost of capture

Wire-level capture with tcpdump has three cost components:

• CPU: the kernel copies every matching packet from the NIC driver into the capture buffer. On a 10 Gbps link with 1 Mpps, this is 1 million copies per second. Modern NICs with AF_PACKET and zero-copy (e.g., tpacket_v3) reduce the cost; on a 1 Gbps link, the cost is typically < 5% CPU.
• Memory: the capture buffer is sized by -B (default 2 MB). With a high packet rate, drops occur if the buffer is too small. tcpdump reports drops with the dropped by kernel line at the end of the capture.
• Disk: the pcap file grows with the capture rate. A 1 Gbps link at full utilization writes ~125 MB/s. A 60-second capture is 7.5 GB.

The production rule: never run an unbounded capture. Every tcpdump invocation must have a time limit, a BPF filter, a ring buffer, or a packet count.

Bounding the cost

The standard bounded capture:

# 60-second capture, 100 MB per file, 5 files ring-buffered
timeout 60 tcpdump -ni eth0 -s 0 \
                  -w /tmp/cap.pcap -W 5 -C 100 \
                  -B 4096 'tcp and port 8080'

The flags:

• timeout 60 — kills the capture after 60 seconds. The capture cannot outlive the change ticket.
• -W 5 -C 100 — 5 files of 100 MB each, replacing the oldest. Worst case: 500 MB on the host.
• -B 4096 — 4 MB capture buffer. Reduces kernel drops on busy nodes.
• 'tcp and port 8080' — BPF filter. Cuts the capture rate by 95% on a node that mostly serves other ports.

The BPF filter is the most underused cost control. A node that runs 100 services has 100 candidate capture targets; capturing every packet is 100x more expensive than capturing the one service being debugged.

Capture and the kubelet’s eviction model

A capture that fills the host filesystem triggers kubelet eviction. The kubelet’s eviction thresholds include imagefs.available, nodefs.available, and nodefs.inodesFree. A capture that fills /tmp (often part of nodefs) trips these thresholds and the kubelet evicts Pods.

# Substitute the node you are capturing on:
NODE=worker-01

# Check the kubelet's eviction thresholds
kubectl get --raw "/api/v1/nodes/$NODE/proxy/stats/summary" | jq '.node.stats.runtimes.imageFs'

# Check current disk usage on the node
df -h /var/lib/kubelet /tmp /var/log

The production rule: capture to a directory the kubelet does not monitor for eviction, or use a separate volume mounted at the capture point. Most production patterns copy the pcap off the host within minutes of stopping the capture.

Retention policy for pcap and flow data

A pcap file contains every byte the operator captured. In a regulated cluster, that is in-scope data. The production retention pattern:

A pcap file in /tmp that survives a reboot is a compliance violation. The standard pattern: copy to a ticket-attached bucket with retention policy, or delete within 24 hours.

The cost/visibility ladder

The choice between capture, flow logs, and metrics is a trade-off:

The production rule: always-on observability is RED metrics and CNI/mesh logs. Wire capture is incident-only, bounded, and declared in the change ticket.

Sampling and selective capture

At high traffic rates, even bounded capture is expensive. The patterns that reduce cost further:

• Sampling: capture 1 in N packets (tcpdump -i eth0 -w /tmp/cap.pcap 'tcp and port 8080' --immediate-mode plus an external sampler).
• Triggered capture: start capturing when a metric crosses a threshold (e.g., 5xx rate > 1%). Tools like tcpdump with a sidecar that monitors metrics can implement this.
• Endpoint capture: capture only the traffic to/from a specific Pod, not the whole node. Cilium’s Hubble can do this via hubble observe --pod <pod>.
• Snapshot capture: capture a specific moment (the failing request) using an Envoy tap or Istio’s on-demand tap.

The most common production pattern is endpoint capture during an incident: the operator enables Hubble or Envoy tap for the failing workload, captures the specific request that failed, and disables the capture as soon as the incident is resolved.

Quiz

Production discipline

• Every capture is bounded. Time limit, BPF filter, ring buffer, or packet count. No unbounded captures on production nodes.
• Capture to a path the kubelet does not evict on. Or copy off the host within minutes of capture.
• Declare the retention path up front. Ticket-attached bucket with automatic deletion, or delete within 24 hours.
• Prefer identity-aware tools for cluster-wide observability. Hubble, Calico flow logs, mesh access logs. Wire capture is incident-only.
• The cost/visibility ladder is the budget tool. RED metrics + flow logs at the bottom (always on); wire capture at the top (incident-only, bounded, declared).