tcpdump and packet capture - seeing the bytes

tcpdump is the standard packet capture tool on Linux. It shows exactly what is on the wire, which is the only way to debug some classes of network problems. Modern alternatives like tshark (Wireshark CLI) and ngrep have more features but tcpdump is the one always installed.

The basics

sudo tcpdump -i eth0                    # capture on eth0
sudo tcpdump -i any                     # capture on all interfaces
sudo tcpdump -i eth0 -n                 # no name resolution
sudo tcpdump -i eth0 -nn                # no resolution of host or port
sudo tcpdump -i eth0 -w capture.pcap    # write to file
sudo tcpdump -i eth0 -c 100             # stop after 100 packets
sudo tcpdump -i eth0 -s 96              # truncate to 96 bytes (headers only)
sudo tcpdump -i eth0 -A                 # ASCII output (for HTTP etc.)
sudo tcpdump -i eth0 -X                 # hex + ASCII

Production use almost always wants -n or -nn (skip DNS, which can be slow) and -w file.pcap (write to file for analysis).

You will see -s 0 in almost every tcpdump recipe on the internet. It is a no-op on anything current: man 8 tcpdump gives the default snaplen as 262144 bytes, which is larger than any Ethernet frame you will capture. -s 0 mattered when the default was 68 or 96 bytes and payload was silently truncated. Set -s deliberately today only when you want smaller captures — -s 96 keeps the headers and drops the payload, which cuts file size by an order of magnitude on a bulk-transfer link and keeps application data out of the pcap in a regulated environment.

Filters

The filter language lets you select which packets to capture or display:

sudo tcpdump -i eth0 host 10.0.0.5          # to or from 10.0.0.5
sudo tcpdump -i eth0 src 10.0.0.5           # from 10.0.0.5 only
sudo tcpdump -i eth0 dst 10.0.0.5           # to 10.0.0.5 only
sudo tcpdump -i eth0 port 443               # port 443 any direction
sudo tcpdump -i eth0 tcp                    # TCP only
sudo tcpdump -i eth0 udp                    # UDP only
sudo tcpdump -i eth0 'tcp port 443'         # TCP and port 443
sudo tcpdump -i eth0 'src 10.0.0.5 and port 443'
sudo tcpdump -i eth0 'net 10.0.0.0/24'      # entire subnet
sudo tcpdump -i eth0 'tcp[tcpflags] & (tcp-syn) != 0'   # SYN packets
sudo tcpdump -i eth0 'tcp[tcpflags] & (tcp-rst) != 0'   # RST packets
sudo tcpdump -i eth0 'port 443 and not host 10.0.0.5'   # exclude one host

Combine with and, or, not. Parenthesise when needed.

Reading the output

12:34:56.789012 IP 10.0.0.10.443 > 10.0.0.5.51234: Flags [S.], seq 12345, ack 67890, win 64240, length 0

Reading this:

• Timestamp (12:34:56.789012): when the packet was captured, with microsecond precision.
• IP (or IP6, ARP, etc.): the layer-3 protocol.
• Source and destination (10.0.0.10.443 > 10.0.0.5.51234): the source IP:port and destination IP:port.
• Flags ([S.]): TCP flags. [S] is SYN, [S.] is SYN-ACK (the . is ACK), [.] is ACK, [F] is FIN, [R] is RST, [P] is PSH.
• Sequence and ACK numbers: TCP sequence state. Useful for detecting retransmissions.
• Window: the receive window size.
• Length: payload size (0 means no payload).

What to look for

Three-way handshake: capture should show [S] from client, [S.] from server, [.] from client. Anything else is a problem.

TCP reset (RST): a peer is rejecting the connection. Cause: no listener on the port, firewall rejecting, or the peer is sick.

Retransmissions: same SYN or same data packet sent multiple times. Cause: packet loss, congestion, or the peer not responding.

Out-of-order: packets arriving with sequence numbers earlier than expected. Cause: routing change, ECMP load-balancing with asymmetric paths.

Half-open connections: client sent SYN, never got a SYN-ACK, eventually gave up. Cause: server overloaded, SYN flood, firewall dropping.

Where tcpdump sits in the packet path

This is the single most useful fact about tcpdump, and it is the one most operators never learn. tcpdump attaches at the device layer through AF_PACKET. That is below netfilter. It is not a view of what the host accepted. It is a view of what crossed the NIC.

Inbound:  wire -> NIC -> [tcpdump sees it here] -> netfilter
          PREROUTING/INPUT -> socket

Outbound: socket -> netfilter OUTPUT/POSTROUTING ->
          [tcpdump sees it here] -> NIC -> wire

So a packet appears in an ingress capture even when the INPUT chain drops it a microsecond later. That asymmetry is what makes the tool decisive. Run the capture on the destination host and read it like this:

The inverse trap matters just as much. Because tcpdump sits below netfilter on egress too, a packet you see leaving in a capture has already passed OUTPUT. Seeing it in the capture does not mean it reached the wire beyond the NIC, and it says nothing about what a downstream firewall did to it.

Capture on both ends whenever you can. “Sent on the client, absent on the server” localises the drop to the path in one step, with no rule reading and no guessing.

# On the client, prove the SYN left
sudo tcpdump -i eth0 -nn 'host 10.0.0.10 and tcp port 443 and tcp[tcpflags] & tcp-syn != 0'

# On the server, prove whether it arrived and what came back
sudo tcpdump -i eth0 -nn 'host 10.0.0.5 and tcp port 443'

To see what netfilter itself did with a packet, you need a different tool: nft monitor trace with a meta nftrace set 1 rule, or the LOG target. tcpdump cannot answer that question, because it never sees the verdict.

Production recipes

# Capture all HTTP traffic for one host
sudo tcpdump -i eth0 -nn -A 'host 10.0.0.5 and tcp port 80'

# Capture TCP resets to find dropped connections
sudo tcpdump -i eth0 -nn 'tcp[tcpflags] & (tcp-rst) != 0'

# Capture for 60 seconds then write to file
sudo timeout 60 tcpdump -i eth0 -nn -w /var/tmp/capture.pcap

# Capture only SYN packets (handshake attempts)
sudo tcpdump -i eth0 -nn 'tcp[tcpflags] & (tcp-syn) != 0 and tcp[tcpflags] & (tcp-ack) = 0'

# Capture to file for offline analysis
sudo tcpdump -i eth0 -nn -w /var/tmp/capture.pcap host 10.0.0.5

Capturing DNS on a stub-resolver host

The recipe everyone reaches for is tcpdump -i eth0 'udp port 53', and on any host running systemd-resolved — which this course teaches as the modern default — it will not show you the query you are looking for. The application resolves against 127.0.0.53, so its query crosses the loopback interface, never eth0. What eth0 carries is resolved’s onward query to the upstream, and on a cache hit there is no onward query at all. An empty eth0 capture is therefore not evidence that the application did not ask.

# Leg 1: what the application actually asked for
sudo tcpdump -i lo -nn 'port 53'

# Leg 2: what the stub forwarded upstream (absent on a cache hit)
sudo tcpdump -i eth0 -nn 'port 53'

# Both legs at once
sudo tcpdump -i any -nn 'port 53'

Two details in those filters matter. Use port 53, not udp port 53: an answer larger than the advertised EDNS buffer comes back truncated with the TC flag set and the resolver retries over TCP/53, and a UDP-only filter makes exactly the failure you are chasing invisible. And -i any on Linux uses a cooked (SLL) capture with no Ethernet header, so -e and any MAC-level filter stop working on it — use a named interface when you need link-layer detail.

Rotate captures in production to limit disk use:

sudo tcpdump -i eth0 -nn -w /var/tmp/cap.pcap -W 5 -C 100

-W 5 keeps 5 files and -C 100 rotates when the current file passes 100 million bytes. man 8 tcpdump is explicit that the unit is 1,000,000 bytes, not 1,048,576 — so -C 100 is ~95.4 MiB, and -W 5 -C 100 puts a ceiling of roughly 500 MB on the capture. Size that against real free space before you start, and write to /var/tmp or a dedicated volume, never to /: a capture that fills the root filesystem on a production host turns a network investigation into an outage.

Knowledge check