Packet drops — softnet_stat, per-NIC counters, drop reasons

A “packet drops” report is a different problem than a “CPU saturation” report: the CPU may be idle while packets are dropped. The drops can happen at three layers: at the NIC ring buffer (the NIC received more packets than the ring can hold); at the kernel’s per-CPU input queue (softnet_data); or at the socket receive buffer (a userspace consumer is too slow to drain its socket). Each layer has a different counter, a different cause, and a different fix.

This lesson is the production reference for packet drops on VyOS 1.5 LTS: the per-CPU and per-NIC counters, the drop reasons each counter reveals, and the canonical fix for each.

Where packets get dropped

A packet traverses three buffers between the wire and a userspace consumer:

flowchart LR
  WIRE["Wire"] --> RING["NIC ring buffer<br/>(DMA ring)<br/>rx_dropped, rx_missed_errors"]
  RING --> BACKLOG["Per-CPU input queue<br/>(softnet_data)<br/>dropped in softnet_stat"]
  BACKLOG --> SOCK["Socket receive buffer<br/>(sk_buff queue)<br/>rx_dropped in /proc/Net/sockstat"]
  SOCK --> APP["Userspace consumer<br/>(FRR, ssh, etc.)"]

The three buffers have different sizes and different drop reasons:

1. NIC ring buffer. Sized by the NIC driver (ethtool -g <nic> shows the max). Typical sizes: 256 to 4096 descriptors. The ring is DMA’d directly from the NIC; the kernel reads from it in softirq context. If the kernel cannot drain the ring fast enough, the NIC’s head pointer overruns the tail pointer; the NIC drops the packet and increments rx_missed_errors (or rx_dropped depending on the driver).

2. Per-CPU input queue. Sized by net.core.netdev_max_backlog (default 1000, raise to 10000 for routers). The kernel’s softirq pass moves packets from the ring into this queue for processing. If the queue is full, the kernel drops the packet and increments the dropped column in /proc/net/softnet_stat.

3. Socket receive buffer. Sized by net.core.rmem_max (default 212 KiB, raise to 4-16 MiB for routers). Each socket has its own buffer; when a packet arrives for a socket whose buffer is full, the kernel drops the packet and increments prune or drop in the socket’s drop counters.

A common operator error is to treat all three as a single problem and apply a single fix (e.g., raise the ring buffer to maximum). The right fix depends on which counter is climbing.

Reading softnet_stat

/proc/net/softnet_stat has one row per CPU and three columns the operator cares about:

$ cat /proc/net/softnet_stat
0123456789abcdef 0000000000000000 0000000000000001 0000000000000000 ...

The columns (from the kernel source) are:

1. Processed. Total packets processed by this CPU’s softirq pass. Climbs steadily under load.
2. Dropped. Packets dropped because the per-CPU input queue was full. Climbs when the softirq pass cannot keep up.
3. Time squeeze. Number of times the softirq pass was scheduled out before draining the backlog. Climbs when the softirq pass runs out of budget.

The columns are little-endian hex. The operator reads the Dropped column to know whether packets are being dropped at the per-CPU queue.

A more readable format uses awk:

$ cat /proc/net/softnet_stat | awk '{
    processed = strtonum("0x" $1);
    dropped   = strtonum("0x" $2);
    squeezed  = strtonum("0x" $3);
    printf "CPU %d: processed=%-10d dropped=%-10d squeezed=%d\n",
        NR-1, processed, dropped, squeezed
}'
CPU 0: processed=12345678 dropped=12345 squeezed=56
CPU 1: processed=12345679 dropped=0     squeezed=0
CPU 2: processed=12345680 dropped=0     squeezed=0
CPU 3: processed=12345681 dropped=0     squeezed=0

In this example, CPU 0 is dropping packets while CPUs 1-3 are clean. This is the signature of single-core softirq saturation (Part L-02); the fix is interrupt distribution or NIC upgrade. The discipline: softnet_stat is the canonical evidence that the per-CPU queue is the drop layer.

Reading per-NIC statistics

The kernel exposes per-NIC counters at /sys/class/net/<nic>/statistics/. The most important counters:

$ ls /sys/class/net/eth0/statistics/
collisions   rx_crc_errors      rx_fifo_errors   rx_missed_errors
multicast_pkts rx_dropped      rx_frame_errors  rx_nohandler
tx_dropped   tx_aborted_errors  tx_carrier_errors tx_compressed
tx_heartbeat_errors tx_packets  tx_window_errors

The operator reads the ones that diagnose packet drops:

• rx_dropped — packets dropped at the NIC driver layer (ring full, allocation failure). Compare with ethtool -S <nic> for driver-specific breakdown.
• rx_missed_errors — packets missed by the NIC hardware (FIFO overrun, descriptor unavailable). Driver-specific; some drivers never increment this and put everything in rx_dropped.
• tx_dropped — packets the kernel queued for transmit but the NIC did not send. Usually a NIC error or a queue saturation.
• rx_fifo_errors — receive FIFO overruns on the NIC. Rare on modern NICs; indicates the NIC’s internal buffer is overrun.
• rx_nohandler — packets that arrived for a protocol nobody is listening for (e.g., IPv6 packet when only IPv4 is configured). Usually indicates a misconfiguration, not a performance issue.

$ cat /sys/class/net/eth0/statistics/rx_dropped
12345

$ cat /sys/class/net/eth0/statistics/rx_missed_errors
0

$ cat /sys/class/net/eth0/statistics/tx_dropped
0

The discipline: read multiple counters. rx_dropped climbing with rx_missed_errors at zero is a software-drop pattern; both climbing is a hardware-drop pattern. Each pattern has a different fix.

Driver-specific counters with ethtool

The kernel’s per-NIC counters are aggregated; the driver has its own detailed counters accessible via ethtool -S:

$ ethtool -S eth0 | head -30
NIC statistics:
     rx_packets: 12345678
     tx_packets: 12345678
     rx_bytes:   9876543210
     tx_bytes:   9876543211
     rx_errors:  1234
     tx_errors:  0
     rx_dropped: 1234
     tx_dropped: 0
     rx_length_errors: 0
     rx_over_errors:  0
     rx_crc_errors:   0
     rx_frame_errors: 0
     rx_fifo_errors:  0
     rx_missed_errors: 1234
     tx_aborted_errors: 0
     tx_carrier_errors: 0
     tx_fifo_errors: 0
     tx_heartbeat_errors: 0

The driver-specific counters reveal exactly where the drop happened. For Intel ixgbe (10 Gbps), the relevant counters are rx_dropped, rx_missed_errors, rx_no_dma_resources. For Mellanox mlx5, the relevant counters are rx_discard_phy, rx_steer_missed_packets, rx_csum_complete. The operator must consult the driver documentation to know which counter maps to which drop reason.

The canonical fixes

Each drop layer has a different fix:

NIC ring buffer full (rx_missed_errors, driver-specific rx_dropped)

The fix is to raise the NIC’s ring buffer size:

# Current
$ ethtool -g eth0
Ring parameters for eth0:
Pre-set maximums:
RX:             4096
RX Mini:        0
RX Jumbo:       0
TX:             4096
Current hardware settings:
RX:             256
TX:             256

# Raise to maximum
$ ethtool -G eth0 rx 4096

The kernel and driver negotiate the new ring size; the change is immediate. To persist across reboot, configure via VyOS:

configure
set system ethernet eth0 ring-buffer rx 4096
commit

Per-CPU input queue full (Dropped in softnet_stat)

The fix is to raise net.core.netdev_max_backlog:

$ sysctl net.core.netdev_max_backlog
net.core.netdev_max_backlog = 1000

$ sysctl -w net.core.netdev_max_backlog=10000

The change is immediate. To persist across reboot, configure via VyOS:

configure
set system sysctl parameter net.core.netdev_max_backlog value 10000
commit

The discipline: if the Dropped column is climbing, the root cause is usually softirq saturation (Part L-01), not a small backlog. Raising netdev_max_backlog to 10000 helps with bursts but does not fix the underlying softirq overload. Combine with RSS (Part L-02).

Socket receive buffer full (rx_dropped in /proc/net/sockstat)

The fix is to raise net.core.rmem_max and the per-protocol buffer:

$ sysctl net.core.rmem_max
net.core.rmem_max = 212992

$ sysctl -w net.core.rmem_max=8388608
$ sysctl -w net.ipv4.tcp_rmem="4096 87380 8388608"

For UDP-heavy workloads, also raise net.ipv4.udp_rmem_min. For applications that use large socket buffers (BGP, telemetry), the application may need to call setsockopt(SO_RCVBUF) explicitly.

To persist across reboot via VyOS:

configure
set system sysctl parameter net.core.rmem_max value 8388608
set system sysctl parameter net.ipv4.tcp_rmem value "4096 87380 8388608"
commit

Production failure modes

The packet-drop failure modes the operator encounters:

• Ring buffer too small. rx_missed_errors climbing on a busy NIC. Fix: raise the ring size with ethtool -G.
• Per-CPU queue too small. Dropped in softnet_stat climbing on one or more CPUs. Fix: raise netdev_max_backlog, combine with RSS for single-core saturation.
• Socket buffer too small. TCP connections slow or stalled; ss -m shows Recv-Q non-zero (data waiting in the socket buffer). Fix: raise rmem_max and tcp_rmem.
• Drop at the application. The kernel’s counters are clean; the application is not reading its socket fast enough. Fix: tune the application (BGP hold-time, telemetry rate, SNMP polling interval) or scale out.
• TCP backpressure dropping SYN. tcp_syncookies is enabled because the SYN backlog is full. Fix: raise net.ipv4.tcp_max_syn_backlog.
• ARP table full. neighbor table overflow in the kernel log; the router is dropping ARP requests. Fix: raise net.ipv4.neigh.default.gc_thresh3 or shorten the GC interval.

Rollback

Packet-drop fixes are typically small but the impact can be large (a too-large ring buffer can hurt latency; a too-large socket buffer can starve other sockets). The rollback discipline:

• ethtool -G <nic> rx <n> — restore the original ring size.
• sysctl -w <key>=<value> — restore the original sysctl value.
• VyOS sysctl knobs — delete system sysctl parameter <key> and commit.
• For all changes, use commit-confirm 5 so the auto-rollback fires if the change has unintended consequences.

Production discipline

Cross-course references

• Part L-01 (L-VyOS-Performance / CPU saturation) covers the diagnostic method that identifies single-core softirq saturation, which is the root cause of many per-CPU queue drops.
• Part L-02 (L-VyOS-Performance / interrupt affinity) covers RSS, which is the canonical fix for single-core softirq.
• Part L-04 (L-VyOS-Performance / crypto load) covers IPsec throughput, where drops are common because the encryption pipeline is single-core.
• Part LI (LI-VyOS-MTU) covers MTU and fragmentation, where large packets are dropped at the fragmentation layer rather than the NIC.
• The Linux course’s V-Linux-NetConfig and XXII-Linux-NetTroubleshoot parts cover the same primitives from the host perspective (sysctl, ethtool, NIC drivers).
• The Ansible course’s XLII-Ansible-BeyondLinux covers the automation hand-off (drop counter dashboards in Grafana).

Quiz