Interface Packet Drops · Break/Fix · VyOS

Reported symptoms

dist1 is a four-core VyOS router aggregating a campus. eth1 is its uplink into the distribution layer and carries most of the traffic on the box.

The ticket is six days old and has collected four complaints from three teams. Monitoring alerts on rising receive drops on eth1. The NOC has responded the way the alert invites: an SFP swap in one maintenance window, a fibre patch lead in another. The counter climbed through both. The storage team has a separate ticket saying the nightly replication used to finish in about 40 minutes and now runs past three hours, in one direction only. The monitoring team has a third saying SNMP polls of dist1 time out every few minutes, leaving gaps in exactly the graphs everyone is trying to read.

Two observations have kept the investigation from converging. Interactive traffic is perfect — SSH through the port is crisp and small transfers complete normally — and only sustained streams degrade, which is the textbook shape of a path-MTU black hole. A full day went into MSS clamping on that hypothesis and changed nothing. And the router is not busy: aggregate CPU is around 24% on the five-minute average, eth1 is far from line rate, so nobody believes this is a capacity problem.

The one piece of structure in the noise is the clock. The drops stop overnight and resume at 21:00, within a minute of the replication window opening.

Evidence provided

Read-only / Safedropped is climbing, errors is zero

vyos@dist1:~$ show interfaces ethernet eth1

eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
  link/ether 52:54:00:8a:3f:11 brd ff:ff:ff:ff:ff:ff
  inet 198.51.100.2/30 scope global eth1
  RX:  bytes    packets   errors  dropped  overrun  mcast
  884213977412  912004431      0  4183902        0  22140
  TX:  bytes    packets   errors  dropped  carrier  collisions
  771540221883  801229118      0        0        0      0

Illustrative output

Read-only / Safeevery driver counter is zero and stays zero

vyos@dist1:~$ ethtool -S eth1 | grep -E 'crc|length|missed|no_buffer|fifo'

     rx_crc_errors: 0
   rx_length_errors: 0
   rx_missed_errors: 0
   rx_no_buffer_count: 0
   rx_fifo_errors: 0

Illustrative output

Read-only / Safetwo carrier transitions since boot — the link has not moved

vyos@dist1:~$ cat /sys/class/net/eth1/carrier_changes

Illustrative output

Read-only / Safeone row per CPU, little-endian hex

vyos@dist1:~$ cat /proc/net/softnet_stat

3661e02b 003fd97e 0001cebf 00000000 00000000 00000000 00000000 00000000 00000000 00000012
0012629e 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000004
00124ac6 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000004
0012538f 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000004

Illustrative output

Decoded, the three columns that matter — processed, dropped and time-squeeze — read:

CPU 0: processed=911884203  dropped=4183902  squeezed=118447
CPU 1: processed=1204118    dropped=0        squeezed=0
CPU 2: processed=1198790    dropped=0        squeezed=0
CPU 3: processed=1201295    dropped=0        squeezed=0

Read-only / Safe24% aggregate, and one core at 92% softirq

vyos@dist1:~$ mpstat -P ALL 1 5 | tail -5

Average:     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal   %idle
Average:     all    3.10    0.00    1.85    0.00    0.00   23.94    0.00   71.11
Average:       0    4.02    0.00    2.71    0.00    0.00   92.15    0.00    1.12
Average:       1    2.88    0.00    1.60    0.00    0.00    1.02    0.00   94.50
Average:       2    2.91    0.00    1.55    0.00    0.00    1.11    0.00   94.43

Illustrative output

Read-only / Safefour queues, one column

vyos@dist1:~$ cat /proc/interrupts | grep eth1

  34:  418822901          0          0          0   PCI-MSI  eth1-TxRx-0
35:  401337882          0          0          0   PCI-MSI  eth1-TxRx-1
36:  399218440          0          0          0   PCI-MSI  eth1-TxRx-2
37:  402881073          0          0          0   PCI-MSI  eth1-TxRx-3

Illustrative output

Read-only / Safethe default mask, on every queue

vyos@dist1:~$ grep . /proc/irq/3[4-7]/smp_affinity

/proc/irq/34/smp_affinity:01
/proc/irq/35/smp_affinity:01
/proc/irq/36/smp_affinity:01
/proc/irq/37/smp_affinity:01

Illustrative output

Work the evidence before reading on

Three teams have three tickets. Decide first how many faults you think are in front of you.

show interfaces reports rising dropped and zero errors, and ethtool -S reports zero on every driver counter. Both claim to describe the same interface. What does the disagreement between them localise?
carrier_changes reads 2 since boot. What hypothesis does that single number retire, and how much did it cost to run?
The aggregate CPU figure is 24% and one core’s %soft is 92%. Which of those two numbers is describing the resource that is exhausted?
Only sustained transfers degrade; interactive traffic is fine. Name two different causes that produce exactly that signature, and the one command that tells them apart.

Then the question that resolves the incident: eth1 has four receive queues and the box has four cores. Why is only one of those cores doing any receive work, and what state is the router in that nobody configured?

Root cause

1. There are two counters called `rx_dropped` and they mean different things

ip -s link show eth1 and show interfaces ethernet eth1 read the kernel’s per-device statistics. ethtool -S eth1 reads the driver’s own counters, which the hardware and the driver maintain for events they can see: CRC failures, length errors, descriptors unavailable, FIFO overruns.

When the kernel receives a packet from the driver and then fails to enqueue it, it drops the packet and charges the drop to the device. The driver never sees that event and has no counter for it. So a rising kernel dropped alongside a flat set of driver counters is not an inconsistency to be explained away — it is the diagnosis. The loss is happening above the driver, in software, on a link that is behaving perfectly.

This is the single most useful comparison in the whole incident and it costs one command. It was available before the first optic was ordered.

2. `softnet_stat` names the layer, and names the CPU

The decoded per-CPU rows say two things at once. The dropped column is non-zero on CPU 0 and zero everywhere else, which places the loss at the per-CPU input backlog rather than the ring or a socket. The squeezed column is also climbing on CPU 0, which says the softirq pass on that core is repeatedly running out of budget before it has drained its queue.

The processed column completes the picture: CPU 0 has processed nearly a billion packets and the other three cores have processed roughly a million each. This is not a busy router. It is a router doing all of its receive work on a quarter of itself.

3. Four queues, four cores, one mask

ethtool -l eth1 shows four combined channels, and /proc/interrupts shows four queue interrupts, so the NIC is doing its part: the hash is spreading packets across four rings. Every one of those four interrupts has the affinity mask 01, which is CPU 0.

Nobody set that. 01 is what the kernel assigns at boot, and it persists until something changes it. The port was commissioned, it worked at commissioning-day traffic levels, and the campus grew into a limit that has been latent since the day the interface came up.

With net.core.netdev_max_backlog at its default of 1000, the queue in front of that single core holds about a millisecond of the offered rate during the replication window. Whatever arrives while it is full is dropped and counted against eth1.

4. Everything else on the ticket is a consequence

The replication job degrades because TCP has no way to distinguish a packet lost to a full backlog from a packet lost to congestion. It reads loss as congestion, collapses its window, and a bulk stream that should saturate the link spends its night in recovery. Interactive traffic sends too few packets to be hit often and recovers instantly when it is, which is why SSH feels fine.

That combination — bulk transfers fail, interactive traffic is healthy — is also what a path-MTU black hole looks like from the user’s seat, and it is a reasonable first hypothesis. It is separable from this one in a single reading: a PMTU black hole drops by size and deterministically, so ping -M do finds a hard threshold and the drop counters stay at zero. Here the drops are size-independent, load-correlated, and counted on the interface.

The SNMP timeouts are the same fault seen from the management plane: the poller’s UDP packets arrive on the saturated port and queue behind everything else. The monitoring is not broken. It is reporting, accurately, that it cannot get an answer.

Under the hood

A packet crosses three buffers between the wire and whatever consumes it, and each has its own counter and its own fix.

The NIC ring buffer is DMA memory the hardware writes into. When the kernel cannot drain it fast enough the NIC has nowhere to put the next frame, and the driver increments rx_missed_errors — a hardware-visible event. The fix is ethtool -G to enlarge the ring.

The per-CPU input backlog is where the softirq pass hands packets on for processing. Its size is net.core.netdev_max_backlog, and overflow is counted in the dropped column of /proc/net/softnet_stat and charged to the device. The fix is a larger backlog, more cores doing the work, or both.

The socket receive buffer is the last stop, for packets destined to a process on the router itself. Its ceiling is net.core.rmem_max, and overflow shows up in the socket’s own drop accounting rather than the interface’s.

The reason this matters operationally is that each fix is completely inert against the other two layers. Enlarging the ring on this router would consume a change window, add worst-case latency, and leave softnet_stat climbing at the same rate. Reading rx_missed_errors: 0 before proposing it takes three seconds.

Two optics and two outage windows, spent on a software counter

The alert said “receive drops on eth1”. The NOC’s response — swap the optic, then swap the patch lead — is the correct response to that sentence, and it was never going to work.

The gap is that an interface drop counter is an aggregate. It answers “did something drop a packet on this port” and it cannot answer “what dropped it”. Physical faults have their own counters, and on this router every one of them reads zero: no CRC errors, no length errors, no FIFO overruns, and two carrier transitions in sixty days.

A useful rule to carry out of this: an optic swap needs a driver-level or physical counter to justify it. carrier_changes climbing, rx_crc_errors climbing, rx_missed_errors climbing — any of those buys a maintenance window. A kernel dropped figure on its own does not, because it is the one counter in the set that a perfectly healthy cable can produce.

Resolution

Confirm the layer before changing anything. Read ethtool -S eth1, /sys/class/net/eth1/carrier_changes and the decoded /proc/net/softnet_stat together. All three are read-only and together they either support the diagnosis or refute it in under a minute.
Take a baseline you can verify against later. Record the softnet dropped value per CPU and the rx_dropped figure with a timestamp — these counters are cumulative, so a later reading is only meaningful as a difference.
Raise the backlog for immediate relief: set system sysctl parameter net.core.netdev_max_backlog value 10000 and commit. This takes effect at once, does not disturb the link, and buys headroom against bursts. Understand what it does not do: a deeper queue in front of one saturated core postpones the drop rather than preventing it.
Distribute the receive queues across the cores. Write a distinct mask into each queue interrupt — 01, 02, 04 and 08 for IRQs 34 to 37 on a four-core box — so each of the four rings is serviced by its own core. The change is immediate and does not bounce the interface.
Do not enlarge the ring buffer. rx_missed_errors is zero, so the ring is not the layer that is full; enlarging it costs worst-case latency and fixes nothing here. Record that this was considered and rejected, with the counter that decided it.
Persist the affinity deliberately. Writes to /proc/irq/*/smp_affinity do not survive a reboot; the kernel resets every IRQ to CPU 0 at boot. Arrange persistence through irqbalance or a boot-time unit and record it next to the interface configuration, because an unpersisted fix is an incident with a delayed timer on it.
If no change window is available tonight, hold deliberately rather than by default. The drops are load-correlated, so rate-limiting or rescheduling the replication job removes the symptom immediately. Give the hold a named owner and a review time — it leaves a port that will drop again as soon as offered load rises.
Close out the hardware line of enquiry explicitly. Return the two replaced optics to stores rather than scrapping them, and put the reason in the ticket so the next operator does not repeat the swap.

Verification

Softnet drops have stopped, measured as a delta. Read /proc/net/softnet_stat twice, ten minutes apart, during the replication window: the dropped difference must be zero on every CPU. A lower absolute number proves nothing, because the counter never resets.
The interface drop counter is static. cat /sys/class/net/eth1/statistics/rx_dropped returns the same value across the same interval.
The work is genuinely spread. cat /proc/interrupts | grep eth1 shows all four CPU columns advancing, and mpstat -P ALL 1 5 shows %soft distributed with no core near saturation.
The application-level check passes. The replication job completes inside its original window. This is the only measurement in the set that does not depend on the router's own bookkeeping, which is exactly why it belongs here.
The check was run where it can fail. Verify at 21:00 under real offered load, not at 04:00 — an idle port drops nothing regardless of how its interrupts are arranged, so an overnight pass is not evidence.
The fix survives a reboot. In a subsequent window, reboot and re-read /proc/irq/*/smp_affinity. Masks back at 01 mean the persistence step did not take and the incident is scheduled to recur.
SNMP polling is reliable again. The gaps in the interface graphs close — a second independent confirmation, from the subsystem that was originally accused.

Prevention

Alert on the per-CPU softnet dropped delta, not only on the interface drop counter. The interface counter says a packet was lost; the softnet counter says which layer lost it. One of those two alerts ends this incident on the first morning.
Put per-core %soft on the router dashboard. An aggregate CPU number cannot distinguish four cores at 25% from one core at 100%, and it was the aggregate figure that made “we are not out of capacity” sound true.
Validate interrupt distribution at build time and after every reboot. Check the channel count with ethtool -l and the masks in /proc/irq, and record the expected distribution alongside the interface configuration so that drift back to the default is detectable rather than discovered.
Require a physical or driver-level counter before booking a hardware swap. carrier_changes, rx_crc_errors and rx_missed_errors justify a window; a kernel dropped figure on its own does not.
Sample counters at peak. Every reading taken during business hours on this router looked healthy, and the entire evidence base existed between 21:00 and midnight.
Treat “bulk transfers fail, interactive traffic is fine” as a family of causes rather than a diagnosis. Path-MTU black holes and indiscriminate loss present identically to the user and are separated by one look at the drop counters.

What if the ring buffer really had been the bottleneck?

The counters would have said so, and they would have said something different from what they said here.

A ring that cannot be drained fast enough makes the NIC discard frames it has nowhere to put, and the driver counts that: rx_missed_errors climbs in ethtool -S, and ethtool -g typically shows a ring left at a conservative default. In that world softnet_stat can be entirely clean, because packets that never made it out of the ring never reached the backlog to overflow it.

The fix there is ethtool -G eth1 rx 2048 or similar, and it carries a real cost that the reflex version of this fix ignores: a deeper ring means more packets can sit waiting before the kernel touches them, which raises worst-case latency. On a router carrying voice or video that trade is not free, and 1024 to 2048 is a more defensible production setting than jumping to the hardware maximum.

The general shape is worth keeping. Three buffers, three counters, three fixes, and each fix does nothing at all for the other two layers. The operator who reaches for a bigger ring because the interface says “dropped” has a one-in-three chance of being right and no way of knowing which.

Why this matters

Six days, two maintenance windows, two optics, a day of MSS clamping, and three teams each holding a piece of the same fault — all downstream of four interrupt masks that nobody ever set, because nobody ever set them.

The transferable part is the discipline of asking which layer reported the number in front of you. Modern network stacks count drops in several places and present them through interfaces that look interchangeable. show interfaces, ip -s link, ethtool -S, softnet_stat and the socket counters are not four views of one truth; they are five different observers, and the disagreements between them are where the diagnosis lives.

The second transferable part is that defaults are configuration. A router that has never been tuned is not neutral — it is running somebody’s guess about a workload, made before yours existed, and it will hold until the traffic grows into it. This one held for the whole life of the port and then failed at 21:00 on a Tuesday.

Cross-course references

The Linux course’s XXII-Linux-NetTroubleshoot covers the same counters from the host perspective: ip -s link, ethtool, and the sysfs statistics tree.
The Observability course’s XCVIII-Observability-Troubleshooting covers building the per-CPU and per-layer drop alerts this incident needed, rather than the aggregate one it had.
The Ansible course’s XLV-Ansible-Debugging covers the fleet question this raises: how to check interrupt distribution on every router you own once you know it is the kind of thing that quietly reverts.

Reported symptoms

Evidence

Root cause

Remediation

Verification

Prevention

Reported symptoms

Evidence provided

Work the evidence before reading on

Root cause

1. There are two counters called rx_dropped and they mean different things

2. softnet_stat names the layer, and names the CPU

3. Four queues, four cores, one mask

4. Everything else on the ticket is a consequence

Resolution

Verification

Prevention

Cross-course references

1. There are two counters called `rx_dropped` and they mean different things

2. `softnet_stat` names the layer, and names the CPU